Google’s parent company posted a big jump in quarterly revenue but also ever-rising costs outside its core online-advertising business, showing the drag from its efforts to diversify.

Despite monster results in internet search, Alphabet Inc. shares fell in after-hours trading Monday as investors zeroed in on shrinking margins and slower revenue growth overall.

Alphabet,…