That's easy: the worst person to sit next to is the captain. Ask any first officer who flies with my kid. That's absolutely not true, of course. He's one of the nicest guys I know.
A couple weeks ago I ran into an interesting twist to the crying baby issue. I was in MSP for a Sherlock Holmes meeting, and (truth in advertising) when I fly on passes they bump me up to first class if there's space. So I was sitting in first on the way home, and in front of me was a couple with their baby who screamed for virtually the whole flight. It was pretty clear that they had upgraded with miles to get F seats, and in a way it's understandable because the extra space makes it a lot easier to deal with an infant.
Now, for the reason I mentioned above, I'm in absolutely no position to complain about this, and I didn't. On the other hand I got to thinking about the people who paid what I consider crazy money for a wider seat and more legroom (surely it's not the FOOD!), and what they must have thought about a bellowing kid. As far as I could tell, in this instance nobody complained, but I imagine some were not happy.
I've thought for a long time that when the day comes (and it will) when you can use cell phones during flight, the scene of the next airline murder will be the first class section (typically 16 seats) on a narrow-body aircraft. Half of the paying customers will have bought the seat for peace and quiet, and the other half will be doing business on their phones. There will be blood in the aisle.