Whether you'll be better on port or starboard is all relative to the particular port you're in. Some ports you may always tie up on one side or the other, while other ports it changes all the time, and yet other ports you're tendered. Plus, on your particular cruise, port side could be 'best' in one port and starboard side could be 'best' in another port.
As I always tell people when they demand the starboard side when doing an Alaskan cruise because they want to see the scenery, while you're on your balcony watching that, you may be missing the whales on the other side of the ship.
Whenever arriving into or departing port, it's always best to be outside up on the highest deck so you can walk from one side to the other to take pictures because it always changes. And if you're on your balcony, you're missing half of it. Plus, the sun moves and what was a bad picture in the morning while you were coming it, may be a spectacular picture in the evening while you're going out.
Besides, most people want to be in port walking around, seeing the sites, going to the beach, or taking excursions, so it's not like you plan to spend all day on your balcony while in port. And if you do, you may be in the sun or shade all day, which may or may not be what you want.
So the only hard and fast rule is about which side of the ship is best.... there is no hard and fast rule.