At sea which side you prefer might be determined by whether you like sunrise or sunsets.
In port, generally in St Thomas to look at the harbor you would
want port side (if the ship docks at Havenside) if the ship docks at the sub base there would be no preference. In Antigua , the would be no better side to be on. In St.Martin it might depend on where you ship anchors but I would think port to be slightly preferable. In Barbados , there is not a whole lot to see from the ship in the harbor and it also depends on where the ship ties up. No preference there. In St. Lucia,the harbor view might be slightly better on the starboard side.
Southern Caribbean is not like Alaska where you might get a better view of something as you cruise by all day. Much of your cruising will be at night,nothing to see and daytime cruisingis not close enough to anything to make one side better than the other.
Bottom line , no real advantage port or starboard.
You might think about which direction you will be headed on your sea days and whether you prefer more sun morning or afternoon.