I've moved twice in my 15 cruises, once because I was seated with two couples who knew each other very well and they totally excluded me from conversation.
Another time Carole Dunham and I moved because there was a gentleman at the table who was very confrontational and all he wanted to do was talk war, politics, guns, etc. We sailed on the Royal Princess out of New York 9/10 and this guy was so wound up he would talk about nothing else. We wanted to try and keep it light and still enjoy our cruise as best we could under the circumstances so we moved.
Generally, I would say I wouldn't move but these two times were impossible so it was necessary.