In a case like that, it would be wrong to ignore the invitation, but formal regrets sent to the Captain would cover you completely.
You are a passenger, and therefore a customer. Your time and your personal itinerary are your own. This doesn't relieve you of the responsibility for a reply to a personal invitation, but the reply need not be affirmative.
In any case, this is about Captain's parties to which the whole ship is invited, which is the much more common situation. If you went to the desk and asked them to tell the Captain that you won't be attending, they'd laugh in your face.