I think much depends on the ports being visited. Some ports have you pass two stages; one to get access to the terminal, and the other to get on the ship. I think generally, your ship card is sufficient to get back on the ship, but port crew require photo ID.
Some ports, require photo ID for the adults but not for kids. I always keep my passports in the safe. I bring our driver's licenses for photo ID to get on/off the ship or through the ports.