I would never carry my passport ashore. Only my license and cruise card. You never know what could happen. If it gets lost or stolen, it could cause a problem trying to get home.
To get back on the ship at ports, the cruise card is usually sufficient to get back on. If a photo ID is needed, the cruise line will usually state so. I still always carry my license just in case though as losing a license isn't as critical as a passport.