$5 is fine if there are a bunch of folks on board. $5 is fine if you're on my boat and that's what you've got to show appreciation for a job well done. Personally, I tip 10 - 15% of what I paid for the tour... but that's just me.
If you're on the Can Can, the captain doesn't expect anything, and the deckhand likes to earn at least $100 per day, but everyone understands the tip is for EXTRA service beyond getting you out and back safely and enjoyably.
Actually, the best tip you can give a small charter operator is a referral or repeat booking. That sort of thing means more to me as a guide than any cash tip I've received.
Personally, I don't like tip jars. I think it's a tacky way to solicit tips. If you had an experience that exceeded your expectations (realizing what is and what is not in the crew's control) and IF you can afford it, give a moderate tip to everyone who helped exceed your expectations. Sometimes the captain will refuse tips and ask you give them to the crew, while other operators will happily thank you for your money and not share with their crew. Every boat is different so it's best to either ask who to give money to, or just hand a bit to each crewmember directly.
More importantly than how much to tip, don't forget to HAVE FUN!