I went on a banana boat ride in the Bahamas on my last cruise there, and we had a blast. No dangers of drowning or anything like that. We all had life vests on, we didn't tip over, and just had lots of fun.
We own our own ski-boat at home, and are in the water all the time - tubing, wakeboarding, skiing, etc., so there was no fear of the water for any of us, and we also communicated to the driver of the boat our vision of the banana boat ride - and a boat driver is obligated to circle back around to pick up passengers who have fallen off. Yes the waves may be rough, but the life vest keeps you afloat, and the boat should never be more than 50 -60 feel away from you to get back and pick you up.
Always the best advice is to use your own best judgement. If the boat driver doesn't seem to listen well or communicate with you, or if you don't feel comfortable, don't go.