Take a small package of Lysol Sanitizing wipes (not Clorox) to clean out the ice bucket if you are worried and then rinse it out to remove any odor or flavor.
There are many places you can pick up bacteria so I personally would opt for a good rinse and clean of the bucket. Just take normal precautions but I wouldn't obsess about germs. The most dangerous place to pick up something is from the hands you shake when meeting people. So wash after going to the bathroom.