It's been a while since my wife and I were in Key West. You'll miss the sunset celebration, but you really need to get off the boat and take a walk up and down the main drag. Go to Papa Joe's for a cold beer or a soda. This is a fun town any time of the day!
Just remember Key West (the part you will be at is a small portion) and 2000 of your new friends are getting off the boat too! So get out early and start your sight seeing and you will have plenty of time to have a cheeseburger in paradise