I agree, I'm 25 and my fiance is 27. We always choose late seating. The early seating tends to be full of families and seniors. You'll find more people your age at late seating.
It's true, though, that you won't be hungry for the midnight buffet. But you won't miss any of the shows or activities.
Fantasy Fest in Key West