My son and his wife did this with a 4- and 6-year old. They had the standard queen bed, and the kids were on pull-down bunks on the side walls. No problems at all, except for the one night when one of the kids dropped his stuffed frog on my sleeping son.
To make it easier, PACK LIGHTLY! And keep everyone's stuff organized and out of the way.