Funny you should put it that way. They started self-debarkation before we went to the dining room. Then we went and had breakfast and it was ironic that we were seated with a family from Indiana who were going to stop for the night in our hometown of Athens, AL. So while I was eating a nice omelet people were lined up like prisoners two decks below me. Then as you said, we went back and got our carry-ons, did a final pit stop in the cabin bathroom, and went to the library which was our designated VIP waiting area. We got there and waited about 10 minutes and then the red shirted "Ask Me" lady arrived and said "follow me". Got to the terminal and a security officer - seeing my wife's use of a walker - pulled us out of line and let us cut across to the elevator ahead of a hundred people. Then we got "the" porter who sweet talked us to the head of the Customs line, and we were out the door. Really, the longest wait all morning was for the hotel shuttle, and part of that was a communications breakdown with the guy who directs shuttle traffic (he thought we were going to a different hotel than what I told him).
The hotel people couldn't have been nicer. They even loaded our trunk without asking us (which would make it a no tip situation although I did tip him).
We probably were on the road just as quickly as many (but not all) self-debarkers.