Definitely go with the khakis and other dressy casual pants. Crisp collared shirts, cardigans with crisp dress tees would be fine. Take a peek inside the Cunard brochure to see what the models are wearing - the cruise lines give those pictures as a very subtle hint of the type of guest and style of the cruise line. (Several top cruise lines executives have told me that hint over the years.) Cunard is more formal and traditional than the other cruise lines, so err on the side of more dressy than casual.
My very British aunt, who was the daughter of a British diplomat during World War II, always suggested that it was never wrong to be overdressed - people don't think badly of anyone who is more nicely dressed and it is removes the chance of discomfort. Never feel badly about being too dressed for the occasion or the company.