I see your point, WDL. In that sense, you are being penalized. But consider the logical conclusion to that line of argument. At home I like eating lobster in my underwear. Why can't I wear only underwear to the dining room as long as it is clean and pressed? Why should other passengers care what I wear? Am I not being penalized in exactly the same way?
Dress codes are indeed a social construct and nothing more, but that doesn't lessen their importance, we being social animals. This is why not following them sends the same message that you say offends you: "I won't dress according to the social code because I don't think the rest of you are important". It is the person not following the code who is doing the looking down.