In the "old day" your hat would be placed in the cloak room. Now the hat should be placed under the chair. It is never acceptable to wear your hat at the table.
I know when I was growing up that if I wore my hat at the table it was promptly removed for me, by my mother or father, and used to swat me across the head.
It's hard for me to wear a hat, which I always do, at almost any sit down eating situation, even a picnic table.
For a woman, the old double standard comes out. It is perfectly acceptable for them to wear hats at the table. It is considered fashionable. I don't really think a baseball cap is fashionable but I guess it covers up "hat hair".