I understand your lack of enthusiasm.
My husband's mother died of cancer in March ten years ago, but her birthday is the day after Christmas. He gets very gloomy on that day. We try to keep him occupied but we also make sure that something on the tree or in the decorations is a tribute to his mom. He lost his dad two years later.
I lost a grandfather near fathers day.
Fourth of July is messed up for me now as is my birthday. It always seems that things happen around holidays or special days.
We have to go on. Our loved ones that have passed would never want us to stop celebrating special days just because they are gone. Celebrate them and pay a special tribute to your brother. Perhaps a special ornament on the tree that is for him.