Orca whale watching is a top tour option in Victoria with the 3 resident orca pods. I am surprised with your "numerous" trips here that you missed this?? Many advertisements and boat tours available. Prince of Whales is a big operator right on the Inner Harbor.
You have no option for an independent vender, you get in too late for all the scheduled tours, unless you are looking for a private, costly charter?