Unless you want that day to be a total fitness program day, do NOT try to walk to Blackbeard's. St. Thomas is very, very hilly, and the castle is on top of a hill. A long, very strenuous walk. And Coral World is quite a distance away from the pier. Although I really enjoyed it. Would definitely go back again. Small, but interesting.
Just my 2 cents,