No problem to hit all of those very easily:
* The temple is right to the left of the pier ... it's about a 5 min walk and you can't miss it.
* The banyan tree is right across the street from the pier
* To get to Huggos, walk right into town, and it's about a quarter-mile-half-mile up on the ocean side. A nice plesant walk as you stroll through town. By the way, Huggos is the only place on the sand -- it was grandfathered in. Service was kinda- eh, but the view was great. We just ordered nachos and two drinks, and it was a nice respite (and not too pricey)
By the way, you can also check out the vacation home of the royalty on the walk, and the first christian church on the island across the street.
Oh, and for a pit stop, they do have the shuttle to the Walmart -- lots of crew and passengers made a quick beeline there, too.