There are many convenient ways to get into Manhattan from New Jersey, so it is a viable option to stay in a NJ hotel and take public transportation. It used to be that you could the PATH train (NJ version of a subway that led into the city) but the line from Jersey terminated in the World Trade Center, so obvioulsy it isn't there now. I don't live on the East Coast so I don't know the best way anymore.
There are Path trains you can take into the city from Newark that will take you right to Greenwich village or Macys. But Newark is a distance away from Bayonne. There are many ferries that cross the river as well, from Hoboken and Weehauken. There mnight be one from Jersey city (which is south of both those cities) as well.
What we need is a New Jersian to weigh in on this post.