A couple of years ago DW and I did a repositioning cruise back to NYC from San Juan. Our ship was the second to dock at Red Hook, the first being the Queen Mary 2. This may give you some perspective regarding the locations of the cruise terminals.
The Manhattan cruise terminals are located on the Hudson River between 41st and 43rd street (west side) about where the Lincoln Tunnel is and next to the USS Intrepid.
Red Hook, Brooklyn is south of Manhattan (towards the East River)opposite of Governor's Island and directly across from Liberty Island and the Statue of Liberty. For more information you may want to Google "Red Hook Cruise Terminal" and look at the website www:brooklyncruiseguide for additional information and maps.
It's quite a sight to be on the deck of a cruise ship at 4:00 in the morning with hundreds of fellow passangers and be going under the Verrazano Narrows Bridge.