Don't know for sure, but from what I see on these boards and elsewhere, very few people who cruise want to visit New York. A quick snapshot of the Statue of Liberty and the skyline and they're happy, then it's off to the airport to get out of town.
In my experience, a good percentage of cruisers are somewhat apprehensive travelers, who choose cruising at least in part because it's a low risk, low decisionmaking, turnkey operation. New York frightens many people in this category, so if they visit at all it's going to be on a tour and not on independent travel. Apparently there is so little interest in Manhattan shuttles that the cruise lines don't bother with them, figuring that the few who are willing to brave the big town can find a cab or a limo like you did. And, of course, the locals will usually have friends or relatives pick them up.
I think it's a shame that relatively few people who sail to and/or from New York want to visit it, but your story about the absence of shuttles seems to confirm the point.