I have done it, and I know people who have done it - and I say (and the others also say) that I would never do that again.
Sorry - the Jamaican government has been trying for decades to change the attitude of its own people towards visitors to no avail.
So, now you have a situation there where the locals are angry because the government keeps them away from visitors - yet when they do get access to any visitors they are downright scary.
You will see when you get there - just walk up to the security gate at the entrance to the port area - and you see the hordes of taxi drivers ready to come down on you.
In other similar ports I have been in situations where taxi drivers have grabbed me and separated me from my own friends. I had to literally fight back just to get loose and rejoin them.