Well, I don't know what the "SE" station is, but the trains to and from London Charing Cross stop at Dover Priory, which is a 5 pound flat rate cab ride to or from the pier (or at least that was the price this past May). You could never walk it with luggage. There are also trains to Dover from Victoria, I believe, and I don't know whether they stop at the same station in Dover. Perhaps they do and perhaps that station is walkable.
But just remembering the port area, it's a big place, and I'd be surprised if anyone would want to walk to the terminal from anywhere outside the port's real estate.