I have not sailed from Norfolk, but I just returned from there last night. Based on my 4 day visit I can tell you that the weather can be very unpredictable. When we arrived there Friday the weather was lovely. It was about 50-60 degrees and sunny. However when we got to our condo which was Oceanfront on Virginia Beach the wind from the ocean was really blowing and we needed jackets as we became chilled very quickly. The next day was much the same. However, Sunday the wind was blowing at gale force, the surf was rough with very high waves, the whale watching tour was cancelled and the temps plummeted. It had to have been 20-30 degrees and felt even colder than that.
It is possible that you may get lucky and sail during a warm snap, but I wouldn't count on it. I'd pack something warm just in case and take a warm coat. Lets hope you don't need it. Bring your binoculars, you may get to see some whales on this trip.