Raileurope is an expensive option. Always try and use the actual supplier rather than a 3rd party. Raileurope puts on at least a 25% surcharge if not more and then charges you for the delivery of the tickets on top of that.
The reason your train tickets were so expensive buying local is that you bought last minute. Experienced travellers will confirm that the best deal on train tickets are bought when they open, 3 months in advance. The closer you get to your date the more expensive they are.
Amsterdam to Venice, I'd fly Easy Jet from Amsterdam to Milan and then the train to Venice. Book your train ticket on Trenitalia and save a bundle.