I travel extensively with my children. I always have taken a letter even though they have passports. Luckily my ex and I get along fine so it has never been an issue for me. Sometimes I have been asked for it on trips to Mexico (and other places) and other times I have not. It depends on the person who is checking you in. If you're asked for it and don't have it they could not let you travel. I would recommend either getting a letter or court order.