You can dine at the buffet on formal night without getting dressed up. Jeans, shorts, anything but an uncovered bathing suit is fine at the buffet. After dinner, you can move about the rest of the ship in your casual attire since many people change after dinner (or do not attend the formal dinner).
Keep in mind, however, that depending on what ship you're on, the menu may be quite different from the main dining room. For example, I have never seen lobster tails at the buffet. Others have, so apparently it varies from ship to ship. If food is important to you, than you may not want to miss the main dining room on formal night since that is when they serve the best meals.
If you want to go to the main dining room, but you don't want to get all dressed up, I discovered on my last cruise that a dressier casual works well. I had not planned on going to formal dinner, but at the last minute my traveling companions decided that they wanted to go. I didn't want to dine alone, so I just put on the nicest outfit that I had with me which was a pair of black slacks and a plain button-down blouse with black loafers (this is one of my work outfits and I work in an office where business-casual is the norm). I really expected to feel out of place, but I didn't. Dress varies so greatly, that as long as you make an attempt to look "nice", you'll be okay. Now that I know how low the price of admission to formal night is, I will never skip the lobster again!