We've never paid for a specialty restaurant and will not do so for the foreseeable future.
My first cruise was in 1991 on Carnival Ecstacy and perhaps the memory fades after all this time, but I do believe the food back then was better quality than it is now. That being said, my 21 yo palette was not as refined then as it is now. At 21 I had probably never eaten in a fine restaurant so perhaps I was more apt to be impressed?
My recent experience is with HAL, RCCL, Carnival and Disney. i would probably rate the food in the order of HAL, Carnival, Royal, Disney. I noticed the biggest drop in quality with respect to desserts. On Disney, I rarely ate more than a bite or two of any dessert. I really look forward to booking our next sailing with Celebrity as I have heard good things about food/service.
For now, we cruise with our young children, and as long as they are not welcome in a speciality restaurant, specialty dining is not an option for us.