In Athens, either just hiring taxis or using a guide (I have a great recommendation if you need one) I would see new Acropolis Museum, Temple of Zeus, Old Olympic Stadium, and Syntagma Square. I would then go to Plaka for shopping, eating, and drinking. All of these have limited walking (depending on shopping) and are relatively flat. If you want, those of your group with more energy could leave the Acropolis Museum and climb the acropolis and come back down through the Agora and meet up with the rest at the Plaka.
I haven't beel to Olympia in a long time; then, if I remember right, I just took a train and walked the ancient sites. If all excursions are too strenuous, I have always found finding a cafe and getting some lamb and retsina is a wonderful way to pass the time.