Well, Old San Juan and El Morro is a very nice walk -- IF you are healthy and fit and used to walking and unencumbered. For your group, it would be a night mare, rough streets and sidewalks and many hills.
I sincerely recommend that you get a local guide with a minivan for your group. We have had good success with Tours by Locals
San Juan Tours with local private guides.
They can design a tour just for you, the fort only or more comprehensive. You will be so much better off. Be sure to let them know you need two child seats.
I suppose you have made a firm decision to do this with the two infants. Honestly, I don't recommend it. Taking infants on a cruise ship is a huge hassle and is really going to cut your fun. They will NOT be eligible for the children's program. In some instances it is possible to arrange baby sitting but it can be very difficult. Frankly on board and ashore you are going to spend more time taking care of the children than enjoying your vacation. But -- if you have decided to do it, then you need to plan to make it as easy as possible. In many ports this means an independent guide and van.