I would not recommend staying in San Pedro. Even though it sounds like you are leaving from that port instead of Long Beach, the only thing to do down there is stroll the restaurants and shops at Ports O Call. Staying near the Long Beach port is an option as it's not that far from San Pedro and is between LAX and San P, so not out of your way.
Long Beach itself is spotty, parts are nice and parts are aweful. The pier area has been revitalized and is quite lovely, beautiful on sunny days. Right there you will find the relatively new Long Beach Aquarium, the Queen Mary, and lots of shopping, movies and restaurants all newish and withing walking distance of some hotels. I think they have a shuttle for free around the area. That way you could avoid driving in LA, never a fun expereince, even for those of us who live there. Then take a taxi from there to San Pedro.
I recommend deciding what attraction you want to see in So. Calif., stay near there and then take a taxi or shuttle to the port.
Margie