Well, William Randolph Hearst's enormous estate/castle -- the inspiration for the fictitious "Xanadu" in Citizen Kane -- is located roughly halfway between S.F. and L.A. (not too far from San Luis Obispo), so you could possibly make a day-trip out of that.
In S.F. itself, I don't think anyone has mentioned the Legion of Honor art museum, which is not far from the cliffs of Lands End, where you get a spectacular view of the Bay. (But I haven't been there myself... I've only seen it in movies.)