"Да" doesn't mean "yes" here, it means something like "well.." and depending on intonation it may show annoyance or hesitation of the speaker. I think "well, not really" is a possible translation of...