There is no implication of reciprocity at all. There are quite a few expressions where "with" doesn't carry the meaning of "togetherness" or mutuality, e.g. to be happy/angry/pleased with...