we don't say "come to me" in the meaning you want to have. It is a very specific phrase, used to ask a person who is standing a few feet or even inches away to come very close to you.