Oh, it does have sense. I don't know the right literary term, it's a poetic contrast based on "I am alone without you - with you I'm alone too"
Besides, the line next to the discussed one, is "mit...