Definitely his clothes is shabby here...Because the broad specifies it. But the broad is using this word as a metaphor, consequently твоя одежда пообносилась, due to metaphor, becomes ты пообносился. This is a classic metaphor.