You are probably confusing "своё" and "твоё".

Let's examine the single sentence first: "То, что своё, нас не минует". The sentence is speaking about "нас" (can I say it in English this way? "the sentence is speaking about..."?) So, "своё" (a reflexive pronoun!) refers to "нас". Better, just consider "свой" as "one's own": its specific referrent is defined by the context. "What is ours (own) will not pass us by". It means "we cannot avoid something what is ours", i.e. what is prescribed to us by our fate.

The last sentence is understandable now: "То, что уходит, не твоё". "What is leaving (you), is not yours". It means if something leaves you, it is not your fate.