This is not a simple sentence, and you may use бы twice here - Если бы меня не трогали, я бы молчал.
This is the quote from Chekhov's humoresque "Язык до Киева доведет" (another proverb) - Это вы разговариваете и мешаете, а не я. Я молчу, брат... И вовсе молчал бы, ежели бы б меня б не трогали б.
He lived in 19th century and the language has changed since those times... a little
And, yes, you're right, you'll find no one nowadays who would say so.