It's an Oscar Wilde quote (though I believe it should be "... one thing in life...".

What he meant was; being in the public eye, having people making gossip and possibly telling untruths about you...