It is used often in speech but rarely in formal texts. No, it is not impolite. But maybe it is a bit more colloquial... The intonation of the sentence is changed and I am not sure how to describe the difference - it is very subjective. It may become more intimate maybe or more energetic...