Yes, it is generally seen as a rather literary word. In fact I didn't even know this word existed until 2 or 3 years ago, and that was only because I was learning about the Platonic dialogues in my philosophy class. It just seems like a useful word that is hardly ever used, and there's no reason it shouldn't be used outside of literary contexts... other than the fact that your interlocutor might not know the word "interlocutor"
Yes, these expressions would be encountered more widely, but "conversational partner" and "person to talk to" aren't exactly concise, and "listener" doesn't have quite the same meaning. In fact, I think the most commonly used phrase is "the person [that] I/you/he/she/we/they am/are/is/was/were talking to", which is horribly un-concise.
(Derailing topics is getting to be a bad habit of mine...)