But what is function of the infinitive in You are nice to talk to?
The infinitive qualifies the adjective to form what you might call a complex adjective/complex predicate. To be honest I'm not sure what the correct grammatical terminology should be. The way I see it it's not all that dissimilar from the way in which other parts of speech can qualify adjectives too.

For example you can say either,
Is this plate safe to put in the dishwasher?
or Is this plate dishwasher safe

Either Fruit is nice as a starter
or Fruit is nice to start your meal with

Either The battery is hot to the touch
or The battery is hot to touch


It's nice to talk to you. = To talk to you is nice.To talk to you is the subject.
Correct. The two sentences ( I mean It's nice to talk to you and You are nice to talk to), although they will often mean more or less the same thing, have different grammatical structures.