You could even say:

What time is it?
It's five to six.
That late?
Not to me it isn't!

I think in most cases you would be understood saying 'So late?' but 'Is it/it's that late?' sounds much...