"To which" is definitely correct here. It is a little formal, but not unusual in a written story or joke.

I'm afraid I can't explain why "to that" is wrong. There are other situations where "to that" is correct, but not here.

It would also be fine to leave out the phrase and begin "The little brother replied ..."