I believe, "Нет, в самом деле" is used to negate the words of the interlocutor and turn the conversation back to the previous topic.
It can have both meaning here: "actually, no" and "no, it's true" (i.e. "no i will not stop; my words are true"). The both meanings merge and complement each other, so it is hard to split them.