Yes you are right,
It's what peope say if they have stayed over with somebody, spent a few days in their house, for example. You usually add something about how nice their house is, or what good food they served, or something like that.
You can also say it if you've only been at somebody's house for dinner.
It's usually in response to a comment from the host:

Host: Well it was so nice that you were able to come. We've wanted to have you over for months, you know! Thank you!
Guest: Not at all! Thank you so much for having me! What a fantastic dinner!