Yes. And your sentence "I was going to have dinner with him two days later" is better translated as

Через два дня я должен был (по)обедать с ним.
or
Через два дня я собирался (по)обедать с ним.
or
Через два мне предстояло (по)обедать с ним.

And I have put (по)обедать because either imperfective or perfective works fine, but it depends on some nuances. If you want to emphasize some details of the activity (i.e. having lunch with him): what you were going to eat, what you were planning to discuss at lunch, what clothes you decided to wear for the event etc., then only imperfective is used.
If you emphasize the fact itself, then perfective is preferable, but the imperfective is still possible as it seems to me. But it's a very subtle nuance.