they do not want to go to Russia...
People usually leave the country when there is no other variant or when they can get something significantly better. There are some Russians even in Tadjikistan.
And also - Latvians did not want to give power to Russians anymore (nationalism was way more stronger back then).
Russians did not rule. Look at the Soviet leaders. Most of them are not ethnic Russians.
And actually, do you suffer the presence of these people? Would it be better if all the Russians were exiled?