This particular sentence is not okay :)
You can say «Я изучаю русский язык по книге» but not «книгой/книгою».
In general, yes, the instrumental case works like you said.
E.g. you can say «Я учу...