You are right, "Let's go" = Пошли.
I'm going / I'm off = Пошёл / Пошла.

It is past tense in form, but expressing the present/future.