There are many idiomatic phrases in Russian, just as in English. Compare «я хотел(а) бы» - "I would like", which also uses the past tense (even though the desire it expresses is in the present).

In the case of «поехали», I would guess it adds a sense of urgency (let's be gone from here sooner rather than later), but I could be wrong about this.