Some clarification: "за + accusative" means direction. Therefore, your choice is "заехать за магазин" (it means from point A, which is not behind the shop, to point B, which is behind the shop).

However, "за + instrumental" means location. "Я сейчас за магазином" = I am behind the shop now. It can also mean motion like "проехать за магазином" (to drive along from point A, which is already behind the shop, to point B, which is also behind the shop).

The verb "заехать" can only be used with "за магазин" (to where?).
The verb "проехать" can be used with both "за магазин" and "за магазином", but the meaning is different:
Я проехал за магазин - I started driving not behind the shop, but then I moved to behind the shop.
Я проехал за магазином - My whole path was behind the shop.

Another meaning of "заехать за + instrumental", as Doomer noted, is to make a stop to pick up something (or someone):
Он заехал за мной. - He came to my place (on his way to somewhere), made a stop to pick me up, and continued driving.
Он заехал в магазин за продуктами. - He made a stop at a shop (on his way to somewhere) for some food.