No you can't. He was with us means something different.

у нас = at our place

I think that may be more American than British. I think in Britain we would be more likely to say At our's