Well, my question is: is it 'Он меня раздражал' or 'Он раздражал меня'? What is more correct?

And just out of curiosity: is it 'был бы' or 'бы был' ?
I must say I often have trouble with word order.