I'll find someone (something) who can force you to do something or prevent you from doing something.

(ладно бы их не было means "it would've been allright if they weren't there (, but...)")