There are plenty of countries that are not colonies of any empire, and in fact have never had anything to do with the British empire, and where almost everyone speaks fluent English now; and it wasn't this way in the time of the British empire. What do you think of that? Who/what made them speak the language? =))