I have tried to bake according to Imperial and it's really hard!
I'm a bit surprised by this comment -- I don't think there's anything intrinsically harder about baking in Imperial than baking in metric!

Well, admittedly, it could be difficult if you were trying to follow an old Victorian-era cookbook that called for "a gill of milk" (~0.1 L) or "a half-peck of apples" (~4.5 L). But I can guarantee you that in any modern (post-WWI) American recipe, almost everything will be measured by the "standard cup" (~240 mL) or fractions of a cup -- or for flavorings used in very small quantities, such as black pepper or ground mustard seed or oil of peppermint, by the "standard teaspoon" (~5 mL).

There are exceptions to this: fresh meats, such as beef or chicken, are usually measured by weight. And butter is measured sometimes by weight and sometimes by volume, so that one recipe may call for "a quarter-pound of butter", while another recipe calls for "a half-cup butter" -- yet the amount of butter is the same in both cases.

On the other hand, I agree that trying to convert an Imperial recipe to a metric recipe can be tricky -- for example, if you have a US recipe for chocolate cake that gives the oven temperature in Fahrenheit, but you're in a UK kitchen and the oven has Celsius or "gas marks". Also, granular or powder ingredients (flour, sugar, rice) are invariably measured by volume (cups) in American recipes, but by weight (grams) in many European recipes.

However, if you're using an American cake recipe with standard American measuring cups and an American stove, than "baking with Imperial units" is as simple as can be!