I think "мой крошка" is allowed, but is not the best choice in the normal speech. You should address a baby boy as "моя крошка", "моя утка", "мой селезень", "мой утёнок", "моё солнышко" and that would be the best. The usage of "сладкий" is usually avoided when addressing a baby for being overly-luscious. Those who address a baby that way in Russian are usually under a significant influence of a foreign language in which that would be a norm.