You are mixing things.
"Крошка" as a crumb is one thing, and "крошка" as a quality of a person is another. The latter it's an emotionally charged word that implies a tiny size (often figuratively, just like "baby" in English does not necessarily mean that the person is young) and corresponds to "крохотный" ("tiny"). They have different grammatical qualities, and it's represented in dictionaries:
крошка
I м. и ж.
(о ребёнке) little one
II ж.
(хлеба и т.п.) crumb
It says clearly that "крошка" in its first meaning can be either masculine or feminine, i.e. of common gender.