You can say "шестерка" or "шесть" in this case, but "шестерка" means that we count them (servants) as a whole. Even grammatically they become singular - моя шестерка слуг, но мои шесть слуг.