The entry for Normal Humans in the Monster section reads:
A Normal Human is a human who does not seek dangerous adventure. A normal human does not have a class. Normal humans have their own saving throws and "to hit" numbers. They have from 1 to 4 hit points (depending on their age, health, and profession). EXAMPLE: An adult blacksmith would have 4 hp, but a young child or sickly beggar would have 1 hp.
In contrast, the entry for Bandit assigns 1 HD and better saving throws for those who band together for the purpose of robbing others.
The difference between the two monster entries, I believe, highlights the impact that leading a dangerous life has on the game mechanics. Why would a Bandit potentially have more hit points than an active and fit blacksmith? Not because of any ability to sustain more physical damage but instead, the bandit's dangerous lifestyle has prepared him better for surviving a fight through skill, luck, etc.
The relative power levels between PCs and Normal Humans also shows that, even though low level PCs in B/X are not of heroic power levels, they are still a definite cut above 98% of the population. The fact that magic-users also have d4 for hit points makes sense as they also do not have the martial abilities of the other classes so their hit points being the same as a Normal Human makes sense. The Thieves d4 hit dice is a bit of a puzzler, however. If Bandits have d8 HD why does a Thief only have a d4? My take would be that Bandits are not "skilled" thieves but instead ruffians and other highwaymen that rob through brutishness instead of the more specialized skills of a thief - Bandits would instead be more like the Bandit Hillmen of the Conan sagas.

This also brings to mind a very sword & sorcery type scenario that low level PCs could survive - a village of evil Normal Humans, led by an vile priest or adept who is aided by degenerate snake men (use Lizard Men stats).
Art Used Without Permission