I have also been browsing my B/X rulebooks for the Red Box Calgary game.
Each of those versions of D&D have information and rules about encountering a castle in the wilderness. This is another one of those "gameist" things about early D&D that I love.
In the OD&D books, the text specifically mentions the Outdoor Survival playing board and that ponds indicate castles.
B/X doesn't have the same relationship with a board game so I have always used the random wilderness encounter tables to indicate when a castle is encountered. The B/X wilderness encounter tables give chances to encounter "Fighter", "Cleric", and "Magic-user" in addition to "NPC Party" and "Adventurers". When the result was one of the singular I would often use that as a castle encounter.
The part that I like best is that each of these editions has a very simple table to determine what happens. I actually somewhat prefer the OD&D tables as they are crazier that the B/X versions.
The OD&D tables give a chance that the Lord or Necromancer or Evil High Priest (EHP) that owns the castle to have retainers such as Giants or Manticores or Vampires.
The B/X tables are much more boring in that they just list the horsemen patrols. It does give a suggestion that the "rest of the force" may include trolls or superheroes mounted on Griffons but I like that these types of things are the default in OD&D.
I also love that the OD&D rules say that Fighting Men will demand a jousting match and will demand the loser's armour if he wins. Magic-users are complete dicks and send passersby on errands by using Geas. And Clerics demand tithes and use a Quest spell or just try to kill the passersby if they are unable to pay.
The B/X rules are more staid with the castle owner either chasing the party off the lord's land, ignoring the party, or being friendly (either feigned or genuine).
Of course, as with all things in D&D (especially older editions), the imagination of the DM and the reactions of the players are the only limiting factors to these types of encounters but I love the fact that the default setting of these early editions includes this kind of stuff.
