I prefer it that way, to be honest. I'm reminded of the Highlander universe--the first movie made it seem like there might have been a few dozen immortals. By the end of the TV series there must have been hundreds, if not thousands, plus the watchers, all of which really bent the whole secret war angle. It's like the difference between low fantasy and high fantasy.
Tangentially, the power of the offhand remark for worldbuilding is known.
Escape from New York: "You flew the Gullfire over Leningrad."
Ronin: "Don't I know you?" "Vienna." "Ah!"
ANH: "General Kenobi, you served my father during the Clone Wars..."