There is only so many ways you can move, shoot and melee, and a designer's background is often revealed in their design choices when they make their own games. Some influential rulesets:
The elephant in the room says hello. Not only are there a zillion homebrew spin-offs (the "I want to make a better 40K" is one of the most common game designer "subtypes" and often the first step in a budding designer's career) but the professional writers seem unable to distance themselves from it: bi name titles like Bolt Action, Flames of War, and Empire of the Dead share a very similar core.
Pretty much every15mm sci fi ruleset (and every man and his dog are pumping them out nowdays) owes Mr Tuffey a debt of gratitude. You aren't a proper indie writer if you haven't made a 15mm hard sci fi ruleset. And it isn't a proper hard sci fi ruleset if it doesn't have at least a hint of Stargrunt in its DNA. Examples: Tomorrow's War/Ambush Alley, plus pretty much every sci fi wargame on sites like the Wargames Vault.
The Savage Worlds multiple dice sizes (d4/6/8/12) to beat 4+ mechanic is quite trendy at the moment (though I would not claim they invented it... ....it's more about who "popularized" it). Warmachine surprisingly has fewer imitators than I'd expect given it's been a popular ruleset for a while, though a few share its mechanic not many follow it as closely as Gruntz. Blood Bowl has spawned more than a few imitators in the fantasy sports realm. Full Thrust (and I'd argue, Starfleet Battles) have strongly influenced the wet-navy-in-space hitboxyness of spaceship combat. DBA's minimalist approach has impacted on many. I'm not a huge mass-battle fan, but I suspect games like DBM have had an influence on game design in the historical sphere. Two Fat Lardies's card based activation is also beginning to exert its influence. Necromunda/Mordheim pretty much defined the "skirmish campaign" genre.
I'd also like to differentiate between "current influence" and "original root cause." I'd argue though you can go waaay back and point to 70s games like Laserburn, Chain Mail etc as being "grandaddies" and the "origin point" of many rulesets - (thus establishing either hipster cred or extreme old age) - they are not as influential as current rulesets they in turn may have inspired. Hmm, I might need to explain that better. Featherstone's or Wells' wargaming treatises might have been "ground zero" but I'd argue they are not as influential as say, 40K 2nd edition or Stargrunt on current designers, today. A bit like fantasy literature at the moment. I'd argue GRRM's Game of Thrones has more influence on the current crop of fantasy authors, than original "grandaddies" like Robert Howard, CS Lewis, Ursula Le Guin etc.
A well rounded education....Many writers can never "break free" of a particular mindset. The 40K Hobby seeming to exert the strongest grip. Stargrunt is often cited as a trigger for "breaking out" of that mindset.
So what rulesets are good "textbooks" on rule design? What rules will "expand your mind" so to speak?
Well, I'll come back and add to this list, but here's some that pop into mind. I've focussed on free, current games, rather than tatty, tobacco-scented 70s rules you have to buy off eBay. Bear in mind these are not necessarily my favourite rules, often they are actually rather poor games to play - but they are ones I believe can teach interesting lessons.
Stargrunt (free) is famous for liberating players from the 40K mindset, emphasizing suppression, morale, troop quality, long weapon ranges. Ambush Alley/Tomorrow's War updates the concepts.
Weirdly enough, I'm going to recommend Lord of the Rings:SBG (yes, a 40K spin-off!) as a good example of clean game design, resource management, and how to "evolve" a ruleset. For those wanting a better 40K, well, Games Workshop already did it. Starship Troopers/Battlefield Evolution is also a similar "40K evolution". A Fistful of Tows has a similar 40K vibe but speeds up modern mass combat with some clever ideas.
Infinity the Game (free) is also a must-read. It's use of unlimited reactions, and resource management in activation is a real eye opener. It also creates lots of agonizing decisions.
Savage Worlds ($10) may technically be a RPG, but it shows how to make great special rules, and is a good example of how to make a "one size fits all" ruleset.
DBA (and it's spin-off Hordes of the Things) was revolutionary for its time, and is interesting today in the way it abstracts mass battle combat.
Too Fat Lardies have a very distinct style - their use of blinds, card activation and an insistence on "friction" is well worthwhile for any historical gamer. Pick your preferred genre, though the latest - Chain of Command is the most polished to date and its pre-battle minigame is unique - worth the price of admission alone, and Bag the Hun is particularly different to most other aerial wargames.
Anything by Brent Spivey (Bombshell Games) as his rules always explore the edges of the box. Pick a genre that interests you. The HAVOC rulebook is a bit inaccessible and I'd perhaps recommend Rogue Planet as the most interesting and unusual.
Makatishi also makes interesting choices in what he abstracts (JuJu Man) as well as the dice pool of CROM. Speaking of dice pools, I found Bushido (free) an interesting read. Hind Commander has both an interesting, niche topic (gunship combat) and has some worthwhile mechanics that can be easily adapted.
Shipwreck! shows not all naval games need to be about frightening amounts of book-keeping, and Battlefleet Gothic (free) (yes, GW again!) blast markers and order system shows space combat can do new things. DP9's unconventional Silhouette system is best showcased in Lightning Strike which is the most interesting space game I own and allows individual fighter aces to fly alongside capital ships. Whilst very flawed, 2300AD Star Cruiser has a fascinating design concept revolving around detection.
Battlestations! (WW2 naval) is a great example of how to stick to a game design concept and is a rare thing - a genuine fleet-level game alongside Starmada: Fleet Ops. General Quarters (1 & 2, and to a lesser degree 3) also give great lessons in how to make a naval wargame. It's a set of rules I often browse through even when making homebrew rules for (apparently) unrelated topics. Highly recommend. Coaling Stations order mechanic is one I've borrowed for my one games, and Triplanetary shows the original and best vector movement system, ever. Voidstriker is also a very interesting and original space rules set.
Song of Blades and Heroes is the flagship for the "simple stats and lots of special rules" trend and has sandbox-y fun in its unit design. However they are a cautionary tale for re-using mechanics. Also guilty of copying themselves too often is the 2HW (Two Hour Wargame) stable, but their "AI" which allows solo play is well worth a look. (free, in Chain Reaction). When it comes to resource management, Lords and Servants updates medieval skirmish and the "battle boards" of SAGA are a very interesting concept.
Malifaux uses cards not dice in an interesting resource management mechanic, as well as possessing excellent scenario design ideas. Dropzone Commander is pretty vanilla but is unique in how well it makes combined arms and scenario design work in concert.
I'll add to this post as I think of more, but it's late so I'm going to wrap up. Remember, these are not always the "best" rules but they do have interesting design concepts or mechanics.
Game designers can get "caught in a rut" or locked into a mindset of their own preconceptions and prior experiences. It's important to look around, and smell the roses. More specifically, other people's roses, so you can experience new ways of looking at the age-old wargaming mechanics and tropes.
I've listed rules which can "expand your horizons" and the reasons they stand out. All your games seem the same? Re-writing a "better" 40K? Why not look around and see how others have approached the problem. Sometimes, evolution is good. Other times, revolution is preferable.
Looking from a new perspective is a great way to get "out of the box."