The First Rule: Many Difficult Choices, Few Easy Rules
Games rules should be simple and easy to remember, whilst presenting the maximum number of difficult choices at every opportunity. A shining example is Mayhem, which at every opportunity you have to make a dice roll, you can either automatically claim half the dice (say 3 on a d6, 4 on a d8) or roll the dice and take a risk. Not a hard rule to remember - but it constantly requires a player decision.
Statlines vs Special Rules
At lot of current games have minimized stats. For example, Song of Blades has "Combat" and "Troop Quality" as its only "stats." It then differentiates between units by adding a zillion traits or special abilities - exceptions to the rule. It's false economy - more rules are needed because of the smaller stat line. Oversimplifying has actually added complexity.
Whilst a game of modern combat (where all soldiers are human and most carry 5.56mm automatic weapons) allows generalisations, fantasy - which can encompass lizardmen, ogres, hobbits, dragons, goblins and gargoyles - tends to have very different troop types. Having a sensible stat line keeps the "extra rules" managable.
I find LOTR: SBG surprisingly underrated - perhaps tooInitative - the Ignored Factor
subtle boring for the average GW powergamer
IGOUGO doesn't cut it anymore. Games should involve players. Troops should not just wait like dummies to be scythed down. A lot of games ignore the initiative sequence and miss out on an opportunity to present players with tactical choices. An interesting option - the card activation of TFL games - where activation is random, but better leaders have more/better opportunities to act; and troops without inspiring leaders tend to act last. Or Song of Blades and Heroes "risk vs reward" activation where a player can roll dice to take more actions but risks failing and losing the initiative.
Again, I mention LOTR: it has a modified IGOUGO sequence that heroes can "interrupt" to seize the initiative. This not only makes the move sequence more tactical, but makes heroes into actual leaders, not simply walking tanks.
The "all units move the same distance" lead by 40K just doesn't cut it. A dwarf does not move as fast as a werewolf. Yes, you can say it represents them "advancing cautiously at combat speed, weapons ready" but this is another unneeded simplification (most players can remember how far a unit moves) which removes flavour.
Choices in Combat
Combat in 90% of skirmish games is just move up, roll dice. There isn't much decision making in it. Something that has interested me a lot lately is the concept of "Dice Pools". Bushido has a dice pool, (free rules here) where players can allocate melee dice to attack or defence, as does Havoc. CROM takes the dice-as-a-resource concept further allowing you to "burn dice" - losing dice (ability) permanently for short-term gain. Alkemy uses cards for attack types - certain attacks counter others more effectively, others do more damage, etc. Even LOTR allows a choice of one handed attacks, two handed attacks, and defensive posture.
Song of Blades did a lot of things right in making fantasy skirmish fun - but the biggest strength is its unit builder allowing the use of random cool models to create a custom warband...
Results of Combat
I liked Song of Blades - losers were either knocked down (tip model over); pushed back, or dead (removed). Plenty of cinematic moments, but no record keeping, or even counters to clutter the table, or (God forbid) hitpoints to tick off a unit card a la Warmachine or Malifaux.
Sensible, Interesting Magic
Magic should steer between the two spells of Song and Blades (freeze, and missile attack) and the fireballs of doom-that-decimate-a-regiment of some Warhammer fantasy editions. There should be a sensible range of spells to allow most generic magic attacks as found in movies, TV shows and books, but it also should be limited in power so the game does not simply become a wizard duel. Making magic a finite resource might add a layer of tactics as well. I quite like how the warcasters in Warmachine are used although they are a poor example as far as the "don't min-max or overpower stuff" are concerned.
Heroic Moves, Cut Scenes
Rather than tank-like demigods with 3 wounds, 10 Defence and 7 Strength 9 attacks, I like games where heroes can influence events around him (again, allowing player choice) rather than decimate a entire regiment solo by virtue of superior stats. In Havoc, when initiative rolls were tied, "named" heroes could act while the rest of the fight was paused around them - just like a Hollywood cut scene.
GW's much maligned LOTR was also good this way - heroes could spend a finite amount of "Heroic Actions" to move friendly units out of sequence, get in the first blow etc. This was also a "Resource" to manage as they could also use these heroic actions to re-roll their personal combat results. Basically, heroes should be heroes because they have special heroic abilities. Not simply because they are unkillable walking tanks with impossibly high stats, who can absorb two fatal wounds and fight on...
The command point system of diminishing returns in Mayhem (which reminds me of clicking in a RTS); the order pool in Infinity, your hand of cards in Malifaux.... all these add a layer of strategy and tactics. If done in a way that does not add excessive complication or record-keeping, this can add a final layer of depth to the game.
There are a lot of people waiting for "the next Mordhiem" (or still playing the old one, 20 years on) - showing the value people attach to campaign-based play
Obviously to be avoided if you are trying to tie the rules to a specific product line, this does make a ruleset more versatile and instantly shoots it to the top of my "buy and try" list. Song of Blades is a great example - it has simple, fun gameplay, but it can be a little shallow - it's real depth is the ability to "stat up" random models and create your own custom warbands using a free Excel spreadsheet. Providing a good reason to paint up old models or cruise the net looking for cool random minis, the unit builder is definitely "value added" to any rules set.
Death match "to the last man standing" rules get old after a while. Scenario books and supplements can address this partially, but there is a big crowd of people who think nostalgically of Mordhiem. Hands up if you wish there was a modern replacement for it. Yup, thought so. Exasperatingly unbalanced at times, the campaigns with random battles kept things from getting "stale" and developed a sense of narrative to games. Having heroes "level up" and gain new abilities (or be crippled by wounds) creates a genuine attachment to your force. You know, that badly painted dwarf model you hate the look of, that just would never die, and has killed two enemy champions? Now you're scared to repaint it in case you break it's lucky run.... Campaign rules add a completely new layer of depth to any skirmish game, and allow it to attract a wider range of gamers.
Random Thought: The Lonely d10
I notice games are increasingly using numbers of d6 to "flatten" the probability curve, often with attack dice cancelling out defence dice or similar. The other day I saw a video of an ex-GW games designer spruiking a new Kickstarter game (it's here somewhere) and he was waxing eloquent about how the game will be awesome "cos it used a ten sided dice." I laughed at that - no wonder GW is remaking the same game, 25 years on... they only just discovered d10? The 80s just called - they want their leg warmers back... Why do game designers stick with d6s? I've always thought a d10 had many inherent advantages... it's not like they are the sole preserve of D&D basement-dwellers - I bought my last set at the local ToysRUs.
I don't think my "ideal" fantasy game exists yet, though many have strong traits that recommend them. Song of Blades has good initiative, combat results and a stellar unit builder; but it's two-stat/million special rules makes it shallower than it needs. Havoc scores high in gameplay but its limited troop types, lack of unit builder and campaign holds it back. LOTR scores surprisingly well as an all-rounder, but has trouble shaking a feel of all-round "blandness." Mordhiem has a strong (albeit unbalanced) campaign element, but has dated IGOUGO gameplay.
Come across any games that meet (most) of the criteria? I'd love to hear about them.