Melee is usually done poorly in wargames, even in games where it is the primary mode of combat, like vikings or fantasy:
Push models together, roll dice, remove the loser.
Once we push the models together, our decision making is done. We just 'roll off.' In a modern combat game in an era of automatic weapons and suppressive fire, keeping it basic makes sense. But shouldn't we expect more in a medieval or gladiatorial game?
Melee has a dilemma. You have to balance speed of play, -vs- interesting decisions and mechanics. It's not sensible to have a complex melee mini-game for a game with 50+ models. If the melee resolution is too complex and time consuming, you may as well remove the models and maneuver factor all together and turn it into a boardgame or cardgame. Balance is important.
I'm going to make up some hypothetical examples, but don't focus on the precise details, but rather bear in mind the big picture - the need for speed/ease of play/quick resolution vs posing interesting risk vs reward decisions.
Example #1 = "Videogame" Complexity
Nearly all PC RPGs share a similar choice of moves. You could tie this to a resource management mechanic, but it could simply be based on modifiers. I.e. a slow power attack is -1 to hit but +1 to damage, a quick attack is +1 to hit but -1 to damage.
Quick Attack = less damage, higher chance to strike first/better 'to hit' %
Normal Attack = default
Power Attack = higher damage, slower so strikes later/lower 'to hit' %
Dodge/Roll = move to back/side short distance + % chance to dodge enemy strike (agility based)
Parry = stand ground, deflect enemy strike, maybe allow a follow-up counter attack
Kick/Bash = knock back/knock down enemy (strength based)
AoE Stomp/Spin = attack multiple foes, knock back in 180/360 arc
Finisher/Backstab = instant-kill disadvantaged enemy who has back turned/stunned/prone
These are some common examples. As a bonus, they are familiar to most gamers. There's choice but it's not excessive - you would probably memorize the attacks after a few games. LOTR:SBG (which handles skirmish of 30-50 per side) kinda falls into the simple end of this territory - there is a similar choice of special attacks, tied to the type of weapon (swords can parry, flails & whips can whirl, two handed weapons can power attack). So not every rank-and-file can access every single melee choice which both speeds things up and gives troop types 'flavour'.
This is my personal favourite level of complexity as it adds some choices and cinematic action while still being quick to resolve.
However, it's still pretty basic. We can delve into resource management and allow far more 'choice' and complexity:
Example #2: The Melee Management Mini Game
This is some sort of resource management mini-game, using ordinary dice or playing cards*. *Because while bespoke dice/rulers/special bases can be useful, it's usually a dick move based on selling extra stuff to gamers.
These are ideas I just randomly made up - they are not 'solutions', just examples to help frame the discussion.
2A ="Card Hand" - maybe each player draws cards, perhaps based on their melee skill. I.e. a Rookie gets 2 cards, Experienced gets 3, and a Veteran/Elite gets 4. Players then choose from their cards and place them down simultaneously. Maybe black cards are defensive and red cards are offensive, or some require stamina to play, or maybe you need to have a card below your relevant stat to play it.
You can see that the very nature of the 52-cards deck means this mini-game could spiral into a game of staggering complexity if you don't stay focussed. Kinda like every time two wizards have a spellcasting duel you play a complete game of Magic: The Gathering before returning to your wargame. So you'd need to exercise extreme caution to prevent a card-based mini-game from becoming excessively complex.
Merely for the sake of avoiding the rabbit hole, I probably favour:
2B = "Dice Pool" - maybe each player similarly has a pool of dice aligning with their melee ability - perhaps tossed and hidden under a cup. There's less potential for complexity to get out of hand if you just use a few d6s each. Maybe you 'bid' to perform certain moves, like in Liars Dice (the dice game they played in Pirates of the Carribean). Maybe you use combinations of dice for special attacks or play particular dice to strike first.
Note: I'd only consider the mini-game method if we had less than, say, 10 models per side - think of the potential time sink when resolving each and every melee using even the simplest card draw mechanics. A 4v4 gladiator game, yep! A mass battle game - nope!
Warning: There's a zillion variations of dice and card games, so there's a lot of rich ground to be explored - but while they can add a lot of complex decisions, maneuvers and cinematic action there is a potential time sink in complexity, where the melee mini game becomes the entire game, almost rendering terrain, models and maneuvering irrelevant.
TL:DR
Melee is usually done poorly even in wargame genres that focus on it. Too often, the only melee 'tactics'/decision is who fights who. Then the dice (RNG) takes over. There's room for simple tweaks - like the 'videogame method' to spice things up without going overboard. However, the dilemma is that all card or dice melee mechanics must be balanced against the speed of resolution. There's lots of amazing potential for card-hand and dice pools to create tactical, cinematic melee duels, at the risk of bloating the game or becoming the sole focus of the game.
