1. Activation & Initiative - who moves or acts, in what order. This is the area I tend to spend the most time agonizing over. This also includes command and control, etc.
2. Game Mechanics - how we resolve the action. It includes sub categories like:
Morale is also technically a sub category but it can fit elsewhere.
My thesis here is that you should view each category or subcategory as easily replaceable and not be too strongly attached to them. Anything can be
Mechanics are interchangeable
For example, Middlehiem (my homebrew fantasy skirmish) has evolved from a modification of my simple-Infinity-ripoff modern pulp rules. At first (as there was reactions and contested rolls) I resolved interactions with d10 + stat (a bit like Warmachine's 2d6+stat but with less weird 2d6 math). I felt it was awkward, and reverted it into a Infinity-style "highest roll which is also under a stat wins" only with d10s. Finally, I ditched reactions (as they are better suited to more modern shooting-focussed games rather than medieval fantasy) and accordingly simplified it to a 40K-style "roll under a stat" again using d10s.
Besides illustrating why I like d10s (it's simple 10% increments making balance calculations easy + you can effortlessly switch between systems - just make sure the chance to succeed - conveniently in 10% hops - is the same) it shows I am not wedded to a particular mechanic. In fact, I sometimes write next to a special rule or spell something like "60% chance to work, 10% miscast chance" without specifying the mechanics or the exact methods so I can more easily port it across if I change systems.
Basically, I feel the mechanics are important merely so much as they are quick, simple and do the job with minimum brainpower or consulting of the rules. Also, they should be consistent across all the sub-categories where possible - i.e. the same dice roll method is used for morale as it is for shooting. This need for uniformity actually tends to disqualify most whacky dice resolution methods.
While I obviously find d10s convenient math-wise, I am not dogmatic about them - sometimes buckets of d6 might work better, or maybe a d100. Or maybe a range of dice from d4s to d20s. Whatever is simplest yet suits the game style of play you are aiming for. A single d20 might be too "swingy" and a bucket of d6s gives a more predictable result.
A quick rant about cards....
I'd like to deviate slightly to briefly discuss cards. Increasing number of games are including a card deck the player manages. Sure, it might add some resource management as he may discard, and save/spend cards to perform actions on the tabletop. However most of these card decks are "bolted on" - i.e. not integral to the game. Sometimes they seem to be a cynical way to sell "content" i.e. extra card packs which are just random mechanics offering gotcha moments to surprise your opponent. It may add a layer of strategy and depth, but it may equally add unnecessary complexity and randomness. Next time you see a wargame with a card deck accompanying it, ask yourself "If I removed the card deck completely from the game, would it make the game less interesting/fun? Would it remove tactics and strategy, or merely randomness?"
If the card deck is included under "advanced rules" it suggests even the developer did not view them as vital to the game.
..and now back to discuss Initiative and Activation
Once thought of as an afterthought - in many cases along with morale rules, the other stepchild - I'd view the activation as the cornerstone of the rules and even influencing the choice of mechanics. If combat resolution mechanics are what you do, activation is the when and how. I'd regard it of equal value to the combined total of all the other mechanics.
Activation is harder to change, as a shift from traditional IGOUGO (side A acts with all units, then side B acts with all units) to alternate activation (side A acts with one unit, side B acts with one unit, etc like Chess) can have a massive impact on gameplay. Furthermore, changing activation is not just a shuffling of dice mechanics which can share the same percentages but usually involves a major revamp.
Again to use Middlehiem as an example - the original game had (similar to Infinity) an activation pool (with one token per model). Models could take up to 2 tokens (actions) as long as they were non-sequential. Opponents could react to any action in line of sight. The change to opponents alternating models, each having a single action, had far reaching consequences and required nearly every rule to be re-written.
This post is probably a corollary to "dice mechanics aren't important" and "your best idea is not necessarily best for this game" but I think the topic focus is more "be prepared, and open to swapping mechanics in and out" - and as you can see from my Middlehiem example, you can even set yourself up to more easily swap them out. When you note in your draft rules that "spell has ~60% chance to cast" rather than the very specific "roll d10 + average stat of 4, to beat a target number of 10" it signals you are open (and able) to change.
The final evolution of your rules may look nothing like your original idea. You may completely swap out any or all of your mechanics and activation. You can make yourself more flexible as a designer by going in prepared to make changes (for example, giving special rules a percentage chance, rather than tying them to a particular dice roll, until your final rules draft). If you go in prepared to change, you are less likely to cling to a rule which seems cool but is perhaps not appropriate for this game.
Activation and initiative systems have a huge impact on the game and bear the most careful thought when changing things. Unlike combat mechanics like shooting where you can swap out dice types and roll methods and get the same end result, activation changes are often far reaching and comprehensive. Imagine swapping a traditional IGOUGO game to something like Infinity. The consequences (and necessary adjustments to the rules) would be far reaching.