which training should i invest in?

Wait until you meet the super-difficult missions that even players with top generals struggle to complete.

Seems like there are multiple topics being discussed here.

Training
I think that ET game is offense oriented. You can see that most of the time you are tasked to complete the objectives within a limit amount of rounds. So mobility and attack stat are preferred. IMO, I prioritized movement over everything (esp. when that additional movement point allows your main troop to move one extra hex) then attack > HP > def. Def stat has always been pretty weak in ET game (except when you really have a ton of it. e.g. gen on armored car. But you get only 2-4 def from training so it's kinda negligible).

Alert VS Riding Master
As mentioned earlier, Alert reduces damage from units with range attack, not attack from 2+ hex range. That means you always get the -25% damage from the skill (even though you attack artillery from melee). So its benefit is solid. So it is all up to whether you want it or not. Riding Master is mathematically weak and the fact that it is not reliable makes the even less useful. BTW, the term 'mathematically' implies that this is not an opinion. Still, the skill adds a little bit of unreliable damage (e.g. less than 3.5% on average). So it's all up to you whether you want a bit of extra damage or additional defense against artillery, ship, and forts. One more thing to consider, Alert stacks multiplicatively with terrain defense. Since Plumer has jungle fighting, he can be a good anti-artillery general.

Surprise Attack VS Tactic Master
Surprise Attack multiplies 1.08 to 1.20 to your output. Tactic Master gives 20% chance to attack again. So Tactic Master adds more damage (unless you can always move 5 hex for surprise attack) but do not forget that it has some drawbacks. Attacking twice means you will be counterattack twice. You may argue that you won't get counterattack if you kill the target but that is what you cannot always do, especially later in the game. Also it is less reliable; you either get double damage or nothing. So that is relatively hard to make sure that kill the unit compared to having extra flat damage. Anyway, both skills are great. I just want to point out that one is not always better than another.

Mainstream VS Unique generals
That's a matter of preference. If you are goal oriented and want to finish this game as fast and easy as possible then using the strongest generals and turning them into Ludendorff's clones (with a bit of variations) might be you way to go. But some players just enjoy this game from different aspects, e.g. using their fav historical figures, having badass-look gens the team, or deploying highly specialized gens in a specific situation. At least that gives the feeling of uniqueness and identity. That's why you can see sub-optimal choices everywhere; Moltke, Mackensen, Lee, Vorbeck, etc. Some player even make infantry oriented team even though generals on infantry are not exactly strong. I also have Leopold and Joffre on my main team.

Is it a good idea to do defense over HP on cavalry? Since they really have massive base def.

I would say HP is better under basic conditions

For example:

LV6 Cav Defense is +24
LV6 Cav HP is +105


Enemy Unit is fictive with 100 Attack and 100% against all Armor Types
Allied Unit is Cossack with basicly 10 def and 230 HP

Case 1 (basic):

230/((100*100)/(100+10))
The Attacker needs ~2,5 Hits to wipe out our Unit

Case 2 (LV6 Def):

230/((100*100)/(100+34))
The Attacker needs ~3 Hits to wipe out our Unit

Case 3 (LV6 HP):

335/((100*100)/(100+10))
The Attacker needs ~3,7 Hits to wipe out our Unit


But it is all without some special skill's like Pierce, Edge etc.

I am happy with that. 
With the resources and general slots available in this game, I can create two almost identical generals with and without desert fighting. So that basically means I have an addition option to choose. But to be honest, I mainly want the movement part of the skill.

i know enemy's tech scales better for conquest challenges so doesn't it make sense to just max attack and mobility and keep defence and health level 1? mobility doesn't give a huge advantage to the enemy but its super useful for the player. also attack is important for artillery - artillery is free damage without getting hit back and also important for cav to be able to 1 shot an enemy unit and not be attacked back. keeping defence and health at lvl 1 means enemy should be easier to kill because they will have lower health + defence. you will also have lower health but that doesnt matter if you can clear enemy with cav - won't get hit back