How about applying the modifiers to amount of soldiers killed rather than the amount that survive?
Basically, if A (has 8k troops)attacks nation B (0 troops). This usually results in a loss of 3 soldiers to the attacker. So A would get modifiers applied to 7997 soldiers. This would be equal to not fighting at all but still recieving the bonus which is illogical.
In the long run, a nation constantly farming the same inactive nations would overpower the nations who have been neutral all along.