Is the score issue to solve the issue with wars. If that is the case just give infra a military bonus modifier.
It makes no sense that a nation with six cities and 600 infra can destroy a nation with four or five cities and 2000 infra.
If you have better infrastructure it is a logical conclusion you have better equipment and such for your military because your nation is more advanced. Just my .02