I don't disagree with your main point, but this doesn't seem to be quite correct. Going by wikipedia and its linked sources, the XM8 substantially outperformed the M4, but the battery life for the built-in, non-modular, sight tested very poorly, and it was just uncomfortable to hold. While H&K was working on it, Congress and the other arms makers got the Army to suspend, and then cancel, the project because the project under which the XM8 was being developed (the OICW) had changed its goals substantially since its inception.
So it was canceled because unforeseen R&D needs, budget cuts and politicking, not the performance of the actual weapon.