Hey, it's more dangerous than dihydrogen monoxide, and that stuff kills thousands of people each year despite is relatively low toxicity.
Fun aside, ingesting NH4NO3 isn't nearly as dangerous as inhaling it, and that's why they were worried about it, at least some of it was pushed into the atmosphere, and you don't have to inhale a lot of it for symptoms to become worrisome, though you would have to inhale a bit before it killed you.
But it is relatively low on the toxicity scale. It's just that if inhaled it attacked your blood cells directly, prevents you from receiving oxygen, and the wrong treatment can worsen symptoms.
Incidentally it also turns your blood into the color of chocolate brown.[DOUBLEPOST=1366820405][/DOUBLEPOST]I actually think the real reason they were worried about it is that it is such a strong oxidizer that it will cause things that are normally considered nonflammable to burn. Usually nonflammable products contain chemicals (and in some cases heavy metals) that, if burned, produce very toxic gasses.
It's unlikely that they were as worried about direct contact with ammonium nitrate as they were about the combustion by products of the stuff that burned because of it. It doesn't evaporate into air, and settles out of the atmosphere readily, but the gasses and toxic by products of the other burned materials nearby can travel for miles and have significant effects when inhaled or come into contact with other organic materials, such as skin, mucus membranes, etc.