Enormity, some people insist, is improperly used to denote large size. They insist on enormousness for this meaning, and would limit enormity to the meaning “great wickedness.” Those who urge such a limitation may not recognize the subtlety with which enormity is actually used. It regularly denotes a considerable departure from the expected or normal <they awakened; they sat up; and then the enormity of their situation burst upon them. “How did the fire start?” — John Steinbeck>. When used to denote large size, either literal or figurative, it usually suggests something so large as to seem overwhelming <no intermediate zone of study. Either the enormity of the desert or the sight of a tiny flower — Paul Theroux> <the enormity of the task of teachers in slum schools — J. B. Conant> and may even be used to suggest both great size and deviation from morality <the enormity of existing stockpiles of atomic weapons — New Republic>. It can also emphasize the momentousness of what has happened <the sombre enormity of the Russian Revolution — George Steiner> or of its consequences <perceived as no one in the family could the enormity of the misfortune — E. L. Doctorow>.
http://www.merriam-webster.com/dictionary/enormityAs does Random House:
—Usage note
3. Enormity has been in frequent and continuous use in the sense “immensity” since the 18th century: The enormity of the task was overwhelming. Some hold that enormousness is the correct word in that sense and that enormity can only mean “outrageousness” or “atrociousness”: The enormity of his offenses appalled the public. Enormity occurs regularly in edited writing with the meanings both of great size and of outrageous or horrifying character, behavior, etc. Many people, however, continue to regard enormity in the sense of great size as nonstandard.
cite: Dictionary.com. Dictionary.com Unabridged. Random House, Inc.
http://dictionary.reference.com/browse/enormity (accessed: March 21, 2010).
American Heritage
agrees with you:
Usage Note: Enormity is frequently used to refer simply to the property of being great in size or extent, but many would prefer that enormousness (or a synonym such as immensity) be used for this general sense and that enormity be limited to situations that demand a negative moral judgment, as in Not until the war ended and journalists were able to enter Cambodia did the world really become aware of the enormity of Pol Pot's oppression. Fifty-nine percent of the Usage Panel rejects the use of enormity as a synonym for immensity in the sentence At that point the engineers sat down to design an entirely new viaduct, apparently undaunted by the enormity of their task. This distinction between enormity and enormousness has not always existed historically, but nowadays many observe it. Writers who ignore the distinction, as in the enormity of the President's election victory or the enormity of her inheritance, may find that their words have cast unintended aspersions or evoked unexpected laughter.
cite: Dictionary.com. The American Heritage® Dictionary of the English Language, Fourth Edition. Houghton Mifflin Company, 2004.
http://dictionary.reference.com/browse/enormity (accessed: March 21, 2010).