Seems the Bush Administration doesn't want to make it any easier to track violent shifts in policy.
They've made it harder for web spiders to glean any info from the whitehouse.gov site if the page contains the word "iraq"
Quoth Dan Gilmore's e-journal site:
Sometime between April 2003 and October 2003, someone at the White House added virtually all of the directories with "Iraq" in them to its robots.txt file, meaning that search engines would no longer list those pages in results or archive them.
Perhaps the White House doesn't want to make it easy for people to compare its older statements about Iraq with current realities -- though that doesn't explain why the pages are searchable on the White House site itself. Maybe, then, the White House wants to know who's looking for these things (e.g. by tracking IP addresses of people who query the government site).
Either way, the blocking of search engines is a bad idea, and fundamentally an abuse of the public trust.
What should be done about this? I'd suggest a manual-labor cooperative, of people willing to download the daily feed from the White House, mirror it and ensure that people can search without having their IP addresses logged. If you have a better idea, post a comment.check it out at
http://weblog.siliconvalley.com/column/dangillmor/archives/001450.shtml#001450--MAB