White House blocks search engines - Iraq content specifically

[DBoon]

As "reported" in Slashdot:

http://yro.slashdot.org/comments.pl?sid=83767&threshold=0&commentsort=0&tid=103&tid=126&tid=95&tid=99&mode=thread&cid=7322178

Based on:

http://www.bway.net/~keith/whrobots/

It appears the White House public Web site is using a "robots.txt" to stop search engines from cataloging much of its content. Apparently, much of the off-limits content has to do with Iraq. The suspicion is that the White House does not want search engines or other archival directories to keep a history of its statements on Iraq.

[Rooktoven]

1. The referenced link:

http://weblog.siliconvalley.com/column/dangillmor/archives/001450.shtml

beat me to the punch there, DB...

[RobertSeattle]

2. Wow - check it out!!!

(I'm not a Robot.txt guru)

http://www.whitehouse.gov/robots.txt

Sample:
Disallow: /911/911day/iraq
Disallow: /911/911day/text
Disallow: /911/heroes/iraq
Disallow: /911/heroes/text
Disallow: /911/iraq
Disallow: /911/patriotism/iraq
Disallow: /911/patriotism/text
Disallow: /911/patriotism2/iraq
Disallow: /911/patriotism2/text
Disallow: /911/progress/iraq
Disallow: /911/progress/text
Disallow: /911/remembrance/iraq
Disallow: /911/remembrance/text
Disallow: /911/response/iraq
Disallow: /911/response/text
Disallow: /911/sept112002/iraq
Disallow: /911/sept112002/text
Disallow: /911/text

[htuttle]

3. Guess I'm going to have to (ht)dig there myself

I'm sure the crack WhiteHouse I.T. staff is aware of it, but the 'robots.txt' file is more 'advisory' than anything else. It doesn't actually 'prevent' indexing, but most of the major search engines honor it.

Since they went to the trouble, I'll have to make sure to add the whole whitehouse site to my local htdig database!

[LiberalFighter]

4. Please explain

What does that involve.

I was thinking that if nothing else just download the whole site.

The major search engines should ignore the robot.txt of public government websites.

[htuttle]

7. htDig is a webcrawler search/indexing engine

Very much like what google or Altavista uses underneath (although both of theirs are custom). (I think bpilgrim has it running at http://globalfreepress.com for searching various things, for example). The nice thing about it is that you just point it at a URL, and it will index the whole site, even a remote site).

The website is here:

http://www.htdig.org/

To get it to ignore the robots.txt file, you need to hack it yourself, since it's considered 'impolite' to not honor the robots.txt file.

That being said, to hack it, merely search through the code that looks for the 'robots.txt' file, and change it to something else, like 'norobots.txt', and recompile (I made that sound easy, didn't I).

Note, this will cause your htdig engine to ignore ALL robots.txt files for all sites.

(obviously the htDig people do not recommend this, since it's rude, and goes against the 'Standards for Robot Exclusion', but hey, this is war...).

Finally, if all of this was unintelligible to you, be aware that it is a fairly technical process to set it all up. It would definitely be easier to just mirror the whole site locally, and search at will in that case.

[htuttle]

8. And for those of you playing along at home...

Comment out Line 76 in htdig/Server.cc

From this:

robotstxt(doc);
break;

to this:
// robotstxt(doc);
break;


That should do it.

I'll post a link if I get it indexed and up on my website. The modified version just finished linking, but it will take a few hours to index the whole WH site.

[htuttle]

9. I'm an idiot...

I always do things the hard way.

:eyes:

Instead of building a custom htdig, I just realized that I could simply set this htdig.conf variable, to make the robot look for a different 'robot name'.

Changing this directive in the config file:
robotstxt_name: htdig

to this (or adding it if it's not already there):
robotstxt_name: whsearch

will cause it to use the same, almost non-existant exclude list that the white house's own search engine uses (and includes all the documents), as opposed to the huge, paranoid exclude list mentioned above.

The idea is that your htdig would then appear to be the whitehouse's own search engine, and wouldn't exclude any pages from your index.

Oh well, the custom one I built is already well into indexing...I'm going to let it run and see what I get.

[denverbill]

6. Yes, please do.

And let us know what you find.

Bush must've made a big speech last year in his big buildup to war, back when he still 'hadn't decided' to go to war (yuck, yuck).

[JohnOneillsMemory]

5. Invisible tech wars abound. I know lots of folx whose comp's have slowed.

Whether it is from viruses or spyware, I don't know. But many people I know here in the North Bay area are having slowed web-page problems, both loading and navigating. Being naturally paranoid, we wonder if there aren't regional attacks and roadblocks put on the internet as well.

[Generic Other]

10. A simpleton's analysis

The White House takes a speech we all hear on tv full of lies, evasiveness and pure bull all delivered in the president's halting, tongue twisted syntactical style. An aide quickly cleans up the script, repairs the prose and posts the lies online in an "edited" form which makes it sound as if the original words made sense. At a later date, someone else restricts access to the laundered lies which have since formed the foundation of more lies that have only multiplied compounding the original deceit. All of this to hide the trail of a simpleminded fool.

So what are they afraid of? The Boogie man?

[supercrash]

11. Hehe

Sounds like a very efficiant Ministry of Truth