Creative Commons image by Patrick Hoesly via Flickr

Editors’ Choice: Robots.txt

The Internet Archive does some amazing work in the Sisyphean task of archiving the web. Of course the web is just too big and changes too often for them to archive it all. But Internet Archive’s crawling of the web and serving it up out of their Wayback Machine, plus their collaboration with librarians and archivists around the world make it a truly public service if there ever was one. Recently they announced that they are making (or thinking of making) a significant change to the way they archive the web…

Read full post here.