This is an RSS file. It is intended to be read by a software program called a "feed reader". Search on Google for more details. <![CDATA[whitters's Feed]]> http://www.simpy.com/user/whitters http://www.simpy.com/ http://www.simpy.com/img/simpy-icon-16x16.png Simpy http://www.simpy.com/user/whitters 1440 <![CDATA[Nikita The Spider - A Robot Exclusion Rules Parser for Python]]> Tagged by whitters under robot exclusion rules, robots, robots.txt, protocol, crawlers, ]]> Mon, 02 Nov 2009 04:46:00 -0500 <![CDATA[robotstxt.org]]> Tagged by whitters under robots.txt, spiders, crawlers, ]]> Mon, 26 Oct 2009 06:15:00 -0400 <![CDATA[Robots exclusion standard - Wikipedia, the free encyclopedia]]> Tagged by whitters under robot exclusion rules, robots.txt, robots, crawlers, standards, ]]> Sat, 04 Jul 2009 09:17:00 -0400 <![CDATA[ACM Queue - Why Writing Your Own Search Engine is Hard - So you have a grand idea; are you ready for the execution?]]> Tagged by whitters under webcrawler, crawler, scalable, ]]> Mon, 12 Jun 2006 10:31:00 -0400 <![CDATA[Become.com's Web Crawler: A Massively Scaled Java Technology Application]]> Tagged by whitters under webcrawler, crawler, java, ]]> Mon, 12 Jun 2006 10:30:00 -0400 <![CDATA[Open Source Crawlers in Java]]> Tagged by whitters under crawler, webcrawler, java, opensource, ]]> Fri, 09 Jun 2006 08:33:00 -0400