links · people · groups · tags | My: links · tags · groups · watchlists · notes login · sign up now! | help · blog
Simpy simpy
 
Home / people / whitters / links / (standards) AND "crawlers"
David Whitmore, member since Jun 9, 2006
.
Search Everyone: (standards) and "crawlers",
Show 50 · 100 · all  
1 - 1 of 1   Watch whitters
 
Copy this Robots exclusion standard - Wikipedia, the free encyclopedia
by whitters 2009-07-04 09:17 robot exclusion rules · robots.txt · robots · crawlers · standards
http://en.wikipedia.org/wiki/Robots_exclusion_standard - cached - mail it - history
1 - 1 of 1  
Related Tags
 
- exclude ~ optional + require
-- ~ + 1 crawlers
-- ~ + 1 robot exclusion rules
-- ~ + 1 robots
-- ~ + 1 robots.txt
-- ~ + 1 standards
Add Dates
Dec 17, 2009
Dec 15, 2009
Dec 4, 2009
Nov 28, 2009
Nov 24, 2009
Nov 15, 2009
Nov 12, 2009
sign up now! · FAQ · about · advertising · blog · Contact · Privacy Policy · Terms & Conditions · © Copyright 2004-2008 Simpy. All Rights Reserved.
Note: If you can read this text, it means that you are not experiencing Simpy at its best. Simpy makes heavy use of Extensible Hypertext Markup Language (XHTML) and Cascading Style Sheets (CSS), which means that a modern, standards-compliant browser is needed for using Simpy. Please see how to get a better browser.