|
|
|
|
Location: Blogs SONNY'S INFINTE WELL OF DEVELOPER HAPPINESS |
 |
| Posted by: Sonny Wilson |
Tuesday, October 17, 2006 |
What is the correct filename to use to exclude folders and files on your website from being accessed by search engine crawlers? Is it Robots.txt or Robot.txt?
The correct answer is 'Robots.txt'. Don't use 'robot.txt'.
The original document titled "A Standard for Robot Exclusion" represents a consensus on 30 June 1994 on the robots mailing list... To learn more, click here
Also, as a side note, you can see this rule in action by looking at the Whitehouse website. You can see that it has a robots.txt file, and doesn’t have a robot.txt file.
To open robots.txt, click here To open robot.txt, click here
|
| Copyright ©2007 Sonny Wilson |
| Permalink |
Trackback |
|
|
|
|
|
|