Robots.txt files (often erroneously called robot.txt, in singular) are created by webmasters to mark (disallow) files and directories of a web site that search engine spiders (and other types of robots) should not access. This robots.txt checker is a "validator" that analyzes the syntax of a robots.txt file to see if its format is valid as established by Robot Exclusion Standard (please read the documentation and the tutorial to learn the basics) or if it contains errors. This has been added to Accessibility Resources Subject Tracer™ Information Blog. This has been added to Bot Research Subject Tracer™ Information Blog.
posted by Marcus Zillman |
4:33 AM
