FREE ONLINE WEBSITE TOOL
Robots.txt Checker
Review robots.txt directives, check user-agent groups, and identify common configuration issues before publishing your file.
STEP 1
Enter your robots.txt content
Paste your robots.txt content or upload a local TXT file.
STEP 2
Analysis report
Analysis results for your robots.txt content.
Directive overview
Parsed directives and their original line numbers.
| Line | Directive | Value | Group |
|---|---|---|---|
| No directives to display. | |||
User-agent groups
Allow and Disallow rules grouped by user-agent declarations.
No groups to display.
Sitemap declarations
Sitemap directives found in the supplied content.
No sitemap declarations found.
Issues & recommendations
Review potential syntax problems and configuration warnings.
No issues to display.
Cleaned content
A normalized text copy with surrounding whitespace removed. Review before using it on your website.
How to check a robots.txt file
- Open the robots.txt file you want to review.
- Copy its text or save it as a TXT file.
- Paste the content into the input area or select Upload TXT.
- Select Check robots.txt to review the parsed directives and findings.
- Review the warnings and download the cleaned text if needed.
The supplied content is analyzed locally in your browser. This tool does not retrieve a live robots.txt file from a website.
What does robots.txt control?
A robots.txt file contains instructions that participating crawlers can use when deciding which URL paths they should request from a website.
Rules such as User-agent, Allow, and Disallow are interpreted according to crawler-specific behavior. Different crawlers may support different directives.
Understanding the results
- Error: A directive or structure that appears malformed or cannot be interpreted by this checker.
- Warning: A configuration that may need review, such as a missing user-agent declaration or repeated directive.
- Info: Additional context about recognized directives or crawler-specific extensions.
These are practical checks, not a complete implementation of every search engine's robots.txt parser.
Important limitations
Robots.txt is not an access-control or security mechanism. It does not prevent people or malicious crawlers from accessing public URLs, and it does not protect private files.
Do not use robots.txt as a substitute for authentication, authorization, or server-side access restrictions.
Frequently asked questions
Does this tool fetch a live robots.txt file?
No. This version analyzes text that you paste or upload. It does not fetch a website, make network requests, or use a third-party proxy. To check a live file, retrieve it yourself and provide its contents.
Is my robots.txt content uploaded to a server?
No. The supplied text is processed locally in your browser. It is not sent to Baiming Service or an external analysis service.
Which directives does the checker recognize?
The checker reviews User-agent, Allow, Disallow, Sitemap, Crawl-delay, and Host declarations. Crawl-delay and Host are treated as crawler-specific or non-standard extensions rather than universally supported directives.
Does a valid result guarantee that every crawler will follow the rules?
No. Crawler behavior and directive support can differ. This tool performs practical syntax and configuration checks; it does not reproduce every crawler's complete parsing behavior.
Can robots.txt protect private pages or files?
No. Robots.txt is a crawler instruction file, not a security control. Use authentication and server-side access restrictions to protect private resources.
What does Disallow: / mean?
It is a broad rule that asks matching crawlers not to crawl paths on the site. Review the intended scope carefully before publishing such a rule.
Can I download the checked robots.txt file?
Yes. After analysis, use Download robots.txt to save the normalized text as a local file. Review the content before publishing it to your website.