FREE ONLINE WEBSITE TOOL

Robots.txt Checker

Review robots.txt directives, check user-agent groups, and identify common configuration issues before publishing your file.

Free to use No signup Local browser analysis No content uploaded

STEP 1

Enter your robots.txt content

Paste your robots.txt content or upload a local TXT file.

Maximum 1 MB
0 bytes · 1 MB maximum No file selected

Paste your robots.txt content or upload a TXT file to get started.

How to check a robots.txt file

  1. Open the robots.txt file you want to review.
  2. Copy its text or save it as a TXT file.
  3. Paste the content into the input area or select Upload TXT.
  4. Select Check robots.txt to review the parsed directives and findings.
  5. Review the warnings and download the cleaned text if needed.

The supplied content is analyzed locally in your browser. This tool does not retrieve a live robots.txt file from a website.

What does robots.txt control?

A robots.txt file contains instructions that participating crawlers can use when deciding which URL paths they should request from a website.

Rules such as User-agent, Allow, and Disallow are interpreted according to crawler-specific behavior. Different crawlers may support different directives.

Understanding the results

  • Error: A directive or structure that appears malformed or cannot be interpreted by this checker.
  • Warning: A configuration that may need review, such as a missing user-agent declaration or repeated directive.
  • Info: Additional context about recognized directives or crawler-specific extensions.

These are practical checks, not a complete implementation of every search engine's robots.txt parser.

Important limitations

Robots.txt is not an access-control or security mechanism. It does not prevent people or malicious crawlers from accessing public URLs, and it does not protect private files.

Do not use robots.txt as a substitute for authentication, authorization, or server-side access restrictions.

Frequently asked questions

Does this tool fetch a live robots.txt file?

No. This version analyzes text that you paste or upload. It does not fetch a website, make network requests, or use a third-party proxy. To check a live file, retrieve it yourself and provide its contents.

Is my robots.txt content uploaded to a server?

No. The supplied text is processed locally in your browser. It is not sent to Baiming Service or an external analysis service.

Which directives does the checker recognize?

The checker reviews User-agent, Allow, Disallow, Sitemap, Crawl-delay, and Host declarations. Crawl-delay and Host are treated as crawler-specific or non-standard extensions rather than universally supported directives.

Does a valid result guarantee that every crawler will follow the rules?

No. Crawler behavior and directive support can differ. This tool performs practical syntax and configuration checks; it does not reproduce every crawler's complete parsing behavior.

Can robots.txt protect private pages or files?

No. Robots.txt is a crawler instruction file, not a security control. Use authentication and server-side access restrictions to protect private resources.

What does Disallow: / mean?

It is a broad rule that asks matching crawlers not to crawl paths on the site. Review the intended scope carefully before publishing such a rule.

Can I download the checked robots.txt file?

Yes. After analysis, use Download robots.txt to save the normalized text as a local file. Review the content before publishing it to your website.