What is robots.txt?

robots.txt is a plain-text file at the root of a domain that tells web crawlers which paths they can or cannot fetch. It's used to keep crawlers out of admin/auth/dashboard paths, block SEO-competitor crawlers from scraping your site, and make explicit allow/block decisions for AI bots like GPTBot and ClaudeBot — but it does NOT control whether a page can be indexed, only whether it can be crawled.

No — it prevents crawling. A blocked URL can still appear in search results if Google discovers the URL through external links (indexed without content). To remove from the index, use a noindex meta tag or HTTP header — and don't block the URL in robots.txt, because Google needs to crawl the page to see the noindex directive.

You create the exact trap the previous answer warns about, in reverse: if you Disallow a URL in robots.txt AND put a noindex tag on that page, Google can never crawl the page to see the noindex directive — so instead of being removed from the index, the URL can sit indexed with no content shown, indefinitely. The correct order is: let the page be crawled with noindex in place, wait for Google to process it and drop it from the index, THEN block it in robots.txt if you want to stop future crawl attempts.

Disallow tells crawlers not to fetch matching paths. Allow overrides a Disallow for a more specific path (e.g., Disallow: /admin/ + Allow: /admin/public/). Most engines support both; legacy bots may only honor Disallow.

Depends on your business model. Marketing-led companies should allow GPTBot, ClaudeBot, PerplexityBot, and Google-Extended — being cited by AI engines is a top-of-funnel win. Content-licensing companies (publishers, paid courses) often block them to protect IP. SearchChamp defaults to allow on the marketing surface.

Google Search Console has a robots.txt tester. For each User-Agent, paste a URL and see whether it's blocked or allowed. SearchChamp's Site Auditor parses your robots.txt and flags syntax errors, conflicts, and blocked critical paths (e.g., accidentally blocking /sitemap.xml).

Fetch it directly — curl https://example.com/robots.txt or just visit the URL in a browser, since it's a plain-text file with no authentication. Read the rules manually: find the block matching the crawler's User-Agent (or the wildcard * block if there's no exact match) and check whether your target path falls under a Disallow rule. It's tedious for a large file with many User-Agent blocks, which is why an automated parser is worth it past a handful of rules.

No — it's a voluntary convention, not an enforcement mechanism. Reputable crawlers (Googlebot, Bingbot, the major AI bots) honor it by policy, but robots.txt has no technical way to block a crawler that chooses to ignore it. If you need to actually prevent access rather than politely request it, that requires server-side blocking (IP allowlisting, authentication, a firewall rule) — robots.txt alone won't stop a bad actor.

Yes, trivially — since it's a voluntary, unenforced convention (see the previous answer), any scraper that doesn't care about following the rules simply ignores the file. robots.txt is a signal for well-behaved crawlers, not a security control. Don't rely on it to protect sensitive content; use authentication or server-side access rules for anything that actually needs to be inaccessible.

At the root: example.com/robots.txt. Not /sub/robots.txt. Not subdomain.example.com/robots.txt unless that subdomain is a separate hostname (then each subdomain needs its own robots.txt).

Indirectly, yes, in one specific way: because robots.txt is a public, unauthenticated file, listing a sensitive path in a Disallow rule to "hide" it from crawlers actually advertises that path's existence to anyone who reads the file. Don't use robots.txt as an obscurity measure for admin panels or staging environments — those need real authentication, not a polite request not to crawl them.

Yes — Site Audit parses your robots.txt for syntax errors, rule conflicts, and accidentally blocked critical paths across all User-Agent blocks, including the AI bot entries (GPTBot, ClaudeBot, PerplexityBot, Google-Extended), not just the legacy Googlebot/Bingbot rules.