Find Out If ChatGPT, Claude, Perplexity and Google’s AI Can Find and Cite You
Enter your website and this free AI crawler checker reads your robots.txt, your llms.txt and your homepage tags, then tells you which AI bots are allowed or blocked: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more. It shows the exact line that decides each result and includes a robots.txt AI bots rule builder and an llms.txt generator.
AI Search + robots.txtFree Checker Tool
OfficialStape PartnerEnter your website. Add a page path if you want to test one page or folder as well as the homepage.
The address you enter is sent to our server, which loads your robots.txt, llms.txt and homepage once each. We keep a simple log of the address checked and the result. No personal details are saved unless you ask for the report by email.
Get the full Excel report with every bot and the suggested rules in your inbox.
I set up tracking that shows visits and leads from ChatGPT, Perplexity, Gemini and Copilot in GA4, next to your ad and search traffic.
Three public files, read once, then tested against the rules crawlers actually follow.
OAI-SearchBot, Claude-SearchBot, PerplexityBot and others that let AI products find and cite your pages.
GPTBot, ClaudeBot, Google-Extended, CCBot and others that collect pages for model training.
ChatGPT-User, Claude-User and Perplexity-User, which open a page when a person asks about it.
Every group and rule, with the line that decides the answer for each bot.
Test any path, not only the homepage, including * and $ patterns.
Meta robots and X-Robots-Tag: noindex, nosnippet, max-snippet and noai.
Whether the file exists and follows the proposed format: title, summary and link sections.
A robots.txt rule builder with four policies and an llms.txt generator filled from your homepage.
Three steps, usually a few seconds.
Add a page path too if you want to test a specific page or folder.
Your robots.txt is parsed the way crawlers parse it, then each of the 26 bots is matched against it.
Pick a policy and copy the lines to add to robots.txt. Generate an llms.txt if you want one.
An AI crawler is a program that reads web pages for an AI company. Some collect pages to train models. Others build a search index so an assistant can find, quote and link to your page. A third kind fetches one page at the moment a person asks about it. Each has its own name, called a user-agent, and your robots.txt file can give each name different rules.
This checker follows the published robots.txt standard (RFC 9309) and Google’s documentation:
User-agent: * group.* matches any text and $ marks the end of the address.Disallow: allows everything. Unknown lines are ignored.llms.txt is a proposed Markdown file at /llms.txt that lists your most useful pages for AI tools. The proposal asks for one H1 title, a short summary in a blockquote, then H2 sections with links. It is a proposed convention, not a standard, and the large AI companies have not committed to reading it. It is cheap to add, so the tool includes an llms.txt generator, but do not expect it to change rankings.
The main user-agent names, from each company’s own documentation.
| Company | Trains models | Powers AI search | Fetches for a user |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | – | PerplexityBot | Perplexity-User |
| Google-Extended (control name) | Googlebot | – | |
| Apple | Applebot-Extended (control name) | Applebot | – |
| Meta | meta-externalagent | meta-webindexer | meta-externalfetcher |
| Amazon | Amazonbot | Amzn-SearchBot | Amzn-User |
| Mistral | MistralAI-Training | MistralAI-Index | MistralAI-User |
| Common Crawl | CCBot (open archive) | – | – |
See in plain words whether AI assistants can read your site.
Audit AI access and catch copied block lists that hide a site from AI search.
Allow citation while opting out of model training.
Test a path against wildcard rules before you ship a robots.txt change.
33 free tools for tracking, analytics, advertising and SEO. No login needed.
Preview how a page looks on Google, Facebook, LinkedIn and X, and fix its Open Graph tags.
Open the preview checker →See a Shopify store’s theme, apps, catalogue size, tracking and speed.
Open the Shopify checker →Check the DNS records that decide whether your marketing emails reach the inbox.
Open the email checker →Find out which CMS, theme, plugins and technology any website is built with.
Open the CMS checker →Hear from our clients. Loved by 600+ businesses worldwide.
“I had an excellent experience working with MD on my Google Ads account. Everything was set up perfectly and worked smoothly without any issues or troubleshooting needed. His expertise and attention to detail made the whole process effortless.”
“I worked with Niamul on multiple Google Analytics projects and was impressed by his expertise and precision. He has a strong grasp of tracking, reporting, and optimization, always ensuring accurate insights. He is proactive, reliable, and easy to collaborate with.”
“MD Niamul is extremely skilled. He fixed my Shopify, Google Analytics, and Google Ads conversion tracking perfectly. Everything works exactly as it should now, and he even added an extra data layer, which was very helpful. Outstanding service, fast delivery, and highly recommended.”
“Reliable, knowledgeable, and truly trustworthy. I’ve been working with Niamul for over a year, and he consistently delivers exceptional results across all analytics tasks. His expertise, communication, and commitment make him my go-to specialist for tracking, measurement, and data accuracy.”
“MD N delivered flawless Meta Pixel, CAPI, and server-side tracking. He understood our goals quickly, explained everything clearly, and ensured full transparency. Communication was smooth, updates were consistent, and the results improved our data accuracy and ad performance.”
Five-star reviews from businesses and agencies in the USA, Canada, UK and Australia.
Read All ReviewsYes. It is free, needs no login and tests 26 bots in one check. You can download the result as an Excel file.
Enter your site. The tool tests OAI-SearchBot, which powers ChatGPT search, GPTBot, which collects training data, and ChatGPT-User, which opens a page when a user asks. Each is shown as allowed, blocked or partly blocked.
GPTBot collects pages that may be used to train OpenAI’s models. OAI-SearchBot is used to show websites in ChatGPT’s search results. You can block one and allow the other.
No. Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls whether your content is used to train Gemini models and for grounding in Gemini.
Yes. Use the “Allow AI search, block AI training” preset in the Fix / generate tab. It writes the lines to add to your robots.txt.
Well-behaved bots follow robots.txt, but it is a request, not a wall. Some user-triggered fetchers say they may not follow it. To enforce a block you need a firewall or CDN rule.
It is a proposed text file that lists your key pages for AI tools. It is not an agreed standard and the large AI companies have not committed to reading it. It is optional and costs little to add.
A bot with no group of its own follows the “User-agent: *” group. If nothing there blocks the page, the bot is allowed. With no robots.txt at all, everything is allowed.
No. It checks whether you allow them to read your site. It cannot see firewall blocks or whether any AI product has crawled or cited you.
I am MD Niamul, a conversion tracking and web analytics specialist and an official Stape partner. I set up tracking that separates visits and leads from AI assistants, search and ads, so you can see what each one is worth.