# The wildcard block below already allows every crawler, AI ones # included — verified against production: GPTBot, ChatGPT-User, # OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, # Amazonbot and meta-externalagent all get 200 today. These are named # explicitly anyway, not because the wildcard needs help, but so a # later rule aimed at a bad actor can't silently catch one of these by # sitting above the wildcard — a named Allow can't be overridden by a # * block added after it by mistake. # # No Disallow for /dashboard or /auth, on purpose. Both pages already # carry (Auth.tsx, # Dashboard.tsx) — that is the correct way to keep a page out of the # index. Disallow blocks the crawl itself, and a page Google cannot # crawl is a page where it can never read the noindex tag. The two # together don't combine, they cancel: if either URL is ever # discovered through a stray link, Disallow is what stops Google from # ever recrawling it to find the noindex and drop it. The previous # version of this file carried both at once, on every group. User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Amazonbot Allow: / User-agent: meta-externalagent Allow: / User-agent: CCBot Allow: / User-agent: * Allow: / Sitemap: https://barberreceptionist.com/sitemap.xml