robots.txt
Viktor — open to AI search and answer engines (AEO/GEO).
Directives for web crawlers
Default:
- Allow all crawlers
- Disallow only API + dev surfaces.
User-agent rules:
- User-agent: *
Allow: /
Disallow: /api/
Disallow: /component-playground/
Content signaling
- Content-Signal: ai-train=yes, search=yes, ai-input=yes
Specific user agents:
User-agent: GPTBot
Allow: /User-agent: OAI-SearchBot
Allow: /User-agent: ChatGPT-User
Allow: /User-agent: ClaudeBot
Allow: /User-agent: Claude-User
Allow: /User-agent: Claude-SearchBot
Allow: /User-agent: Google-Extended
Allow: /User-agent: Applebot-Extended
Allow: /User-agent: Bingbot
Allow: /User-agent: PerplexityBot
Allow: /User-agent: Perplexity-User
Allow: /User-agent: Mistralai-User
Allow: /User-agent: meta-externalagent
Allow: /User-agent: cohere-ai
Allow: /User-agent: DuckAssistBot
Allow: /User-agent: Amazonbot
Allow: /User-agent: Bytespider
Allow: /
Integration pages
- Pages are split per indexation tier so Google reports indexed/submitted separately for each.
- Pages outside these files are still crawlable - they carry
noindex, followuntil they clear the content bar. - The tier 2 line appears automatically once the first batch is released (see @/lib/integration-indexation).