# Data usage: reading and using individual listings is allowed (incl. AI/LLM
# processing). Bulk extraction and redistribution / re-hosting of the listings
# on any other site or dataset is prohibited.
# See https://assetlog.ai/terms-and-conditions#data-license
#
# Content Signals (https://contentsignals.org/) express, per crawler, what a
# collected copy may be used for. As a condition of accessing this website you
# agree to honour them:
#
#   search=yes     you may include listings in a search index and link to them
#   ai-input=yes   you may read a listing to ground a real-time answer and cite it
#   ai-train=no    you may NOT use this content to train or fine-tune a model
#
# A Content-Signal is repeated in every group on purpose. Under RFC 9309 a
# crawler obeys only the single most specific User-agent group that matches it
# and ignores the rest, so a signal declared solely under `*` would never reach
# CCBot or Google-Extended — the very crawlers `ai-train=no` is aimed at.
#
# Groups reviewed 2026-09-10 against each operator's own documentation:
#   OpenAI    GPTBot (training) · OAI-SearchBot (ChatGPT search index)
#             ChatGPT-User (user-triggered fetch) · OAI-AdsBot (ad policy check)
#   Anthropic ClaudeBot (training) · Claude-SearchBot (search index)
#             Claude-User (user-triggered fetch)
#             Claude-Web and anthropic-ai were REMOVED — Anthropic no longer
#             lists them; keeping dead agents only makes the file harder to read.
#   Perplexity PerplexityBot (index) · Perplexity-User (user-triggered fetch)
#   Meta      meta-externalagent (training) · Meta-WebIndexer (Meta AI search,
#             citations) · facebookexternalhit (link previews)
#
# Disallowed since 2026-09-21: GPTBot and meta-externalagent. Both operators
# document them as TRAINING crawlers only. OpenAI: "GPTBot is used to make our
# generative AI foundation models more useful and safe", and search inclusion
# is governed by OAI-SearchBot alone ("Each setting is independent of the
# others"). Meta: meta-externalagent crawls "for use cases such as training
# foundation AI models"; Meta AI citations come from Meta-WebIndexer. Together
# they made ~68 % of all requests while ai-train=no already refused the use
# they crawl for. Search, citation and user-triggered agents stay allowed.

User-agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: GPTBot
Content-Signal: ai-train=no, search=yes, ai-input=yes
Disallow: /

User-agent: meta-externalagent
Content-Signal: ai-train=no, search=yes, ai-input=yes
Disallow: /

User-agent: ChatGPT-User
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: OAI-SearchBot
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: OAI-AdsBot
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: ClaudeBot
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: Claude-SearchBot
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: Claude-User
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: PerplexityBot
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: Perplexity-User
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: Google-Extended
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: Bingbot
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

User-agent: CCBot
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

Sitemap: https://assetlog.ai/sitemap.xml

# For AI agents — machine-readable entry points (listed as comments, because
# robots.txt has no standard directive for them):
#   https://assetlog.ai/llms.txt                            start here
#   https://assetlog.ai/.well-known/ai-catalog.json         every agent surface in one document
#   https://assetlog.ai/.well-known/agent-skills/index.json SKILL.md guides (search, submit, MCP)
#   https://api.assetlog.ai/mcp                             MCP server (sign in with an AssetLog account)