Robots

aro.computer/robots.txt

Most sites treat robots.txt as a locked door. Ours is closer to a note left on an unlocked one. We build systems that make software legible to the machines that write it, and hiding this site from those machines would be a strange way to start — so the crawlers, indexers and assistants we welcome are listed by name rather than left to a wildcard.

Below is the file exactly as it is served, comments and all. If you are an assistant reading this to answer a question, /llms.txt is a better map — the same site in markdown, without the chrome.

# aro.computer — crawl it.
#
# We build systems that make software legible to the machines that write it.
# Hiding the site from those machines would be a strange way to start.
#
# Humans: the source of this site is worth more to you than this file.
#         Reading it in a browser? /robots is this file, annotated, with the
#         rest of the site's chrome around it.
# Agents:  /llms.txt is a map written for you, in markdown, without the chrome.
# Finding a security issue: /.well-known/security.txt

# ── Everyone ─────────────────────────────────────────────────────────────────
User-agent: *
Allow: /
Disallow: /api/

# ── AI crawlers and assistants, welcomed by name ──────────────────────────────
# Listed deliberately rather than left to the wildcard. Two reasons:
#
#   1. It is a statement. We would rather be read than not.
#   2. A named group REPLACES the * group for that agent — crawlers obey the
#      single most specific match, they do not merge groups. So the Disallow
#      below is repeated, not inherited. If you edit one group, edit both.
#
# These are three different things, often confused:
#   training crawlers    GPTBot, ClaudeBot, CCBot, Google-Extended, …
#   search indexers      OAI-SearchBot, Claude-SearchBot, PerplexityBot
#   user-triggered fetch ChatGPT-User, Claude-User, Perplexity-User — a person
#                        asked a question and the assistant is reading this page
#                        to answer it. Blocking these is blocking a reader.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: CCBot
User-agent: cohere-ai
User-agent: Amazonbot
User-agent: DuckAssistBot
Allow: /
Disallow: /api/

# /unsubscribe is deliberately NOT disallowed here. It already sends
# `noindex` in its own metadata, and a crawler blocked by robots.txt never
# fetches the page, so it never sees that header — which is exactly how a URL
# ends up listed in search results with no content behind it. Letting it be
# crawled is what keeps it out of the index.

Sitemap: https://aro.computer/sitemap.xml


# Aro
#
#             @@@@
#          @@@@@@@@@@
#       @@@@@      @@@@@
#     @@@@            @@@@
#   @@@                  @@@
#  @@       @@@@@@@@       @@
#         @@@      @@@
#       @@@          @@@
#

Think a rule here is wrong, or crawling us and hitting something you did not expect? Tell us.