Swarmnest Search

robots.txt (RFC 9309)

The Robots Exclusion Protocol standard.

Visit robots.txt (RFC 9309)Docs

More in Agent Protocols & Standards

Agent2Agent (A2A) Protocol

Open protocol for agent-to-agent communication.

Agent Communication Protocol (ACP)

Open protocol for agent interoperability from IBM's BeeAI.

AGNTCY

Open infrastructure for the internet of agents.

llms.txt

Proposal for a /llms.txt file helping LLMs use websites.

Information is summarised in our own words and may be out of date; check the vendor's site for current terms and prices. Not affiliated with robots.txt (RFC 9309). Suggest a correction.

CategoryAgent Protocols & Standards
AuthenticationNo auth
PricingFree
Websitehttps://www.rfc-editor.org/rfc/rfc9309
Docshttps://www.rfc-editor.org/rfc/rfc9309
Last checked2026-10-14
As JSON/api/v1/directory/robotstxt-rfc-9309.json