Agentic Engine Optimization (AEO): Designing Documentation for Coding Agents
Originally published in Turkish. Read the original →
Why organizing technical content not only for humans but also for AI coding agents that fetch documents, process them as text and work within limited context has become a separate discipline, and where to start.
For years we measured developer portals against human behavior that "clicks, browses and finishes the tutorial": session duration, scroll depth, conversion funnel. Yet a significant share of documentation is now consumed by automated clients: in-IDE agents, CLI helpers, MCP servers and large language models with limited context windows. This traffic stays almost invisible in classic analytics, even though whether your product "is understood by an agent" is directly tied to integration success.
You can use the AEO auditor tool on this site for this.
In this post, I look at a framework that is a candidate to fill this gap: Agentic Engine Optimization (AEO), the practice of structuring and serving technical content with AI consumers that fetch and process it autonomously in mind.
What is AEO?
In short: AEO is a field of practice that aims to make documentation and API-related material genuinely usable by AI agents that write code. As with search engine optimization (SEO), saying "quality content is enough" doesn't cut it here either; who fetches the content, how, and how it is consumed in context become decisive.
A human reader can browse headings in order; an agent, on the other hand, usually fetches the page with one or a few HTTP requests, converts it to text, truncates it based on token count, or gives up entirely. That is why AEO thinks less in terms of visual hierarchy and more in terms of text structure, accessibility (robots), discovery (index files) and token budget.
Why is the human journey different from the agent journey?
A human developer usually moves through the menu, clicks a few internal links and uses tools to try the sample code; analytics captures this behavior.
A coding agent, however, usually does not mimic the page hierarchy step by step. It tries to download and process the document in one go; signals like client-side interactions, ad pixels and "time on page" mean nothing to it. The result: even if your documentation has been read, you may see "zero engagement" on your dashboard. This doesn't mean the content isn't being used; it means the measurement doesn't fit the model.
Token economics: the risk of invisibility
Agents' context windows are not unlimited. In practice, a single "guide" of hundreds of thousands of tokens can fill the entire task context or is doomed to be silently truncated. Typical outcomes on overly long pages are:
- Critical warnings toward the end of the text getting dropped (truncation),
- The agent turning to a shorter source,
- Latency and error risk in chunking attempts,
- The model drifting to generic answers from its training (hallucination).
That is why a rough token estimate per page is no longer just an editorial luxury; it is a signal for the agent's "should I read this now?" decision. A crude but useful estimation method, such as dividing the character count by four, is enough in many scenarios.
Practical upper limits (as guidance):
| Type | Approximate target |
|---|---|
| Quickstart | Under ~15,000 tokens |
| A single API reference page | Under ~25,000 tokens |
| Concept explanation | ~20,000 tokens; split detail out with links |
| Entire reference on one page | Avoid if possible; split by resource or endpoint |
An AEO stack, layer by layer
AEO is not a single setting; it consists of complementary layers.
1. Access: robots.txt and permissions
Agents usually check boundaries with robots.txt before fetching. An accidentally broad Disallow rule can make documentation invisible to automated clients; while the page seems to "open" in the browser, the agent may not receive any data at all.
Where needed, approaches like agent-permissions.json, which provide detailed rules for automation, are also on the table; but for most teams the first step is a consistent robots audit.
2. Discovery: llms.txt
llms.txt is an "agent-friendly index" file hosted at the root, usually in Markdown format. For humans it resembles a sitemap; for agents it gives short answers to the question "what is each document for?" A good index:
- Is grouped by task and use case (it doesn't have to mirror the product tree),
- Fills each item with a one-sentence description saying what you will learn,
- Adds an approximate token note for critical pages where possible,
- Is itself kept lean so it doesn't exceed a few thousand tokens.
3. Capability signal: skill / capability files
The index answers the "where" question; a capability declaration gets closer to "does this API meet this intent?" For example, a skill.md-style file could summarize:
- What can be done (capability list),
- Which inputs are required (keys, environments, redirect URI),
- Limits (rate limits, durations, mandatory security measures),
- Links for in-depth reading.
This way, the agent can make a better internal decision about compatibility before pulling a ten-thousand-token guide from start to finish.
4. Format: content that agents can parse easily
- Where possible, offer a raw Markdown or low-noise text path; text derived from plain HTML wastes tokens with tag and navigation leftovers.
- Don't skip levels in the heading hierarchy; in each section, put the result / what to do first, then the reasoning.
- Keep the sample code right below the claim.
- Compressing parameters into tables instead of long sentences is usually better for both humans and models.
5. Making tokens visible
Publishing an approximate token or character count per document, or over HTTP, helps the agent, or the system managing it, prioritize. This information is more useful when it is consistent between llms.txt and in-page metadata.
6. "Copy for AI" and AGENTS.md at the repo root
When a developer copies from the browser, menu and footer noise comes along too. A button that copies clean Markdown may look like a small UX improvement, but it seriously increases context quality.
On the other hand, root-level files like AGENTS.md build a bridge to documentation by giving agents that open the code repository a single entry point for project structure, commands, constraints and external documentation links.
Traffic and measurement
Leaving agent traffic only to referrer reports can fall short; some clients arrive directly or with unique user-agent strings. Collecting client signatures, referrals from chat interfaces and known AI assistant domains into a separate segment in server logs provides an early-warning indicator for seeing the impact of your AEO investment.
Summary checklist
- [ ] robots.txt doesn't create unwanted blocks on AI and documentation paths.
- [ ] There is an llms.txt (or an equivalent lean index) at the root or in a discoverable place.
- [ ] Long references are split into sensible chunks; token/character estimates are generated where possible.
- [ ] Capabilities and constraints can be summarized in plain text; it states not only "how to call it" but also "what is promised".
- [ ] Access to Markdown or a low-noise source is possible.
- [ ] Human-centric metrics don't stand in for "agent success" on their own; they are complemented with logs or custom segments.
AEO adds a second axis next to SEO: designing content around the real constraints of the autonomous consumer. The good news is that clear headings, simple examples, accessible raw text and honest token information also benefit human readers. The bad news is that measurement stays silent: agents are usually invisible in your funnel. That is why both improving the format and producing agent-friendly signals in logs and indexes are part of the same package.
