---
title: Content negotiation for AI agents, serving markdown with the Accept header
canonical_url: https://tkmstudio.com/library/content-negotiation-for-ai-agents
language: en
last_updated: 2026-07-13
organization: TKM Studio, AI Visibility & Adoption Studio, Madrid, working worldwide
---

# Content negotiation for AI agents: serving markdown with the Accept header

AI agents burn most of their token budget parsing HTML built for browsers. Content negotiation lets the same URL serve a clean markdown rendition when an agent asks for it, without touching what humans and search bots see. Here is how to do it correctly, and the mistake to avoid.

A modern web page is mostly not content. Framework markup, styling hooks, scripts and navigation scaffolding wrap a few hundred words of actual substance. Humans never see the wrapping. An AI agent fetching the page has to carry all of it through its context window before it can use any of it.

Markdown renditions solve this: a clean, structured text version of each page, a fraction of the size, with the substance intact. The interesting question is not whether to offer them but how an agent finds and receives them.

## The Accept header is the right trigger

HTTP has had the answer since the beginning: content negotiation. A client states what formats it accepts; the server picks the best representation of the same resource. An agent that wants markdown sends Accept: text/markdown to the canonical URL and receives the markdown rendition, with the correct content type and a Vary: Accept header so caches keep the representations apart. Everyone else requesting the same URL gets full HTML.

Discovery is explicit rather than assumed: each page declares its rendition with a link rel alternate tag pointing to the markdown path, and a markdown sitemap lists every rendition in one place. An agent can negotiate, follow the link, or fetch the .md path directly.

## The mistake: triggering on user agent

The tempting shortcut is to sniff bot user agents and force markdown onto anything that looks like an AI crawler. In live testing this breaks in a specific and damaging way: the fetchers behind AI search and user-triggered browsing expect HTML, because their extraction pipelines are built for it. Force markdown on them and their extractors can come back empty. The site becomes less visible to the exact systems the layer was built to serve.

The rule that follows: negotiate only on the Accept header. Agents that ask for markdown get markdown. Every bot that does not ask gets what its pipeline expects, which leads to the second requirement.

## Prerendered HTML stays the default

A markdown layer is a complement, not a substitute. The default representation of every route must be fully populated server-rendered HTML, because most AI-related fetchers do not execute JavaScript and never send a markdown Accept header. A JavaScript shell with an empty root node is invisible to them no matter how good the markdown layer is.

The renditions also have to be maintained like content, not like an export. When page copy changes, the rendition changes in the same commit, or agents receive a stale version of the site while humans see the current one.

## Running live

This is not a proposal. The layer described here runs in production on tkmstudio.com: request any page of this site with Accept: text/markdown and the server returns its markdown rendition, token count in a response header, sitemap referenced, Vary set. The page you are reading has one. It is the same layer we ship on client builds.

Get an AI-ready build: https://tkmstudio.com/services/visibility-sprint