Introducing Markdown for Agents: Simplifying AI Content Processing

Cloudflare has introduced a new feature, Markdown for Agents, which enables real-time content conversion at the source, allowing AI systems to receive structured markdown directly from the source. This feature is designed to simplify the process of parsing and converting HTML to markdown, which is essential for AI content processing.

Key Technical Details

Markdown for Agents uses content negotiation headers to enable the conversion of HTML to markdown. When an AI system requests a page from a website that uses Cloudflare and has Markdown for Agents enabled, it can express its preference for text/markdown in the request. Cloudflare's network will automatically convert the HTML to markdown on the fly, with the converted response including an x-markdown-tokens header indicating the token count.

Practical Implications for Developers

Developers can use Markdown for Agents to simplify their AI content processing pipelines. By enabling Markdown for Agents on their Cloudflare zones, they can ensure that AI systems receive structured markdown directly from the source, reducing the need for manual conversion and parsing. This feature is particularly useful for developers building AI agents using Workers, who can use TypeScript to request the markdown version of a page.

Example Code

Here's an example of how to use Markdown for Agents with Workers:

const r = await fetch(
  `https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/`,
  {
    headers: {
      Accept: "text/markdown, text/html",
    },
  }
);
const tokenCount = r.headers.get("x-markdown-tokens");
const markdown = await r.text();

Overall, Markdown for Agents is a significant improvement in AI content processing, enabling developers to simplify their pipelines and improve the efficiency of their AI systems.