html_to_markdown (for ai_automator)

Reads the HTML in one field and writes it into another field as Markdown. Markdown holds the same text in far fewer tokens, so it costs less to send to a model.

Why HTML to markdown for Automators?

A model that reads HTML pays for every tag, every attribute and every class name. None of that carries meaning for the model. On a EUR-Lex notice the same document is 239,452 characters as HTML and 73,430 characters as Markdown, a drop of 69%.

The source can be any field that holds HTML: a body field, a WYSIWYG field, the output of a crawler, or imported content. This module knows nothing about where the HTML came from.

What the conversion does

• script, style and noscript are removed
• Tables become Markdown pipe tables
• Headings, lists, links, bold and italic keep their meaning
• A tag with no Markdown equivalent loses the tag and keeps its text
• The input is read as UTF-8, so accented characters survive

Requirements

• Drupal 10.4+ / 11.x / 12.x
• AI ^1.4, with the ai_automators submodule enabled
• One field that holds HTML, and one field for the Markdown

How to use

1. Add a field for the Markdown. A plain long text field keeps the Markdown intact. A formatted long text field also works.
2. Open the settings of this field.
3. Enable the AI Automator and select HTML to Markdown.
4. Set the base field to the field that holds the HTML.
5. Give this automator a higher weight than the automator that fills the HTML field, so it runs after it.

An automator cannot write into its own base field, so the HTML field and the Markdown field must be two different fields.

Selector

The module has one setting. Leave it empty to convert the whole field, or give one CSS selector to convert only a part of it.

Selector Result (empty) The whole field #document1 The content of the element with id document1 div.content The content of every div with class content main article The content of every article inside main

The module returns the content inside each match, not the match itself. When the selector matches more than one element, the parts are joined in document order. When the selector matches nothing, or when the selector is not valid, the field stays empty.

Allowed source fields

text_long, text_with_summary, text, string_long and string.

Related

• AI Automator: EC URL Transformer turns a EUR-Lex link into a Cellar URL that a crawler can read.
• Simple Crawler fills a field with the HTML of a page.