Writing my last Markdown Parser

Too long, didn't read? View the project at fgmd.dev.

Introduction

Every post on this site is written in markdown. So is the privacy policy on the klyx website, and so are the posts on xyphia.uk. Three sites, all doing broadly the same thing -- turning a text file with some asterisks in it into a web page -- and until this week, three completely separate ways of doing it.

This site had a PHP function made of regular expressions, about a hundred and forty lines long. Klyx had a TypeScript parser of its own, about a hundred and eighty lines, written on the 8th of September. xyphia.uk leans on marked, a library off npm, with its own opinions about everything. None of the three shared a line of code with another, and none of them fully agreed on what a piece of markdown meant.

So I wrote another one. Hear me out.

It's called fgmd, which stands for the Fucking Good Markdown Parser (the npm listing politely bleeps it). It passes every one of the 652 examples in the CommonMark spec and all 24 of GitHub's extensions to it, and the plan is for it to be the only markdown parser I ever have to think about again.

I fear this was a long time coming...

How the old one worked

The parser that builds this site is, genuinely, a pile of find-and-replace. It hides code blocks behind placeholders so nothing else can touch them, then runs a list of regular expressions over the whole post from top to bottom: headings, then rules, then quotes, then images, links, bold, italics, lists, tables. Anything left over gets wrapped in a paragraph tag.

This works surprisingly well, right up until it doesn't. Every rule runs over the output of the rule before it, and none of them know the others exist. Here's what it does with a few perfectly ordinary bits of markdown:

Maths turns into italics. Backslashes, which are supposed to mean 'I actually want an asterisk here', get printed and then ignored. A link with a hover title produces broken HTML. Nested lists just don't exist. A quote with two paragraphs leaves a stray > on the page between them, and an image whose alt text contains a quotation mark closes its own attribute halfway through.

The fix for most of these, in practice, is to write around them. Which means the parser was quietly getting a say in how I write.

You can read the history of the thing in its comments. Links recurse through balanced brackets, so that a Wikipedia link ending in Mull_of_Kintyre_(song) (which i used on the Whiskey Distillery post!) keeps its closing bracket. HTML comments get stripped, because a comment left in a post still ships to every reader in the page source. And there's a line in the table code that reads NB: no \0 in the charlist -- default trim() would eat the \x00INLINE..\x00 markers protecting inline code, which is the sort of comment you only write after it has already gone wrong once.

It went wrong twice, as it turns out. That table up there was meant to be a code block, and when I first wrote this post it came out on the page as the word CODE0. The step that wraps paragraphs calls trim() on every block, and that eats the exact same \x00 markers -- so a fenced code block has never once rendered on this site, all the way back to the first commit. I just never noticed, no post had ever used one.

Then klyx

Klyx is a minecraft event and public server, I'm friends with the main developer and have been working with them on the site. klyx.org if you fancy a look at my work.

Klyx needed different things. The site is built in Svelte, and there's a rule I have with svelte: nothing gets rendered with {@html} - the Svelte feature that pours a raw string of HTML straight into the page. Its absence is treated as a second line of defence -- if a string of HTML can never reach the page, a whole category of bug can't happen. So its markdown parser couldn't produce HTML at all. It builds a tree of blocks instead (a heading here, a list there), and a Svelte component draws each one. Links only get through if they start with http, https, mailto or a slash.

That's a good design, and a completely different one. It handles headings up to level three, lists, quotes, bold, italics, inline code and links. No tables, no images, no code blocks. And because it was a separate parser written from scratch, it had its own copies of bugs I'd already fixed here. Its link pattern stops at the first closing bracket, so that same Mull of Kintyre link that works on this site breaks on klyx.

That's the bit that wears you down. Every fix lived in exactly one place. Fixing a bug here did nothing for klyx, fixing it on klyx did nothing for here, and xyphia.uk (another friend's site that I helped to build) wasn't running either of them. Three sites with three sets of rules, and me holding all of them in my head. Unideal!

Suppose its my fault for doing this junk in php. In 2026. I just wanted to do something different...

Writing it properly

Start from the answers. CommonMark is a specification for markdown, written because the original description was vague enough that every parser ended up disagreeing. More usefully than the prose, it comes with 652 examples: a snippet of markdown, and the exact HTML it should produce. GitHub's flavour adds 24 more, for tables, strikethrough, task lists, footnotes and bare links.

So fgmd has a script that downloads those examples and a test that runs every single one. The test suite also keeps a baseline, and fails if the number of passing examples ever goes down. That turns the spec into a ratchet: you can only ever move forwards. Right now it sits at 652 of 652, and 24 of 24.

Don't invent the algorithm. The regex approach falls over because whether an asterisk means emphasis depends on what's either side of it, what's already open, and what comes later -- and find-and-replace can only see one of those at a time. The people who wrote the spec also wrote a reference parser, commonmark.js, and fgmd follows its structure rather than trying to be clever.
(It also made it much easier for me to develop. But dont tell anyone, okay?)

It works in two passes. The first goes line by line, keeping track of which containers are currently open -- a quote, inside a list item, inside another list -- and deciding, for each new line, which of them it continues and which it closes. That's how nested lists and multi-paragraph quotes just work: nesting is the thing it's actually tracking. The second pass handles the text inside each block using a 'delimiter stack'. Every run of asterisks or underscores gets written down as a possible opener or closer, depending on what's next to it, and they're paired up at the end. That's why 2 * 3 * 4 stays as maths. An asterisk with a space on both sides can't open or close anything.

GitHub's additions follow cmark-gfm, GitHub's own implementation, in the same way.

A tree, not a string. fgmd's parser doesn't produce HTML. It produces a tree -- plain JSON, using the same node names as mdast, which is what most of the JavaScript markdown world already speaks. HTML is just one way of drawing that tree.

Which solves klyx. fgmd ships a Svelte component that draws the same tree as real Svelte elements, with no {@html} anywhere in it. The test suite renders every node type, option and plugin both ways and checks the Svelte version matches the HTML version element for element, so the two can't drift apart.

Assume the input is hostile. Everything these three sites render is markdown their owners wrote. But the whole point was to use this everywhere, and one day 'everywhere' is going to include something other people type into.

So by default, raw HTML is shown as text rather than run. Links and images may only use http, https, mailto, tel or a relative address. The old parser here would happily turn [click](javascript:alert(1)) into a working link; fgmd turns it into the word 'click'. It catches the sneaky spellings too, JaVaScRiPt: and entity-encoded ones and all. There's a middle mode, html: 'sanitize', which lets a GitHub-like subset of HTML through by parsing it and rebuilding it as real elements, rather than trying to filter a string.

Then there's the other kind of hostile: input designed to make a parser slow. Naive parsers go quadratic on things like twenty thousand [a in a row, or a quote nested twenty thousand levels deep. cmark keeps a list of these, and fgmd's tests borrow it -- every one has to finish in under two seconds, with and without HTML, plus a matching set for every plugin. Anything nested more than a hundred levels deep is kept as text because no real document gets anywhere near that! And unlimited depth would eventually crash it. Again, unideal.

Build the plugins on the plugin API. Everything beyond the two specs is a plugin: GitHub-style callouts, smart quotes and dashes, {#id .class} attributes, definition lists, abbreviations, maths, emoji shortcodes and wikilinks. The important bit is that all eight are written against exactly the same public plugin API anybody else would use. If a built-in needs something the API can't do, the API is wrong. So by the time all eight were done, I knew it could handle more than toy examples.

Make it usable from PHP. This site is built by a PHP script that deliberately has no dependencies, and fgmd is TypeScript. So the build also produces fgmd.mjs: the whole command-line tool in one file with no imports, which you can drop into a project with no package.json and run with plain node. It has a --serve mode that stays running and takes one JSON request per line, so a PHP build that renders dozens of posts starts node once instead of dozens of times. The replacement markdown() function for this site is about twenty lines of PHP that just talks to it.

This site is using fgmd right now. Congrats!

Every quirk became an option

Seriously, all those quirks are now options! This is what im actually super happiest with, nothing got lost. Everything this site's parser did on purpose, and everything klyx's did on purpose, is now a setting:

That's also why fgmd's README has a section called 'Coming from a regex-based parser'. It's about this site.

What it costs

It isn't all free.

It's big. The old parser was a hundred and forty lines. fgmd is around five and a half thousand lines of TypeScript, before you count the tests. It's not something I can read top to bottom in one sitting and hold in my head, which the old one was.

But like, i dont need to?? Like at all? The trade is that I don't need to because those 187-something tests hold it for me! And i suppose the other trade is that this site's build will need node. It was PHP and nothing else, which was deliberate. It'll now be PHP and one JavaScript file.

And the spec isn't always what I'd have picked. Under CommonMark, __init__ comes out bold, where the old parser left it alone. A quote carries on into the next line unless there's a blank line after it. And the big one: markdown inside a block of raw HTML stays literal. The imperial post's footnotes are hand-written <details> blocks with markdown inside them, so they needed converting to proper [^1] footnotes before this site could switch over. That's the right fix anyway -- it's exactly what the details footnote style is for -- but it is a job.

So

In On Making Things I asked if you'd ever written a parser for the first time and found out it wasn't the dark art you assumed it was. Turns out the second and third times teach you something too: mostly, that the dark art was never the parsing. It was not having a spec. This is the first one I've written where, when something comes out wrong, the question isn't 'what did I mean by this regex' but 'what does the spec say' -- and there's a test that already knows the answer.

It's on GitHub at decbr/fgmd, MIT licensed, on npm as @decbr/fgmd, and on the web as fgmd.dev. If you've got your own site with a hand-rolled markdown function held together with regex and hope, give it a go. And if it gets something wrong, there's a spec for that now, so it's a bug.

Maybe even open a pull request, and help make something yourself.

Thank you for reading.

- Dia