How syntax highlighting actually works

Syntax highlighting looks like the simplest feature in an editor. Keywords are purple, strings are green, comments are grey; surely this is a list of patterns and a loop.

It is, for about the first twenty minutes. Then you hit a string containing the word if, a comment containing an unmatched quote, a regex literal that looks exactly like a division, and a template literal with an expression inside it that is itself code and needs highlighting of its own. Getting those right means knowing the structure of the program, and knowing the structure of the program means parsing it — on every keystroke, fast enough that nobody notices.

Stage one: lexing

The first pass turns a flat sequence of characters into a sequence of tokens: the smallest units the language recognises.

const total = price * 1.2;  // with tax

keyword(const) space name(total) space op(=) space
name(price) space op(*) number(1.2) punct(;) comment(// with tax)

A lexer is a state machine. It reads characters, and each character either extends the current token or ends it. The states matter more than the patterns: once the lexer has entered a string state on an opening quote, it stays there consuming everything — including keywords, including braces — until it finds the matching close. That is what makes "if you like" a single string token rather than a keyword surrounded by text, and it is the first thing pure pattern matching gets wrong.

Lexing is fast, linear in the length of the input, and sufficient for a surprising amount of highlighting. Keywords, numbers, strings and comments all fall out of it directly. It is where highlighters stopped for a long time.

Why lexing is not enough

Tokens have no relationships. The lexer sees name(total) and cannot tell you whether that is a variable being declared, a function being called, a property being accessed, or a type annotation — because the answer depends entirely on surrounding tokens that the lexer has already forgotten.

Three examples where the difference is visible:

• Function calls. Highlighting parse differently in parse(input) than in const parse = ... requires looking ahead for the parenthesis and knowing it is a call rather than a grouping.
• Regex versus division. In JavaScript, /pattern/ is a regex in one context and two division operators in another. Distinguishing them requires knowing whether the previous token ended an expression — a parsing question, not a lexical one.
• Nested languages. A template literal can contain an interpolated expression; an HTML file contains CSS and JavaScript; Markdown contains fenced code in arbitrary languages. Each needs a different highlighter to take over for a region, and hand back afterwards.

All of these need a tree, not a list. Parsing arranges tokens into a hierarchy — this identifier is the callee of a call expression, which is the initialiser of a declaration, which is inside a function body — and the tree is what lets a highlighter answer questions about role rather than shape.

The performance problem

Parsing a whole file is cheap once. Doing it on every keystroke in a 10,000-line file is not, and “on every keystroke” is not negotiable: type a quote and the highlighting after it must change immediately.

The naive options are both bad.

Reparse everything, every time. Correct and simple. Fine for small files, visibly laggy on large ones, and the lag arrives exactly when the file is big enough to matter.

Reparse only the visible region. Fast and wrong. The meaning of what is on screen depends on what came before it — whether you are inside a comment, a string, a class body. Starting mid-file means guessing that context, and guessing wrong produces the classic bug where scrolling changes the colours.

Incremental parsing

The real answer is to reuse the previous parse. When one character changes, the vast majority of the syntax tree is unaffected, and an incremental parser exploits that: it keeps the old tree, identifies the subtrees the edit could have invalidated, reparses only those, and splices the results back in.

Editing inside a function body typically means reparsing that function and nothing else. The cost scales with the size of the edit rather than the size of the file, which is what makes highlighting a large file feel identical to highlighting a small one.

Two further properties matter as much as speed. The parser must be error-tolerant, because code in an editor is syntactically invalid most of the time — halfway through typing, every program is broken. A parser that gives up on the first error would make highlighting flicker off constantly, so instead it inserts error nodes, recovers, and produces a usable tree from broken input. And it must be interruptible: on a very large paste the parser can be stopped mid-work to keep the frame rate, resuming afterwards, so the editor never blocks on it.

From tree to colours

The last stage maps tree nodes to visual styles, and it is deliberately indirect.

A parser produces language-specific node names — VariableDeclaration, ClassBody, ArrowFunction. A theme cannot possibly know about those for every language it supports. So node types are first mapped to a small set of language-neutral tags: keyword, string, number, comment, function name, type name, and a few dozen more.

ClassDeclaration/Identifier  →  tag.className
CallExpression/Identifier    →  tag.function(tag.variableName)
LineComment                  →  tag.comment

Themes then style tags, not node types. This is why a colour scheme written years ago works for a language released last month: the language supplies the mapping into the shared vocabulary, and the theme only ever speaks the vocabulary. It is a small piece of indirection doing an enormous amount of ecosystem work.

What highlighting can and cannot know

Worth being precise about, because the boundary confuses people who expect editor-grade intelligence from a highlighter.

A syntax highlighter is syntactic. It knows shape and structure. It does not know types, does not resolve imports, does not know whether a variable is defined, and cannot tell a local from a global. Those are semantic questions requiring a compiler or language server that has read the whole project — a fundamentally heavier tool, usually running in a separate process, that answers in tens of milliseconds rather than under one.

Full IDEs run both and layer the semantic information over the syntactic colours once it arrives. A browser-based editor generally runs only the first, which is why it can highlight eleven languages instantly and cannot tell you that you misspelled a method name.

In practice

NoobProMax uses CodeMirror 6, whose parser system — Lezer — is incremental, error-tolerant and interruptible in exactly the sense described above, with the tag indirection sitting between the eleven supported grammars and the themes.

Language grammars are loaded on demand rather than bundled together, which matters for a tool people open from a link: shipping every parser to everyone would mean most of the download being grammars for languages the visitor is not using. Switching language in the command palette fetches that grammar and reparses, which is why the first switch to a new language takes a moment and subsequent ones do not.

The parse tree also earns its keep beyond colour. Bracket matching, auto-indent, code folding, and structural selection all query the same tree — which is the real reason it is worth building. Highlighting is the visible output of a structure the editor needs anyway.