docs/internal/embedded-language-highlighting.md
AI-generated: describes Fresh's architecture and design rationale, not implementation details; where it disagrees with the source, the source is authoritative.
Status: IMPLEMENTED for TextMate-engine hosts: Markdown fenced code
blocks (the motivating case — issue #2689) and Vue <script>/<style>
blocks (the proof of generality: two region kinds in one host, lang
attributes, and default languages, added with spec-table entries plus
grammar scopes and no engine control-flow changes). Companion to
syntax-highlighting.md, which describes the
checkpoint/incremental engine this mechanism extends.
Some files legitimately contain more than one language: Markdown fences,
HTML <script>/<style>, Vue/Svelte components, templating languages. The
highlighting engine is deliberately incremental and viewport-bounded
(checkpoints + windowed parsing, no full-buffer rescans on edit), so any
mixed-language support has to preserve those properties. A point fix that
scans the whole buffer for fences on every edit, caches per buffer version,
and turns itself off above a size threshold breaks both design rules
("avoid full-buffer scans", no size cliff) — that's what this mechanism
replaces.
There are now three ways to get a second language highlighted, and they are not interchangeable. Rule of thumb for future mixed-language cases:
The grammar itself embeds the other language (via the shared
SyntaxSet): HTML embedding CSS/JS, PHP embedding HTML, and most
templating languages work this way already, because TextMate grammars
can push contexts from another grammar in the same set. The engine's
sequential ParseState tracks those transitions natively — checkpoints,
forward extension and convergence all just work (see the embedded-CSS
e2e tests). Reach for this first whenever the embedded language is
statically known to the host grammar. It's pure grammar data: add or
extend a .sublime-syntax in the build-time dump; zero engine changes.
This is also the answer for genuinely interleaved templating (Jinja,
ERB, Twig-style): a template grammar that switches contexts mid-line
between template markers and host text is exactly what TextMate
grammars are good at, and the engine already supports it.
The engine-level embedded-region mechanism (this doc): for hosts
where the embedded language is named by the document, so no grammar
can enumerate it statically. Markdown fences are the canonical case:
the info string ("rust", "py", "~~~{.python}") can name any of
the ~140 syntaxes in the set, including user/plugin-registered ones.
Vue single-file components are the second: <script lang="..."> /
<style lang="..."> name the language, with js/css as per-region
defaults when no lang is given. Block-delimited, line-granular
regions only.
highlight_string: one-shot highlighting of a detached string
(hover popups, the markdown preview renderer). Never use it for
buffer content — it has no incrementality and no cache.
The TextMate engine's resumable parse state is a composite snapshot:
the host parser's (ParseState, ScopeStack) plus, while inside a
recognized region, an embedded child parser's (syntax, ParseState, ScopeStack). That snapshot — not just the host state — is what
checkpoints, the cache tail state, and the convergence comparison carry.
Because every incremental path already flows through those snapshots, the
mechanism inherits the engine's whole lifecycle for free: resume-from-
checkpoint into the middle of a region, forward extension while scrolling,
partial update with convergence after edits, and the streaming-tail rules.
Region detection is driven by the host grammar's own scopes, not by a second lexer. A host declares one spec per region kind (Vue has two: script and style), each with two scope selectors and an optional default:
region_scope — the scope the host grammar keeps on the stack for the
whole region (Markdown: markup.raw.code-fence; Vue:
meta.embedded.block.script / meta.embedded.block.style);language_scope — the scope the host grammar puts on the language
token of the opening line (constant.other.language-name for both);default_language — used when the opening line names no language or
names one that doesn't resolve to a syntax in the set. Vue uses js/css;
Markdown uses None, meaning such regions keep the host's own raw-code
styling.A compact TextMate TypeScript grammar is bundled solely for embedded
contexts (lang="ts" in Vue, ```ts fences): the grammar catalog skips it
by name — mirroring the JavaScript skip — so .ts buffers keep the richer
tree-sitter highlighting, while find_syntax_by_token("ts") still
resolves it for regions. (Vendoring Sublime's official TS grammar is not
an option: it needs branch_point, which no released syntect supports.)
Per line, the host parser runs first (inside a region it is in a cheap "raw" context — it must run regardless, because only the host knows where the region ends), and the region-scope presence at line start vs end classifies the line:
| region scope before → after | meaning | styled by |
|---|---|---|
| absent → absent | ordinary host line | host |
| absent → present | region opened; language token resolved via find_syntax_by_token | host |
| present → present | region content | child (host, if language unrecognized) |
| present → absent | closing delimiter; child state dropped | host |
Driving detection off the host grammar's scopes has a correctness property worth preserving: the highlighted region is exactly what the grammar recognizes as a region (marker-length rules, indentation quirks and all), so region styling can never disagree with the fence rendering itself. It also means zero extra scanning: detection rides along the line parse the engine was doing anyway.
Unrecognized languages (and fences with no info string, and anything
resolving to plain text) keep the host's own styling (markup.raw →
string color), so nothing regresses.
onig::Region equality
is allocation identity — a clone is already unequal to its original;
pre-existing, also true on the host-only tuple this replaced). The fence
context holds captures for its close-marker backreference, so an edit
inside a region re-parses to the region's end and converges at the
first checkpoint after it, still bounded per pass by the convergence
budget. Fixing that requires value-equality regions upstream (syntect's
regex-fancy backend has them; regex-onig does not).<style>).Add one EmbeddingSpecDef entry per region kind (host syntax name, the
two scope selectors, and the optional default language) in the engine,
and unit tests mirroring the Markdown/Vue ones:
recognized region, unrecognized language fallback, an edit to the
language token restyling the whole region, and a region past
MAX_PARSE_BYTES. If the host grammar doesn't scope a language token or
region, fix the grammar first (tool 1) — the engine mechanism assumes the
host grammar tells the truth about regions. That is exactly what the Vue
grammar needed: its hand-rolled pseudo-JS/CSS contexts were replaced with
honestly-scoped raw regions (meta.embedded.block.*) plus a scoped lang
attribute value, and the engine does the rest.
textmate_engine.rs mirror does not implement the
mechanism yet.