Engineering field log · github.com/mparrett/fmpl
How the metacircular-parser ledger — 120 ignored tests gating self-hosting — went to zero: the decisions, dead ends, and five stacked bugs hiding behind a single error message.
The first transcript is the milestone. The second is the point of it: the same grammar that generates the Rust parser now also runs as an ordinary grammar value. Background reading: the engineering tour · try it in the browser REPL.
After the public-release rehabilitation (the 24-commit sweep reviewed in PR #1 and chronicled in its own field log), the repo had a working build, CI, and an honest ledger of what didn’t work: docs/known-gaps.md, dominated by one bucket of ~120 ignored tests labeled metacircular parser not yet complete. FMPL’s flagship commitment (DESIGN-001) is that the canonical parser is written in FMPL and the Rust parser is stage-0 scaffolding — so this bucket wasn’t a feature gap, it was the language not yet being what it claims to be.
The plan ordered three workstreams deliberately. First a documentation test harness, because the parser work would change language behavior and we wanted the docs to break loudly when it did. Then a WebAssembly build — independent of parser internals, safe to do while thinking. Then the critical path. The sequencing mattered: by the time the grammar changed, every code block in the tutorial was executing in CI and would have caught a regression the unit suite missed.
Issue #2 made TUTORIAL.md, DEMO.md, and README.md build inputs: a harness (fmpl-core/tests/doc_examples.rs) extracts every fmpl code fence, executes it against the real VM, and asserts -- Returns: / -- => comments against actual results. Blocks that can’t run are marked, not skipped silently — fmpl-doctest: skip for network calls, a fmpl-sketch fence tag for design sketches.
Building the harness immediately caught two real documentation bugs — examples that read correctly and behaved otherwise. Both traced to the same root: the parser is newline-insensitive, so a [ opening a line indexes the previous expression instead of starting a list. That gotcha (issue 5 in specs/parser-limitations.md) became a recurring character in this log; it later explained two “failing” tests in §6 as well.
Issue #3’s opening assumption — that wasm would need a no_std campaign — died in the first hour, pleasantly. wasm32-unknown-unknown ships full std (threads and files trap at runtime rather than failing to compile), so the real blockers were exactly four native-only dependencies: curl, fjall, stacker, and tokio’s multi-threaded runtime. Target- and feature-gating those got fmpl-core compiling for wasm32 with the native build untouched.
The gating surfaced a wrong assumption worth recording: fjall was not all behind the fjall-persistence feature flag. Stream overflow spill, parse-state save, and the persistence envelope are live in default native builds — so the wasm exclusion had to be cfg(not(target_arch = "wasm32")) seams at each touchpoint, not a feature toggle. Those seams are now the documented slot-in points if a persistent browser image (IndexedDB-backed) ever becomes a goal — a direction the project’s creator, Norman Nunley, raised in review.
The deploy fork point: commit the built .wasm to the repo (simple, binary-in-git) or switch GitHub Pages to Actions-based deploys that rebuild it from source on every push. We chose Actions — the binary never lands in git, and the live REPL can’t drift from main. That choice quietly paid off in §4: when the parser changed, the browser REPL picked it up on the next push with zero extra work.
The milestone test was precise: load lib/core/ast_to_ir.fmpl — the tree grammar that lowers AST to IR — through the generated parser. It failed with Parser { token: 738, message: "negative lookahead matched" }, and that error message was the first adversary. Token 738 mapped to an innocent-looking [ mid-file; a scratch tool that dumped the token stream around it pointed at a line structurally identical to twenty lines that parsed fine. Then a truncated 50-line version of the file produced the same error at the same position — position 738 turned out to be a byte offset, and the byte offset of the start of the failed statement. The error was pointing at where the parse began, not where it died.
So we stopped reading the error and started bisecting: truncate the file, parse, narrow. What looked like one bug decomposed into five, stacked such that each was only visible after fixing the one above it:
GrammarDef arm — any source containing a grammar literal died with “Unknown AST node type.” The full conversion existed in an orphaned module that was no longer compiled anywhere (§5, fork 1).digit = [0-9] letter = [a-z], the sequence parser consumed letter as a rule reference inside digit’s body, then choked on the =. The legacy Rust parser had a rule-start lookahead; the FMPL meta-grammar didn’t. One negative-lookahead rule fixed it.[:Int, any:n], nested [[:Binding, …]] — had no rules at all. Flat patterns silently mis-parsed as character classes (the char-class rule accepts almost any characters); nested ones failed outright. The biggest single piece of new grammar, mirrored against the legacy parser with AST-equality tests.generated_parse returned success when the parse stopped mid-input, so syntax errors became wrong programs: [1] @ g.rule parsed as just [1], and multi-statement files parsed as their first statement. Enforcing end-of-input revealed that prelude.fmpl had never once fully parsed.input.get(pos..pos+1), which returns None on a multi-byte UTF-8 character. The final blocker for the whole milestone was a ⇒ arrow in an ast_to_ir comment. Matchers now advance by UTF-8 length.Bug 4 explains an old mystery. A long-ignored test applied a grammar with "5" @ g.digit and asserted the result "5". It “passed” during this work — but the @ application was being silently dropped by the prefix parse, and the expression evaluated to… the string "5". The expected value equaled the input, so a parser that discarded the interesting half of the program produced exactly the asserted answer. A new test whose transformation output differs from its input ([:Int, 42] → [:LoadInt, 42]) is what exposed the drop. Choose fixtures where wrong ≠ right.
Five decisions shaped the work more than any single fix. Recorded here with the road not taken:
value_to_ast.rs) exists but is registered in no module tree — dead code, maintained by hand for months, with one function reduced to a match arm that can only return an error. Delegating to it would have deduplicated ~700 lines but meant trusting code that provably never ran. We extended the postlude (the epoch-bump history shows that’s the established pattern) and used the orphan only as a porting reference — where its rot was itself informative.[…] a character class or a list pattern? The legacy parser decides with a 20-character lookahead scan for commas. The PEG formulation instead attempts the list-pattern parse behind a lookahead and falls through to char-class on failure — which turned out strictly more robust: [_a-z] hard-errors in the legacy parser but degrades gracefully in the canonical one. We kept the divergence and documented it, rather than reproducing a 20-character quirk bug-for-bug.x @ g.rule, the legacy parser reads a full postfix expression and retroactively splits the final .rule off it. Faithfully mirroring that in PEG is awkward; instead the canonical operand is a plain (possibly qualified) variable. Narrower in theory than the legacy parser — but the old rule could never match anything (its operand ate .rule as property access), so the restriction is a strict improvement, sized to the real corpus.io::load. The rewrite drives generated_parse directly, no globals. Process-global toggles and parallel test runners compose exactly once.The plan sized the remaining bucket — 98 ignored tests in core_prelude.rs — as open-ended, likely multiple sessions. Those tests do something more ambitious than the milestone: they run fmpl_parser.fmpl as an interpreted grammar in the VM’s grammar runtime — "true" @ fmpl_parser.code — the same grammar that generates the Rust parser, executed live.
Every one of the 98 failed with the identical error: Type { expected: "callable", got: "null" }. Uniform failure across a hundred tests is not a hundred problems; it’s one. Isolating by rule showed bool, expr, and stmts all working and only the top-level code rule failing — whose semantic action calls length(). That function existed only as a Rust helper baked into the generated parser; interpreted, it resolved to null. One pure-FMPL length in the prelude later, the next null was string.to_symbol — referenced by the prelude’s symbol helper since its beginning and never implemented as a VM builtin. Two definitions, ninety tests.
The five stragglers were a different genre: tests pinning semantics the language had deliberately moved past — assertions on the deleted Tagged value shape (canonical list form replaced it, DESIGN-002) and grammars using bare n as a binding where bare identifiers had since become rule references (DESIGN-004). Modernizing a test is legitimate exactly when the language change was on purpose and documented; all five updates cite the design decision they align to. Two long-ignored “stack overflow” tests turned out to just pass — the grammar engine had grown segmented-stack support in the meantime, and nobody had re-run them.
Milestone 1 was expected to be the hard part and the burn-down the long tail. The reality inverted: the milestone was five distinct bugs across two parsers and a code generator; the “open-ended” burn-down was two missing definitions found in under an hour. The signal was there in advance — one error shape across the whole bucket — and reading it early is the cheapest estimation tool we know.
Condensed for whoever picks up the next bucket:
Error positions lie; bisection doesn’t. The reported position was the start of the failed statement, three layers of tooling below where the failure lived. Truncate-and-retry located every one of the five bugs; reading the error message located none of them.
A parser that accepts prefixes hides every other bug. Bugs 1–3 were invisible for months partly because bug 4 converted their failures into quietly-wrong successes. When a milestone is “make X parse,” the first commit should be making failure loud.
Uniform failure means one cause. Ninety-eight tests, one error string, two missing definitions. Sizing work by test count assumes failures are independent; check that assumption before writing the estimate.
The whole input space includes comments. A single ⇒ in a comment blocked the flagship milestone. Character-level code that indexes bytes must be tested on multi-byte input, and “source code” is multi-byte the moment someone writes prose in it.
Keep the two parsers honest with equality, not vibes. Every tree-pattern rule added to the FMPL grammar is pinned by a test asserting the generated parser’s AST equals the legacy parser’s, node for node. DESIGN-001’s “both parsers describe the same language” is only real if something fails when it stops being true.
The loop this log is named for: an FMPL grammar describes FMPL, generates the Rust parser that parses FMPL, and now also runs directly in FMPL’s own grammar runtime, parsing the language that defines it. The remaining distance to full self-compilation is roadmap, not mystery — and the next traveler starts with a suite that tells the truth.