feat(fields): model complex fields — parse, AST, read tokens, offset widths #41

Companion bug, data loss in every mutator that rebuilds runs: #42. It lands its first step before this issue and introduces the parser from §1.

Problem

Word writes nearly every field as a complex field: a sequence of runs <w:fldChar w:fldCharType="begin"/> … <w:instrText> REF _Ref… \h </w:instrText> … <w:fldChar w:fldCharType="separate"/> … cached result runs … <w:fldChar w:fldCharType="end"/>. The reader has no model of it. Only <w:fldSimple> is recognized (as a transparent run-bearing wrapper), and only marginalText turns that into a token ({page}, {date}, …). Everywhere else a complex field reads as its cached result in plain text: no hint in read and nothing in read --ast. instrText is dropped, fldChar is ignored, and the result runs become ordinary TextRuns.

Verified on main @ e176d62 (0.29.0) against committed fixtures:

Consequences:

• An agent can't tell computed text from typed text. It hand-edits citation or caption numbers, which the next field update in Word overwrites. It retypes "Page 2 of 4", which freezes that number on every page. And it can't see where a cross-reference points.
• HYPERLINK fields aren't hyperlinks to any verb: hyperlinks list/replace can't see the résumé's LinkedIn link.
• Every mutation path that rebuilds runs treats the field as its result text, which is the data-loss bug in bug(edit): whole-paragraph edits silently flatten fields to frozen text; span replace leaves orphan fldChar #42 (fields silently flattened to frozen text, and an orphan fldChar end left behind).
• It violates "everything we parse must be in the AST": the instruction is in the XML and missing from the AST.

Scope note: this issue is the field model (parse, AST, read, offsets, discovery). #42 makes every mutator preserve fields. Cross-reference authoring (REF/PAGEREF/NOTEREF + bookmark locators + internal links) is #40, which builds on this model. Header/footer page numbers (#39, blocked on #37) consume this model instead of extending marginalText.

Proposal

1. Parse: one field parser, run-child granularity, a stack per story

New module src/core/fields/ (reader-side .ts, no JSX):

• Tokenize at run-CHILD level, not run level. fldChar/instrText/delInstrText can share a run with each other and with text. The contract footer has begin, instr, separate and end inside one <w:r>.
• Walk in document order through every run-bearing wrapper (wrapperContent, so mc:AlternateContent counts once). A field's begin and end may sit in different wrappers: the begin outside a <w:ins> and the result inside it, or a whole PAGEREF field inside a TOC entry's <w:hyperlink>.
• Keep a stack, so nested fields work, e.g. IF { MERGEFIELD x } = "y" …, or the PAGEREFs inside a TOC result. A field nested inside another's instruction contributes its result to the parent's instruction string (that is how Word evaluates it). A field nested inside a result is an ordinary child field.
• Instruction = the concatenation of every instrText/delInstrText between begin and separate (or end). Empty runs in between are ignored. Covers the split REF … \h + \* MERGEFORMAT.
• Parse the instruction once: kind from the leading keyword (page, numpages, sectionpages, section, date, time, createdate, savedate, printdate, seq, ref, pageref, noteref, styleref, hyperlink, toc, index, mergefield, formtext, formcheckbox, formdropdown, docproperty, filename, author, title, if, else other), args (the bookmark name, SEQ identifier or URL, with quotes handled), switches (\h, \r, \* MERGEFORMAT, \@ "…", …). fieldToken in marginals/text.ts moves here and becomes the one instruction-to-token map, so <w:fldSimple> and complex fields share it.
• No separate is legal: the field has no cached result. Word computes it on render (verified: Word and LibreOffice both render [1] for a PAGE field with no separate, which read shows as [] today).
• Flags from the begin fldChar: w:dirty, w:fldLock, and w:ffData (legacy form field). <w:fldSimple> is folded into the same model as a one-node field (simple: true).
• Scope: the parser stack is per block container: the body flow, each table cell, each text-box story, each footnote/endnote, each comment, each header/footer part. A field may span paragraphs within its container (TOC, INDEX, and long REF results do), and the parser state therefore lives in WalkState, not per paragraph. It never crosses a cell wall or a story boundary.
• Malformed (an end with no begin, a begin never closed before its container ends, a stray separate): no field semantics, and today's plain-text rendering, so a broken file reads no worse than now. Record malformed: true in the AST, and add a validate package-consistency check, "unbalanced field characters in pN" (warning). Today validate reports the file our own replace produces below as valid. Note that fldChar balance is invisible to the XSDs.

2. AST

Two shapes, decided by what the cached result means:

• Volatile fields (page, numpages, sectionpages, section, date, time, savedate, printdate, createdate, edittime): the cached result is a snapshot of one layout or one moment and is misleading as text. These become an atomic run, like EquationRun:

  type FieldRun = { type: "field"; id: string /* fldN */; kind: FieldKind; instruction: string;
                    token: string /* "{page}" */; result: string /* cached, lossless */;
                    trackedChange?: TrackedChange; comments?: string[] };

  The result runs are NOT also emitted as TextRuns.
• Content fields (everything else: ref, seq, noteref, pageref, hyperlink, mergefield, formtext, toc, docproperty, if, …): the cached result is the document's words, and agents need to read and find them ("Figure 1", "[1, 3, 4]"). The result stays as ordinary runs (formatting, comments and revisions intact, so paragraphText/wc/find don't change). Each result run is decorated with field?: { id; kind }, the same pattern as TextRun.hyperlink. A container FieldRun can't work here because a field can span paragraphs and wrappers.
• Every field, either shape, gets a record in Body.fieldReferences (id → begin node and parent, end node and parent, block ids spanned, record), and read --ast gains a top-level fields: Field[]:

  type Field = { id; kind; instruction; args: string[]; switches: string[]; result: string;
                 blockId: string; endBlockId?: string; parent?: string /* nesting */;
                 simple?: true; dirty?: true; locked?: true; malformed?: true; trackedChange?: TrackedChange };

• HYPERLINK fields also populate hyperlinkById as linkN with field: "fldN". They become links to read, hyperlinks list and the existing link-aware --text diff.
• Tracked changes: a field is visible in a view iff its begin run's revision chain is (isRevisionVisible). Inside <w:del>, the instruction is read from w:delInstrText.
• fldN numbering is global document order across stories, the way tbxN is numbered. Widen cli/info/schema.ts.

3. read: three renderings, all deviation-free for documents without fields

1. Volatile → the token, in place of the cached value: Experiment #1 Page {page} of {pages}, {date}. This is the same vocabulary as headers/footers today, and the same thing feat(page-numbers): read Word's page numbers as fields, edit them in place, and set start/format #39 asks for in marginals. Use {section-pages} per feat(page-numbers): read Word's page numbers as fields, edit them in place, and set start/format #39. Literal brace text that matches a token exactly is escaped (\{page}) so the two can't be confused.
2. Content → the cached result, marked with an HTML span: …in brackets [<span data-field="fld3">1</span>, <span data-field="fld4">3</span>, <span data-field="fld5">4]</span>. Why a span and not a comment or a {…} token:
   • A comment can't show extent. Which preceding characters are the field? For 4] the bracket is inside the result, and for p56 the field is a whole sentence. And comments may never round-trip, by invariant.
   • A {fld3:1} token would change the offset space. find "Figure 1" would stop matching, and every existing pN:S-E and comment anchor after the field would shift. A span is zero-width markup, exactly like the formatting spans that already ride <span>.
   • A span can round-trip as a re-binding. bug(edit): whole-paragraph edits silently flatten fields to frozen text; span replace leaves orphan fldChar #42 makes data-field="fldN" in edit markdown refer back to the live field of that id (the same pattern as ![alt](sha256.ext) re-stating an existing image, and the known-note fix feat(move): relocate blocks in place — untracked splice + tracked w:moveFrom/w:moveTo #38 asks for with [^fn1]). It never mints a field from markdown, because DDEAUTO/INCLUDETEXT "http://…"/INCLUDEPICTURE would make field authoring through markdown an injection surface. An unknown fldN is an error, never literal text.
   • It merges with an existing formatting span (<span style="…" data-field="fld3">) instead of nesting.
   • Shown kind: data-field="fld3" is the handle. Whether to also print the kind or target inline (data-field="fld3 ref:_Ref279753835") is an open question, settled by eval:read's signal ratio, not by taste. The full record is one fields list away either way.
3. HYPERLINK fields → markdown links: [linkedin.com/in/jane-smith](http://www.linkedin.com/in/jennifer-smith-of9e839), identical to a <w:hyperlink>. The display/target mismatch becomes visible.
4. Generated multi-paragraph fields (TOC, INDEX, BIBLIOGRAPHY): bracket them like a text box story: <!-- docx:field fld0 toc instr="TOC \o &quot;1-3&quot; \h \z \u" --> before the first paragraph and <!-- docx:field-end fld0 --> after the last, with the entries as plain text. Nested PAGEREF/HYPERLINK spans are suppressed inside a generated region, because the bracket already says "regenerated by Word, don't hand-edit" and per-entry spans would triple the TOC's size.

read --current shows a tracked field insertion or deletion as CriticMarkup around the whole token or span, like an equation.

4. Offset width: the equation rule, split by shape

Kept in lockstep on the AST side (runViewText in find/index.ts, the reader's comment offsetRef) and the XML side (runTextLength/sumRunBearingTextLength/inlineMarkerWidth and every walker that counts equations), from ONE helper in core/fields, as equationToken does for equations:

• fldChar, instrText/delInstrText, and anything inside an instruction (including nested fields' results) have width 0. The instruction is never in the find/replace text, which is already true today.
• Volatile field: width = its token ({page} = 6), and atomic. A span that covers the token moves or deletes the whole field. A span that cuts it is refused (fieldSpanRefusal, mirroring equationSpanRefusal). There is no "inside".
• Content field: width = its result text, exactly as today, so existing anchors, wc and find don't move. The begin/end boundaries are the atom edges. A span strictly inside the result, a span covering begin through end, and a span cutting one boundary are the three cases. Mutators decide what each means (bug(edit): whole-paragraph edits silently flatten fields to frozen text; span replace leaves orphan fldChar #42). Placement walkers that can't refuse snap outward: comments add --anchor "1, 3" brackets the whole of fld3 and fld4, instead of ending between fld4's result and its end the way it does today, and a footnote reference lands after the field's end.

5. Discovery

• docx fields list FILE [--json] → [{id, kind, instruction, result, blockId, …}], the id being the handle (house list contract). Add a top-level docx --help one-liner. Weak agents find features there.
• fldN is a locator for raw get --at fldN (serialize begin…end) and comments add --at fldN. delete --at fldN and the authoring verbs are left to bug(edit): whole-paragraph edits silently flatten fields to frozen text; span replace leaves orphan fldChar #42 and follow-ups.
• find output can name the field a match sits in (p34:152-161 (in fld3, fld4, fld5)) under --json.

6. Stale cached results: preserve, don't fake, don't nag

We never lay out pages, so we can't recompute PAGE/PAGEREF/NUMPAGES, and after an edit the cached results of SEQ/REF/TOC may be stale. Findings so far:

• w:dirty="true" on a field's begin fldChar makes Word for Mac block on open with a modal: "This document contains fields that may refer to other files. Do you want to update the fields in this document?" Verified: one dirty SEQ field in an otherwise clean document wedged docx render --engine word until its 60 s open timeout, and left the modal on screen for a human to click. <w:updateFields w:val="true"/> in settings.xml is documented to raise the same prompt (not separately probed). Either one makes Word ask every person who opens the document that question, and the wording ("may refer to other files") reads as a security warning to a recipient. Opening a contract, or sending one, and getting a scary prompt is a real cost, and it breaks unattended Word automation, including our own render. So neither may be set by default, or as a side effect of an ordinary edit. If we ever offer it, it's an explicit opt-in (fields update --on-open?) whose help states the prompt.
• Word's PDF export recomputed every stale cached value we planted (PAGE 99→1, NUMPAGES 55→1, SEQ 77→1, DATE 1999-01-01→today, REF → "Error! Reference source not found."). LibreOffice recomputes all of them on import too. So docx render shows recomputed values that can differ from what read shows (the cached ones). That's expected, not a bug, but render can't be used to test a stale cache. What Word shows on screen before an update (cached REF/SEQ until F9, as far as we know) is unverified.
• Policy for this issue: read shows the cached content result (it's what the file holds and what the screen shows) and the token for volatile kinds (whose cache is always stale somewhere). Mutators never edit cached results behind the agent's back. Follow-up (not this issue): docx fields update that recomputes the deterministic kinds in place (SEQ numbering in document order with \r/\c/\s; REF to a bookmark's text; NOTEREF) and reports what it can't (page-dependent kinds, \r paragraph numbers), without setting dirty.

7. Headers/footers (dependency: #37, then #39)

The parser is part-agnostic, so MarginalsView parts get the same field records. But marginalText only walks direct <w:p> children of the part root, so the content-control page numbers (academic-paper, large-mixed) and tables in headers stay invisible until #37 makes headers block containers. This issue ships the parser + fieldToken. #39 switches marginals onto it. Don't patch marginalText with a second field walker in the meantime.

Implementation notes

• src/core/fields/: parse.ts (the run-child tokenizer + stack, producing begin/separate/end markers and the parsed instruction), token.ts (fieldToken, isVolatile, the width helper shared by both sides of the offset bridge), instruction.ts (keyword/args/switches).
• Reader: readRun/walkRunContainer consult the parser state in WalkState. In instruction phase they suppress text (today instrText is already skipped). In result phase they decorate content-field runs or accumulate a volatile field's result into its FieldRun. They register fieldReferences in the same single walk. Notes, text boxes and cells reuse it via collectScopedBlocks.
• Offset bridge: a new walker obligation in src/core/CLAUDE.md beside the equation rule. "Adding an offset walker: count a volatile field with fieldWidth, never its cached result." <w:fldSimple> stays in RUN_BEARING_WRAPPER_TAGS for content kinds. For volatile kinds the walkers stop descending its runs and count the token instead (one predicate, both sides).
• Markdown: cli/read/markdown.ts emits the token/span/link. gatherHtmlSpans learns data-field (merge with style spans). formatNote gains the docx:field/docx:field-end bracket.
• hyperlinks lens: a HYPERLINK-field linkN lists with field: "fldN". Whether hyperlinks replace/delete rewrite the instruction's URL / unwrap the field in v1 or refuse with a pointer is an open question below. Either way it must not hit the rId path.

Tests and coverage

• Unit (tests/core/fields/*.test.ts): parser cases: every committed fixture row in the table; instr split across runs; all fldChars in one run; no separate; nested (IF{MERGEFIELD}, TOC{HYPERLINK{PAGEREF}}); begin outside / result inside <w:ins>; tracked deletion with delInstrText; spanning two paragraphs; cell- and story-scoped; malformed (orphan end / unclosed begin); fldSimple parity with the complex form.
• Offset parity: for each fixture, find offsets == XML walker offsets across every field (the existing equation parity tests are the template). Comment --anchor over a field snaps outward.
• eval:read: snapshot diff reviewed as a product change. p34/p56/p62, the résumé link and the TOC fixture are the cases to look at. Watch signalRatio on multi-column (19 REFs).
• Fixture: a new tests/fixtures/setup/fields.ts (no existing fixture fits, since nothing authors fields yet) that dogfoods raw insert to plant each shape above, plus fields list in its assertions, and is added to CORE_FIXTURES so it round-trips LibreOffice. Include a TOC, a SEQ-numbered caption pair, a REF to it, a field in a cell, in a text box and in a footnote, a HYPERLINK field, and a fldSimple.
• Weak-agent scenario: fold "my LinkedIn link points at an old profile, fix it to …" into resume (needs a résumé source with a HYPERLINK field, e.g. the resume-styling one). criteria.md checks the field's instruction URL changed and the display text stayed.
• validate: a fixture-clean pin for the new unbalanced-field check (no false positives on any committed fixture).
• Docs: REFERENCE.md (read output, fields list, fldN), src/core/CLAUDE.md (offset bridge rule), src/core/ast/CLAUDE.md, src/cli/CLAUDE.md.

Open questions

1. Span payload: bare data-field="fld3", or kind and target inline (fld3 ref:_Ref279753835)? Decide on eval:read numbers. The cross-reference authoring issue will want the target visible.
2. Volatile token switches: {date} drops \@ "d MMMM yyyy". Keep tokens bare (the record has the instruction) or {date:d MMMM yyyy}? Same question as feat(page-numbers): read Word's page numbers as fields, edit them in place, and set start/format #39's PAGE \* roman.
3. HYPERLINK-field verbs: should hyperlinks replace rewrite the instruction URL in v1 (likely what the résumé task needs), or refuse until a follow-up?
4. Legacy form fields (FORMTEXT/FORMCHECKBOX/FORMDROPDOWN with ffData): common in legal and HR templates. Should FORMCHECKBOX read as a GFM task checkbox, and is that this issue or the task-list package?
5. Mid-paragraph docx:field for content fields in generated regions: is the TOC bracket enough, or do INDEX/BIBLIOGRAPHY need their own treatment?
6. wc: proposed that a volatile field's token counts 0 words, as an equation already does (wc on equations.docx p2, $x^2$, is 0). A content field counts its result words, as today.