Skip to content

Improve Alden original-file search, reading and knowledge graphs; adopt alden-cli - #48

Draft
twoimo wants to merge 727 commits into
mainfrom
codex/alden-explicit-references-20261025
Draft

twoimo wants to merge 727 commits into
mainfrom
codex/alden-explicit-references-20261025

Conversation

@twoimo

@twoimo twoimo commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Alden now searches complete retained UTF-8 TXT/Markdown originals, including
matches beyond the parser's16,000-byte projection and across that boundary.
A separate reversible original index retains general text/vector policies.
Current project/target/version/raw/config/capture bindings admit candidates
before limits, and selected query-centered excerpts are rebuilt from verified
original bytes. Their proof separates whole-input/projection hashes, delivered
text and exact or disclosed covering original byte ranges. Initial parser
coverage remains independent. Capture commits before optional derived indexing;
index failures report a separate failed stage, and lookup never builds an index.

Original ranks replace parser ranks for the same file, with one interleaved
lexical budget. Selected-file term contributions center multi-term excerpts so a
common head match cannot hide the discriminating tail. Cached scores remain
advisory. Final source validation inherits cancellation and checks again before
returning. Typed knowledge authority rebuilds the selected original excerpt and
raw range without rerunning search. Voice keeps its actual match within the
existing fact limit. Unicode query terms preserve accented letters and combining
marks while retaining the prior ASCII/Hangul particle rules.
Korean compound-particle queries append roots for six closed suffixes only
when the existing24-term budget has spare slots. All legacy terms remain the
same prefix; source identity, scope and the graph literal filter stay intact.

Explicit project lookup prompts separate a literal project/source wrapper and
requested output format from the content search terms. Existing host intent and
project resolution run first; unknown syntax keeps the original query. A bounded
hash-only trace binds request and retrieval text without replacing user messages
or widening permissions.

Exact retained-original body windows, desktop first/next/reset and the read-only
alden_knowledge_source_body MCP tool remain available with their prior bounded
payloads, no original-path fallback and current source authority checks.

Original-file reads also preserve keyboard focus during asynchronous loading.
The busy guard rejects repeat activation while accessible disabled states keep
the focus target. At EOF, Next transfers focus to Reset only when it is still
focused; another control selected during the read keeps focus. Source and byte
verification, stale-result fencing and callback contracts remain intact.

alden-cli becomes the canonical internal command across execution, packaging,
help and completions. The existing compatibility executable and persisted app,
data and permission identities remain supported. The interface uses Hallmark's
warm navy direction, with explicit partial, stale and unknown states.

Knowledge requests bind exact model routes, source authority and retained
versions. Typed requests permit at most two model hops and one authorized source
read, with bounded nonstream responses and one final commit. Malformed tool
calls, truncation, source changes and incomplete replies stay typed failures.
Rejected typed local completions retain bounded HTTP finish/shape/call-count
observations before strict parsing. Duplicate calls still cause zero source
reads. Sanitization excludes content, arguments and arbitrary wire values;
diagnostic compaction respects the existing4 KiB receipt limit after query/
attachment context and late failure-code fields are added.

Native inventory and observed local endpoints are validated separately from
router aliases; partial router availability does not create remote health or an
automatic fallback.

The new explicit file-set adapter reads declared original documents rather than
treating Spark's derived index as original-file acquisition. It retains stable
file IDs, exact original bytes, revisions and structural locations; equal-byte
copies share a blob without merging identity. Acquisition failures cannot prune
the prior set. Verified-root descriptor reads and pre-publication source/target
guards reject races. PPTX linked notes are read with real slide/part/paragraph
positions. Coverage is explicit: Office/PDF/conversion extraction stays
conservatively incomplete, even when original-byte capture is complete.
Manifest, limits and evidence.

Graph and terminal changes retain source provenance through GBrain/open-ontologies
integration and normalization. The ordinary relationship budget stays 512. A
bounded integer-coordinate neighbor table improves the pure pin solver while
preserving results; other coordinates retain the original lookup. Worker faults
release ownership and invalidate late callbacks. A subsequent explicit solve
creates a fresh worker. The terminal f(x) counts deduplicated permitted stored
document transitions by UTC interval and binds its full snapshot checkpoint
separately from history paging. It does not infer original-source completeness
or live intake from node modification dates.

Validation:

  • Rejected knowledge replies at b848633: 39 new regression methods /
    254 subcases
    pass with generated sources and mocked transport; the normal
    repository unittest module command also passes39 methods. The copied prior
    source detects absent observations. 106 related existing methods were
    accepted with only owned parser/image/CLI children and one ephemeral HTTP
    fixture admitted. Tests cover privacy, zero-read duplicate rejection,
    persisted failure/replay, positive tool/final/no-call cases and4 KiB pressure
    from query provenance, attachments and late errors. Optional diagnostic shapes
    or older observations can be explicitly omitted without changing primary
    evidence, actual calls or protocol acceptance. Exact-source CI
    passed 4/4 jobs. These are plumbing tests, with no real model or original-data tool execution or local
    app startup. The prior local7b4a57f candidate and installed0.3.219 are preserved;
    this change has no new local installed-runtime proof.
    Diagnostic contract.
  • Actual offline fixed27B diagnostic at 7b4a57f: the existing pinned
    checkpoint14 files/18,219,586,850 bytes match before and after;34 runtime
    distribution versions and5,751 RECORD hashes were verified before imports.
    Two real native loads/generations ran under the canonical exclusive lease
    and memory/state guards. Strict single-tool decision acceptance:0/2.
    Attempt001's specific rejection cause is unknown because failure text was
    not retained. Only diagnostic recording changed before a separately reviewed
    second attempt. Attempt002 stopped normally with382 prompt/34 generation
    tokens but emitted two identical empty tool calls. Exact token replay matches
    SDK text; the single-call guard correctly rejects the whole response without
    dedup/drop/repair. Both runs executed zero source reads/SQL/facts/final answers.
    Genuine source previews separately fail CANTOPEN: current WAL databases have
    no sidecars under the physical no-outside-write diagnostic profile. Owned
    generated controls support that condition; full source/final remains not
    admitted. No source write/checkpoint/immutable/nolock/raw-copy/backend bypass
    occurred. The2-attempt budget is exhausted. Source/helper/protected state,
    installed85-file app and Alden OFF are preserved. Independent native public
    receipt review agrees with NOT_ACCEPTED. This is native decoder-component
    failure evidence, not successful product HTTP/voice/installed integration.
  • Korean compound queries at 7b4a57f: 323/323 local methods pass,
    including28 new methods/173 subtests and11 existing Unicode methods;
    1,400 generated combinations retain the legacy prefix/cap/NFC behavior.
    The parent deny-all harness initially blocked one existing owned socket
    fixture; that test passed with only its ephemeral listener admitted, without
    changing product or existing test code. Exact-source
    CI40794626319
    passed 4/4 jobs, including unsigned arm64 package/runtime checks.
    After CI, a fresh40-file copy of the same22 retained originals was admitted
    against358 source pins and32 exact readiness files. A source-only annotator
    fixed12 questions/31 quotes from6 previously excluded originals before
    implementation; baseline scores stayed sealed until source freeze.
    Paired retrieval at2or4 entities/400-character original excerpts/8,000 context
    characters remained unchanged: source inclusion@1 10/12, @5 11/12,
    delivered source 11/12, all annotated quote ranges 2/12. No metric or
    individual case lost acceptance, but no measured gain on this reference
    packet
    is claimed. Both40-file states remain byte/mode/path identical;
    source/version/hash checks pass, outside-scope facts and guarded network/
    embedding calls are zero. These author-selected references may have a phase
    directory topic cue; they are not representative human semantic gold or
    actual voice/model answers. Old exposed references were not replayed.
    Lexical contract.
  • Keyboard reading at f0353df: 286/286 related reader/detail tests,
    full TypeScript and isolated frontend build pass. Exact-head
    CI40790119879
    passed 4/4 jobs, including unsigned arm64 bundle/runtime checks.
    Native Chrome keyboard activation reproduced the original focus loss on
    generated data; the correction preserves pending/non-EOF/reset focus, blocks
    duplicate reads, transfers EOF focus and respects a user focus change.
    After CI, one retained original in the22-document snapshot passed six
    first/next/reset transitions at light1024x768 and dark1440x1000. Native focus,
    DOM SHA-256, byte ranges and EOF matched an independent raw-capture oracle;
    all40 state files preserved bytes/modes/paths after cleanup. This is source
    and isolated component proof, not installed Alden/native/GPU/physical-input or
    actual model-answer acceptance. The frozen reference baseline was not tuned
    or replayed. Contract.
  • A separate source-only annotator froze16 questions and41 original-byte
    quotes from8 retained technical documents before lookup. At b87d009,
    indexed-lexical retrieval with5 entities/1200-character excerpts
    included the expected source first in10/16 and within five in16/16; all
    annotated support ranges fit the delivered excerpt in9/16. Applying the voice
    path's2-or-4 entity/400-character budgets delivered the expected source in
    13/16 and all support ranges in6/16. These are author-selected reference
    measurements, not representative semantic gold or actual model-answer
    accuracy. The graph UI's AND literal filter is a separate API. No query/answer
    tuning, embedding calls, product edits or native app startup occurred;40 state
    files and the frozen references remained unchanged. Evidence sufficiency,
    independent semantic/time/entity adjudication and actual answers remain open.
  • The existing file-list entry now has actual retained-data component acceptance.
    In a user-authorized isolated Chrome for Testing context, the source
    settings markup and public controller selection binding displayed12 files.
    Mouse, Tab/Enter and Tab/Space selected the same admitted reader target.
    Across two themes and two desktop sizes,12 first/next/reset transitions
    matched independent original-byte hashes and ranges for that one document.
    Backend list loads, body failures and console warnings/errors were zero;
    all40 retained state files preserved their bytes/modes/paths after cleanup.
    This is a component display/selection test; controller.start, the other11
    original bodies, native app/installed readiness and full semantic gold remain
    separate. That earlier checkpoint used b87d009 and its4/4 CI.
  • Full-original/scoped-query component at b87d009: 636/636 affected
    backend/packaging tests
    ,10 new methods/124 subtests and93 independent native
    checks pass with stable source pins. The preserved pre-change typed fixture
    selected the wrong head document; the corrected host query selects its unique
    retained tail and persists exact proof plus hash-only request/query trace.
    Existing intent, original messages, permissions and zero-argument tool remain.
    Unknown grammar falls back unchanged; NFC expansion preserves the old input
    truncation order. Prior generated original gold passes96 search conditions,
    24 graph/direct conditions and448 hash/scope checks.108 fresh-process affinity
    tests and independent148+34+9 owner/resource controls pass; the permanent SQL
    authorizer remains unchanged and legacy API behavior is preserved.
    Exact-head CI40770362365 passed4/4 jobs, including unsigned arm64 bundle
    and runtime-resource checks. The same byte-verified22 historical original
    snapshots (182,711 bytes) pass12/12 tail searches,6/6 graph queries,6/6 strict
    affinity references and12/12 MCP searches across both strategies. Before index,
    the same12 queries had0 hits. A corrected typed HTTP mock completes two hops
    with one authorized read, original hash/proof and durable derivation trace.
    All29 retained data files stay byte-identical; initial coverage remains
    20 complete/2 partial. No new original acquisition, current original-disk
    permission, real decoder, installed app or normal release is claimed.
    Native review is separate from the held formal external audit.
  • Current retained-original reader component at b87d009: two actual historical
    original snapshots (19,407 and28,078 bytes) across two themes and two desktop
    sizes pass24/24 first/next/reset DOM hash/range/EOF checks against a separately
    decoded raw-byte oracle. There are12 distinct layout cases and28 observations,
    with0 horizontal/source/console errors. All40 copied state files keep exact
    bytes and modes; source-only SQLite coordination timestamps are separate.
    Five native inline captures were inspected. Image-file export was rejected by
    browser URL policy and stopped; no alternate route was used. The overview
    samples12 source-file references and shows0 edges/visible caption labels;
    source-only entry affordance is under review, not accepted as full graph
    quality. This is source-component evidence, not native, installed, GPU,
    semantic-quality or real-decoder acceptance. The owned server/tab are closed.
  • Retained-original window source at dc0afcf: 460/460 affected backend tests
    and 322/322 focused frontend tests pass, with unchanged current source pins.
    Full TypeScript and isolated Vite pass. Real generated collection/capture/API
    fixtures reconstruct exact UTF-8 bytes after original paths disappear, reject
    reflected permission/version/config/capture changes and preserve independent
    file IDs for equal bytes. Boundaries cover BOM/NFD/CRLF/emoji, strict cursor and
    schema checks, whole-original hashes and no parser/network fallback.
    Independent review reproduced numeric exponent overflow in strict capture JSON;
    the fixed decoder passes6 numeric and3 generated-reader closure cases. Its
    earlier MCP overflow probe stopped at the existing abort frame and was not
    retried; no wire-level closure is claimed for that probe.
    Generated real HTTP/API + reader rendering passes28/28 states across two themes
    and two sizes, including exact displayed hashes, EOF/partial separation,
    permission denial and tampered response hashes. Existing source coverage and
    excerpts are retained; errors clear the prior visible original. These are
    component/fixture checks, not installed Alden or private-original acceptance.
    CI40741979176 at
    dc0afcfba8573b5540efc83b9ded61e20ffe83b2 passed4/4 jobs, including unsigned
    arm64 bundle/runtime checks. After CI, current stdio MCP over a byte-verified
    copy of all29 retained-state files read22 actual original snapshots through
    24 windows: all182,711 bytes and individual input hashes match the independent
    captured originals. Both initially partial documents (19,407 and28,078 bytes)
    reconstruct exactly; their original parser receipts remain partial after EOF.
    Stale versions, unknown targets, invalid cursors and outside scopes withhold
    bodies. State hashes remain identical; guarded network/embedding/original
    fallback calls are zero and the owned MCP server exits0. This is retained
    snapshot acceptance, not fresh acquisition, current original-disk permission,
    live app state, installed reader or actual model inference acceptance.
  • Original-file changes: 296/296 affected tests pass, including 53 parser and
    25 file-set integration methods; source hashes match before/after. Independent
    local review reproduced three defects, and independently reran their minimal
    closure probes after the fixes. All three are accepted within that scope.
    48/48 packaging/build-receipt tests pass. New modules are declared in the
    desktop resource maps and the prepared runtime includes their import closure.
    Focused Rust resource tests pass7/7. The whole-tree formatting check reports
    existing differences; the changed resource file has the same formatter diff
    as its c0b5d88 baseline. No broad reformat is included.
  • Seven generated formats pass the actual registration → collection → retained
    capture → scoped FTS path. PDFKit and textutil execute in owned parser children;
    all seven originals retain exact bytes, including linked PPTX notes. Only the
    small Markdown and CSV bodies claim complete text coverage. Owned readonly MCP
    stdio returns seven search results, seven graph nodes and seven version-bound
    details; stale/outside requests reject, state hashes match and guarded
    network/embedding calls remain zero. These are generated originals, not private
    Spark corpus or installed-desktop acceptance.
  • Removing either source-change or target-change publication guards in owned
    copied code makes its regression fail; unmodified controls pass. This is
    bounded mutation sensitivity, not an implementation-wide proof.
  • Exact source-head CI at 058bad1d60aa78af6350a816251347ecc2f40a3a passed
    4/4 jobs: run40715508627,
    including the focused suites and unsigned arm64 resource/runtime checks.
    After CI,22 actual technical Markdown originals (182,711 bytes) passed
    collection/capture into task-owned private state,22 per-file FTS checks and22
    version-bound MCP source reads. All22 original-byte hashes match the retained
    historical index. Replaying the unchanged set preserves22 versions and the
    capture identity.20 bodies are complete;2 exceed the16,000-byte parser bound.
    Actual source files, the current Spark index and live app state were not
    modified. This is a finite technical subset, not full Spark discovery.
  • Original-file coverage at 612e5c0: 421/421 affected backend tests and
    153/153 current reader tests pass; full TypeScript and isolated Vite pass.
    A bounded parser receipt binds the exact source, version, archive and delivered
    body. It reaches details, search/MCP provenance and the typed tool-role model
    input. Missing or invalid file coverage stays unknown; the existing response
    truncated flag still describes only the12,000-character excerpt. Reader
    notices precede long bodies and discard stale async hash results.
    Independent review reproduced a post-verification mutation gap; its25-case
    closure passes. Generated IAB component checks pass24/24 across two themes
    and two viewport sizes. Current read-only MCP over a byte-verified copy of the
    prior22 retained originals reports20complete/2partial with unchanged state,
    stale/outside rejection and zero guarded network/embedding calls. These are
    not fresh acquisition, installed-desktop or actual decoder acceptance.
    Exact source-head CI40732267381
    passed4/4 jobs, including unsigned arm64 package/runtime checks.
    Exact retained-original window source and post-CI acceptance follow below.
  • Earlier unchanged trend aggregation at c0b5d88: 336/336 related tests pass;
    independent proof/date cases match. Contract and limits.
    Its CI run40687380021
    passed 4/4 jobs. Original-data A/B admission saw no existing WAL/SHM in45s:
    zero SQL opens and zero accepted pairs; no live performance gain is claimed.
  • Earlier graph transport/layout suites and original-data component rendering
    retain their recorded revisions. Authorized isolated Chrome for Testing
    exercised pointer/keyboard selection, pin/reflow, bounded expansion,
    stop/resume and disposal over 1,984 nodes / 2,693 display groups. Actual hiding,
    native/GPU, physical Retina and fresh positive activity remain open.
    Failure contract,
    pin measurements.
  • Current local candidate 0.3.220 / 7b4a57f includes the original-file
    features. One normal build and one canonical check pass; 2,135 tracked
    source files
    match the primary checkout by bytes, modes and Git objects.
    The app has 103 files /97 script resources /89 Python files. All96 source
    resources match current source; the one declared generated native voice
    library matches the fresh normal build and its pinned Swift sources. An
    initial parent comparator wrongly compared that generated library with an
    older ignored binary in the primary checkout; correcting only that binding
    required no source change or rebuild. App arm64 identity, strict/deep
    signatures, installed designated-requirement continuity and both bundled
    CLI version outputs pass. ZIP 22,562,559 bytes, SHA256
    5584830b40b499dd091614ba8f2d5b9593b19a95ce10bd34b298554f7efb3e19;
    extracted103-file bytes/modes/signatures match. The previous99-file candidate,
    installed0.3.21985-file app, current settings and protected service snapshots
    are preserved. Independent source/resource/ZIP verification passes with seven
    in-memory invalid controls; these are not product mutation proofs. This is
    Apple Development signing with linker ad-hoc CLIs;
    Developer ID, notarization, installed runtime and release remain unverified.

Alden remains OFF. Broader Spark original-file acquisition still requires exact
source-scope reconciliation. Broader file-format/full-original search coverage,
full source/final fixed27B and IQ3.3 inference, latest GBrain promotion, semantic gold, native input/GPU,
installation and release remain open. Supplemental native/Argon feedback does
not replace the reserved formal web-Astra → Gemini → reverse review; that stage
is still held. No local model/service adoption, global route/account/permission/
SDK/capacity mutation, local app startup, notarization or release is included. The full
original18, follow-ups, ST01–16 and Cursor objective remain active. This draft
source candidate remains externally unvalidated for release.

Async extraction reference fence (2026-11-01)

A source-extraction hash could finish after a node or selection reference changed and evaluate an introduced getter. The verifier and its separate DOM continuation now check own property descriptors before reading those live references, keeping stale extraction status unknown and original-body text hidden.

The repair includes two deterministic delayed-sibling regressions and 26 reference-mutation cases across the hash/DOM boundaries. Local validation: all 1,864 frontend tests, TypeScript and Vite build passed. Exact commit 5f4843f1b577cfeb3e88fa33a112058a6b9c0590 passed all four CI jobs. Existing draft/release holds remain; this is not installed-app, live-model, browser/GPU or deployment verification.

twoimo added 30 commits October 19, 2026 01:41
Long saved titles stay readable at detail zoom without changing graph membership, while short subjects remain one line.
A stronger radial display map keeps stored positions and order while moving crowded inner nodes outward.
Fresh overview communities keep their direction and membership, while the biggest groups sit farther from the center than before.
The overview warning now follows the saved collection when that read succeeded, and still appears for memory-only or failed collection reads.
Off-center saved titles move away from the viewport center while central labels and stored positions stay put.
Blank labels, generic directory names and asset filenames remain stored and searchable, but they no longer take the first overview slot of a source.
Blank stored labels stay blank. The overview and focus views use the first substantive line only for display.
A longer sentence is preferred. A short saved subject still appears when it is the only line, and the stored label is not rewritten.
contains, in-group and import links stay stored and remain visible outside the overview. The overview draws the remaining topical links.
Generic media groups and one-word source groups were still the strongest overview stars after their labels were hidden. The overview now keeps those nodes and stored relations, but does not draw their spokes, so career topics stay connected.
A fresh overview was packing hundreds of members onto the inner ring, so separate topics landed in one ball. Large communities now sit farther along their own ray and open around that center.
Star hubs were pulled to the origin because their leaves surround them. A large overview now keeps each busy topic with its own notes in a small disk around a distinct direction.
A long retained graph note can end in a short run of bare numbers. The reading view collapses that tail, and the original text stays in the archive.
Graph notes imported with incremental_json were skipped by the legacy reading fold even when the retained post and its URL matched.
The same saved label can appear on many separate nodes. A large overview now draws those copies in one disk and leaves the stored identities unchanged.
Directory names such as overlay were repeated often enough to form their own disk. Hubs and code nodes without Korean stay stored, but they are no longer drawn as a topic.
@twoimo twoimo changed the title Adopt alden-cli and Hallmark UI; validate graph and local replies Adopt alden-cli and Hallmark UI; improve knowledge routing and graph layout Oct 27, 2026
@twoimo twoimo changed the title Adopt alden-cli and Hallmark UI; improve knowledge routing and graph layout Adopt alden-cli and Hallmark UI; improve knowledge routing, graph layout and trends Oct 27, 2026
@twoimo twoimo changed the title Adopt alden-cli and Hallmark UI; improve knowledge routing, graph layout and trends Improve Alden knowledge provenance, graph behavior and original-file coverage; adopt alden-cli Oct 28, 2026
@twoimo twoimo changed the title Improve Alden knowledge provenance, graph behavior and original-file coverage; adopt alden-cli Improve Alden original-file reading, knowledge provenance and graphs; adopt alden-cli Oct 28, 2026
@twoimo twoimo changed the title Improve Alden original-file reading, knowledge provenance and graphs; adopt alden-cli Improve Alden original-file search, reading and knowledge graphs; adopt alden-cli Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant