Skip to content

add trustless IPFS gateway, use patched rclone, remove bitswap#646

Open
parkan wants to merge 7 commits intomainfrom
feat/trustless-gateway
Open

add trustless IPFS gateway, use patched rclone, remove bitswap#646
parkan wants to merge 7 commits intomainfrom
feat/trustless-gateway

Conversation

@parkan
Copy link
Collaborator

@parkan parkan commented Mar 17, 2026

remove the dead bitswap protocol and add an /ipfs/ endpoint implementing the trustless gateway spec via boxo/gateway

StorageBlockStore pools rclone handlers per storage and holds open a streaming file reader to serve sequential block reads efficiently -- DAG nodes return from the DB without storage I/O, and leaf blocks from the same file share a single held-open rclone stream

supports dag-scope=all/entity/block, path resolution, entity-bytes, format=car and format=raw. stale source files detected at read time are mapped to HTTP 409 via boxo's error status mechanism

@parkan parkan force-pushed the feat/trustless-gateway branch from fe93173 to adfce2b Compare March 17, 2026 09:57
@parkan parkan marked this pull request as draft March 17, 2026 10:07
@parkan parkan changed the title add trustless IPFS gateway, remove bitswap add trustless IPFS gateway, use patched rclone, remove bitswap Mar 17, 2026
@parkan parkan marked this pull request as ready for review March 17, 2026 15:33
@parkan parkan force-pushed the feat/trustless-gateway branch 3 times, most recently from 51b8666 to a8d4dd4 Compare March 17, 2026 16:01
parkan added 7 commits March 26, 2026 16:56
remove the dead bitswap protocol and add an /ipfs/ endpoint implementing
the trustless gateway spec via boxo/gateway.

StorageBlockStore pools rclone handlers per storage and holds open a
streaming file reader to serve sequential block reads efficiently --
DAG nodes return from the DB without storage I/O, and leaf blocks from
the same file share a single held-open rclone stream.

supports dag-scope=all/entity/block, path resolution, entity-bytes,
format=car and format=raw. stale source files detected at read time
are mapped to HTTP 409 via boxo's error status mechanism.
replace github.com/rclone/rclone with parkan/rclone@singularity-v1.68.0
which adds sync.Map + singleflight caching for /metadata/ API calls in
the internetarchive backend. reduces per-item metadata hits from N (one
per file) to 1.

temporary measure until metadata caching is upstreamed to rclone.
- remove uptobox backend (dropped upstream)
- fix fs.Duration type change in config overrides
- handle []string option defaults in ToCLIFlag type switch
@parkan parkan force-pushed the feat/trustless-gateway branch from ca2e329 to c55bddf Compare March 27, 2026 10:04
@parkan parkan requested a review from anjor March 27, 2026 10:27
@anjor
Copy link
Collaborator

anjor commented Mar 27, 2026

I found 3 issues worth addressing in this PR:

  1. handler/wallet/export_keys.go
    migrateWalletAssignments skips unresolved legacy wallet_assignments rows, but still drops the table afterward. That makes the first export-keys run destructive for malformed keys, failed exports, or any actor that never gets a corresponding wallets row, because the preparation-wallet link is lost permanently. I think this should either fail when unresolved rows remain, or preserve the legacy table until every row has been migrated successfully.

  2. store/storage_blockstore.go
    The cached s.active reader is shared across requests, but it was opened with the ctx from whichever request populated it. readerWithRetry stores that context and checks it on every read and reopen, so a later request can inherit a canceled or timed-out context from an earlier one and fail spuriously. Reusing the stream only within a single request, or decoupling the cached reader from request-scoped context, would avoid that.

  3. store/storage_blockstore.go
    The mutex in readFileBlock is held across handler.Read, object validation, and the actual block read. That means one slow backend read serializes every /ipfs/ request through this process-wide blockstore. Under concurrent gateway traffic this looks like a single-reader bottleneck. I think the critical section should be narrowed to cache bookkeeping only, or moved to per-request state instead of a global lock.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants