add trustless IPFS gateway, use patched rclone, remove bitswap#646
add trustless IPFS gateway, use patched rclone, remove bitswap#646
Conversation
fe93173 to
adfce2b
Compare
51b8666 to
a8d4dd4
Compare
remove the dead bitswap protocol and add an /ipfs/ endpoint implementing the trustless gateway spec via boxo/gateway. StorageBlockStore pools rclone handlers per storage and holds open a streaming file reader to serve sequential block reads efficiently -- DAG nodes return from the DB without storage I/O, and leaf blocks from the same file share a single held-open rclone stream. supports dag-scope=all/entity/block, path resolution, entity-bytes, format=car and format=raw. stale source files detected at read time are mapped to HTTP 409 via boxo's error status mechanism.
replace github.com/rclone/rclone with parkan/rclone@singularity-v1.68.0 which adds sync.Map + singleflight caching for /metadata/ API calls in the internetarchive backend. reduces per-item metadata hits from N (one per file) to 1. temporary measure until metadata caching is upstreamed to rclone.
- remove uptobox backend (dropped upstream) - fix fs.Duration type change in config overrides - handle []string option defaults in ToCLIFlag type switch
ca2e329 to
c55bddf
Compare
|
I found 3 issues worth addressing in this PR:
|
remove the dead bitswap protocol and add an /ipfs/ endpoint implementing the trustless gateway spec via boxo/gateway
StorageBlockStore pools rclone handlers per storage and holds open a streaming file reader to serve sequential block reads efficiently -- DAG nodes return from the DB without storage I/O, and leaf blocks from the same file share a single held-open rclone stream
supports dag-scope=all/entity/block, path resolution, entity-bytes, format=car and format=raw. stale source files detected at read time are mapped to HTTP 409 via boxo's error status mechanism