Skip to content

OSM diff-sync implementation #63

@koinsaari

Description

@koinsaari

Goal

Implement the diff-sync subcommand of cmd/ingestion, currently stubbed to return an error. After a one-time full-import, diff-sync brings the registry up to date with incremental upstream changes — without re-reading the entire .osm.pbf.

Tracking under #10.

Approach options

  • OSM diff files: OpenStreetMap publishes minutely / hourly / daily diff files (.osc.gz). Pull a window of diffs, apply them. Standard approach; library support exists.
  • Overpass periodic re-query: re-query a bounded region via Overpass on a schedule. Simpler but expensive and harder to keep precisely consistent.

Recommend OSM diff files for correctness; Overpass only as a fallback for spot-checking.

Open questions

  • Diff cadence: minutely / hourly / daily? Daily is plenty for accessibility data.
  • State tracking: where do we persist "last applied sequence number"? A small ingestion_state table?
  • Failure recovery: if a diff application fails mid-batch, how do we resume cleanly?
  • Deletions: OSM diffs include node deletions. Do we hard-delete from places, soft-delete via place_status, or skip deletions entirely until the precedence rules issue is settled?

Out of scope

  • Diff-sync for non-OSM sources (not on the roadmap yet).
  • Backfill of historical diffs older than the full-import cutoff.

Acceptance

  • cmd/ingestion diff-sync applies pending OSM diff files and updates places accordingly.
  • Last-applied-sequence is persisted across runs.
  • Idempotent: re-running with no new diffs is a no-op.
  • Has at least one integration test covering a small diff fixture.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ingestionOSM and other data source ingestionenhancementNew feature or requestpriority:shouldShould-have, rough edges

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions