README.md

kdp-data

Dataset index + Polars tabular loaders for kdp-processed Kalshi market data (the Parquet per-ticker directories kdp-process writes — see docs/data-guide.md at the repo root).

Tabular reads only: deterministic ordering, trade dedup, and book replay live in the Rust kdp-load crate — usable from Python via the kdp_load bindings (see "Full-depth replay" below). A trades frame from a directory holding both WS and REST copies of a print contains both rows.

Install

uv add "kdp-data @ git+https://github.com/PCeltide/snapevent#subdirectory=python"
# or, inside this repo:
cd python && uv sync

Use

from kdp_data import DatasetIndex, load_trades, load_book_top

idx = DatasetIndex.build("D:/data/kdp")        # walk-on-demand, no cache
idx.to_frame()                                  # the index as a Polars frame
df = load_trades(idx.entries(ticker="..."))     # list -> concat + source_path

# Incomplete captures refuse to load unless acknowledged (R3):
load_trades(entry, allow_incomplete=True)

# Day tarballs (all-2026 backfill) index lazily; extract on demand:
from kdp_data import extract_day_tars
dirs = extract_day_tars(idx.tar_entries(), "D:/data/extracted")

Coverage — is the data trustworthy?

from kdp_data import coverage, holes

cov = coverage(idx.entries())          # one row per dir: span, hole_us, uptime
holes(entry)                           # one row per gap: where the hole opened/closed

Coverage deliberately does NOT apply the R3 gate — it is the tool that reports trustworthiness, so it reads incomplete dirs too and surfaces complete/reasons as columns. A hole runs from a gaps row to the first re-anchoring snapshot after it (unresolved holes run to the span end); overlapping holes are unioned and clamped to the observed span, so uptime stays in [0, 1] — a gap outside the span signals via n_unresolved_gaps; trades-only dirs report null gap metrics (uptime is an L2-capture concept). Coverage also carries the manifest's verification stats as nullable columns — verify_checks / verify_mismatches (REST cross-checks of the replayed book against the venue's own order book) and underflows (deltas that drove a level strictly below zero during replay). They are null for directories processed before those fields existed — absence of a check is never shown as a passing one. A verify_mismatch gaps row opens a hole like any other gap. Rollups are one group_by away:

cov.group_by("date").agg(pl.mean("uptime"), pl.sum("hole_us"))

Example: uv run python examples/btc_hourly_book_top.py --root <tree>.

Full-depth replay — kdp_load (compiled bindings)

crates/kdp-load-py wraps the canonical Rust replay in a Python module (pyo3/maturin, needs a Rust toolchain to build). From the repo root:

powershell -File scripts/check-load-py.ps1   # maturin develop into .venv + smoke

from kdp_load import Loader, IncompleteData

ld = Loader(path)                    # R3 gate, like the loaders above
for ev in ld.events(): ...           # merged time-ordered dicts, integer units
for ev in ld.between(t0_us, t1_us): ...  # synthetic opener first
ld.book_at(t_us)                     # the FULL ladder at one instant:
                                     # {"yes": [(price_micro, qty_centi)...],
                                     #  "no": [...], "suspect_gaps": [...]}

Non-empty suspect_gaps means a capture hole since the last re-anchoring snapshot — treat that ladder with suspicion. Worked example: uv run python examples/full_depth_replay.py (runs against the committed fixture, no data needed).

Dev: uv run pytest / uv run ruff check . (or powershell -File ../scripts/check-py.ps1 from python/, scripts/check-py.ps1 from the repo root).