Skip to content

Commit d180155

Browse files
ElektrikAkarclaude
ElektrikAkar
and
claude
committed
Update progress summary, delete obsolete plan/report files
- Updated progress-summary to cover all completed phases (0, 1, 2, 2.5, 4, 10, HPC fixes) with remaining phases and next steps - Deleted obsolete files: PLAN_lemire_envelope.md (documented in LESSONS), 3 adversarial review sub-agent reports (findings incorporated long ago) - Cleaned up untracked session-specific report files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent c73fec3 commit d180155

5 files changed

Lines changed: 147 additions & 1300 deletions

‎.claude/PLAN_lemire_envelope.md‎

Lines changed: 0 additions & 45 deletions
This file was deleted.

‎.claude/reports/progress-summary-phases-0-2.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# DTWC++ Transformation Progress Summary
2+
3+
**Date:** 2026-03-29
4+
**Branch:** Claude (ahead of main by ~40 commits)
5+
**C++ Tests:** 34 passing (+ 3 pre-existing pruned DM failures)
6+
**Python Tests:** 100 passing (0.26s)
7+
**Context:** This document captures everything accomplished for continuity across context resets.
8+
9+
---
10+
11+
## Completed Phases
12+
13+
### Phase 0: Critical Bug Fixes (COMPLETE)
14+
- 24 bugs fixed: `throw 1`, ODR violation, `.at()` in hot loop, dtwBanded 512MB allocation, int overflow, DBI formula, broken examples, PAM mislabeled
15+
- 12 CMake fixes: option() syntax, output dirs, Armadillo pin, CONFIGURE_DEPENDS
16+
- CI fixes: uncommented main triggers, ASan+UBSan job
17+
- Python packaging: fixed pyproject.toml, deleted conflicting setup.py, created VERSION file
18+
19+
### Phase 1: Core Architecture Refactor (COMPLETE)
20+
- 9 new headers in `dtwc::core`: ScratchMatrix, TimeSeriesView, DistanceMetric, DTWOptions, dtw.hpp, DenseDistanceMatrix, ClusteringResult, z_normalize, lower_bounds
21+
- FastPAM1 algorithm (Schubert & Rousseeuw 2021, JMLR)
22+
- Documentation: algorithms.md, metrics.md
23+
24+
### Phase 2: Performance — Memory First (COMPLETE)
25+
- Column-major ScratchMatrix, NaN sentinel DenseDistanceMatrix
26+
- LB_Keogh + LB_Kim implementations with property tests
27+
- Early abandon parameter in dtwFull_L
28+
29+
### Phase 2.5: Core Integration (COMPLETE)
30+
- Wired all core types into production code
31+
- **Armadillo fully removed** from library (zero references, not linked)
32+
- FastPAMResult unified with ClusteringResult
33+
- z_normalize tests now use production code
34+
- Pruned distance matrix re-enabled
35+
- DenseDistanceMatrix owns its I/O (write_csv, read_csv, operator<<)
36+
- 7 adversarial test suites (~30K assertions)
37+
38+
### Phase 4: Python Bindings (COMPLETE)
39+
- **nanobind** (not pybind11) — GPU-native ndarray, stable ABI, 128KB wheels
40+
- scikit-build-core build system
41+
- DTWClustering sklearn-compatible class (fit/predict/fit_predict)
42+
- All DTW variants exposed: dtw_distance, ddtw, wdtw, adtw, soft_dtw
43+
- DenseDistanceMatrix zero-copy to_numpy()
44+
- 100 Python tests + 15 cross-validation tests + 3 examples
45+
- CI: python-tests.yml (every push), python-wheels.yml (main + tags only)
46+
47+
### Phase 10: DTW Variants (COMPLETE)
48+
- DDTW (derivative preprocessing) — warping_ddtw.hpp
49+
- WDTW (weighted, logistic) — warping_wdtw.hpp (rolling buffer banded)
50+
- ADTW (amerced, penalty) — warping_adtw.hpp
51+
- Soft-DTW (differentiable, gradient) — soft_dtw.hpp
52+
- DTWVariant enum + DTWVariantParams in dtw_options.hpp
53+
- Problem::set_variant() with std::function dispatch
54+
55+
### HPC Performance Fixes (COMPLETE — Sub-phase A)
56+
- **Band rebind bug fixed** — lambda captures `this` not `band` by value (14x fillDM speedup)
57+
- **std::min({a,b,c}) → nested std::min** — 2.5-3x all DTW functions
58+
- **Branchless LB_Keogh** — max(0, max(q-U, L-q))
59+
- **FastPAM SWAP parallelized** — OpenMP on outer candidate loop
60+
- **compute_nearest_and_second parallelized** — OpenMP static schedule
61+
- **std::reduce** for reductions, multiply by 1/stddev, redundant add removal
62+
- **WDTW banded rolling buffer** — 128MB → 32KB for n=4000
63+
- **Early abandon running min** — O(1) per row instead of O(n) min_element
64+
- Roofline analysis: DTW is latency-bound on recurrence (10 cycles/cell), not memory-bound
65+
66+
### Cleanup & CI
67+
- Deleted obsolete files: main.py, test.py, .python-version, uv.lock, develop/TODO.md, setup.py
68+
- Merged benchmark/ into benchmarks/
69+
- CMake: VERSION file, complete PUBLIC headers, absolute test paths, -fPIC
70+
- Bumped Python >=3.10, numpy >=1.26, benchmarks/pyproject.toml (uv compatible)
71+
- README: Python tests badge
72+
73+
---
74+
75+
## Remaining Phases
76+
77+
| Phase | Scope | Status | Priority |
78+
|-------|-------|--------|----------|
79+
| Sub-B | SIMD via Google Highway | **Planned** | High — 3-4x multi-pair DTW, 4-8x LB_Keogh |
80+
| Sub-C | MPI + CUDA | **Planned** | High — required for N>10K scale |
81+
| 3 | GPU/CUDA kernels | Planned (part of Sub-C) | High |
82+
| 5 | FastCLARA + MPI | Planned (part of Sub-C) | High — required for 100M series |
83+
| 6 | Checkpointing (HDF5/CSV) | Not started | Medium |
84+
| 7 | Missing data (DTW-AROW) | Not started | Medium |
85+
| 8 | I/O — HDF5 + Parquet | Not started | Low |
86+
| 9 | MATLAB bindings (C++ MEX) | Not started | Low |
87+
| 11 | Build system + CLI | Not started | Low |
88+
89+
### Sub-phase B: SIMD via Google Highway (NEXT)
90+
91+
Layered parallelism architecture designed:
92+
```
93+
Level 2 (MPI): P ranks across nodes — pair blocks
94+
Level 1 (OpenMP): T threads per rank — dynamic scheduling
95+
Level 0 (SIMD): Highway vectorization (AVX2/512/NEON runtime dispatch)
96+
Level 3 (CUDA): Alternative GPU path
97+
```
98+
99+
Key items:
100+
- B0: Highway CPM integration + DTWC_ENABLE_SIMD option
101+
- B1: SIMD LB_Keogh (4-8x, called O(N²) times)
102+
- B4: Multi-pair DTW — 4 pairs in AVX2 lanes (3-4x fillDistanceMatrix)
103+
- B2/B3: SIMD envelope, derivative stencil
104+
105+
### Sub-phase C: MPI + CUDA
106+
107+
- C1: MPI distance matrix partitioning + MPI_Allreduce
108+
- C3: CUDA batch DTW kernel (one block per pair, anti-diagonal wavefront)
109+
- C4: nanobind nb::ndarray<T, nb::device::cuda> GPU tensor interop
110+
111+
---
112+
113+
## Key Architecture Decisions
114+
115+
| Decision | Rationale |
116+
|----------|-----------|
117+
| nanobind over pybind11 | GPU-native ndarray, stable ABI, 5-10x smaller wheels, 68-line sunk cost |
118+
| C++ MEX API over legacy C | No longjmp, RAII-safe, typed arrays |
119+
| Google Highway for SIMD | C++17, runtime dispatch, broadest ISA, built-in exp() |
120+
| Separate functions per DTW variant | Recurrence differs structurally; conditionals hurt memory-bound kernel |
121+
| std::function dispatch in Problem | ~2ns overhead negligible vs 1-100ms DTW |
122+
| Column-major ScratchMatrix | Matches DTW inner loop access pattern |
123+
| NaN sentinel for distance matrix | Handles future negative-distance metrics |
124+
| O(n×band) envelope scan | Cache-friendly for small band; Lemire O(n) deferred |
125+
126+
---
127+
128+
## Test Summary
129+
130+
| Suite | Count | Status |
131+
|-------|-------|--------|
132+
| C++ unit tests (Catch2) | 34 | 31 pass, 3 pre-existing pruned DM failures |
133+
| C++ adversarial tests | 7 suites (~148 cases) | All pass |
134+
| Python tests (pytest) | 100 | All pass (0.26s) |
135+
| Cross-validation (C++ ↔ Python) | 15 | All pass |
136+
137+
## Performance Summary (24-core Intel @ 2496 MHz)
138+
139+
| Benchmark | Before | After | Speedup |
140+
|-----------|--------|-------|---------|
141+
| dtwFull n=4000 | 238ms | 97ms | 2.5x |
142+
| dtwBanded n=1000 band=10 | 320us | 108us | 3.0x |
143+
| fillDM N=50 L=500 band=10 | 54ms | 3.95ms | **14x** |
144+
| fillDM N=50 L=1000 band=50 | 214ms | 32.8ms | **6.5x** |
145+
146+
DTW is latency-bound on recurrence chain (10 cycles/cell, 265M cells/sec).
147+
Multi-pair SIMD (Sub-phase B) expected to give 3-4x additional throughput.

0 commit comments

Comments
 (0)