Improve profile computation memory efficiency#525
Improve profile computation memory efficiency#525schroedtert wants to merge 16 commits intoPedestrianDynamics:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Updates PedPy’s profile computation APIs to avoid requiring a precomputed full grid×pedestrian intersection matrix, enabling per-frame intersection computation to reduce peak memory usage for large datasets.
Changes:
- Adds
_compute_frame_grid_intersectionand updates profile methods to compute intersections per frame whengrid_intersections_areais omitted. - Refactors
compute_profilesto iterate frame-by-frame and share per-frame intersections between density and speed computations. - Updates plotting title handling, notebooks/user guide examples, and pre-commit tooling (including nbstripout metadata stripping).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
pedpy/plotting/plotting.py |
Ensures plot titles aren’t overwritten by plot_walkable_area defaults. |
pedpy/methods/profile_calculator.py |
Implements per-frame intersection computation and updates profile APIs/logic accordingly. |
notebooks/user_guide.ipynb |
Updates examples to use the simplified API without precomputing intersections. |
notebooks/getting_started.ipynb |
Strips notebook kernel metadata (via nbstripout configuration). |
notebooks/fundamental_diagram_at_measurement_line.ipynb |
Strips notebook kernel metadata. |
notebooks/fundamental_diagram.ipynb |
Strips notebook kernel metadata. |
.pre-commit-config.yaml |
Bumps tool versions and configures nbstripout to remove kernel/language metadata. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Benchmark resultsRan
Peak memory drops from 1.2 GB → 7 MB (~170× reduction) with a ~40% speed improvement on the combined path. The per-frame approach trades some compute overhead for dramatically lower memory usage, which was the goal. Benchmark script added in |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Used pre-commit autoupdate to get the dependency revisions to the current version. Additionally sync the dependencies from mirror-mypy with project dependencies.
Profile functions (compute_density_profile, compute_speed_profile, compute_profiles) no longer require pre-computed grid intersections. When grid_intersections_area is not provided, intersections are computed per frame internally, avoiding the allocation of the full (grid_cells × pedestrians) matrix. - Add _compute_frame_grid_intersection helper for single-frame intersection - Change _compute_grid_polygon_intersection to process frame-by-frame using NumPy instead of a single broadcast - Make grid_intersections_area optional in compute_density_profile and compute_speed_profile (on-the-fly when omitted) - Simplify compute_profiles to delegate to compute_density_profile and compute_speed_profile per frame with shared intersection - Update user guide to use the simpler API without compute_grid_cell_polygon_intersection_area
Let compute_speed_profile use its own default (0.5) when gaussian_width is not provided to compute_profiles, instead of hardcoding the value at the call site.
… work compute_profiles was calling compute_density_profile and compute_speed_profile per frame, which each redundantly called get_grid_cells() and data.groupby(FRAME_COL) again. Call the internal _compute_*_profile helpers directly instead, computing grid cells, centroids, and bounds once up front.
…section Remove duplicated intersection logic by delegating to the single-frame helper introduced in this PR.
Return an empty (num_grid_cells, 0) array instead of crashing with ValueError from np.concatenate on an empty list.
Only compute grid cell centroids when a Gaussian method is selected. Use shapely.centroid(grid_cells) instead of np.vectorize(shapely.centroid) for better performance.
The parameter is used for both DensityMethod.GAUSSIAN and SpeedMethod.GAUSSIAN in compute_profiles.
Verify that compute_density_profile and compute_speed_profile produce identical results whether grid_intersections_area is precomputed or computed on-the-fly per frame.
Measures wall time and peak memory for compute_profiles, compute_density_profile, and compute_speed_profile with both precomputed and on-the-fly intersection paths.
chraibi
force-pushed
the
improve-large-traj-profiles
branch
from
ecdceba to
a2ca5be
Compare
Variables like grid_intersections_area_frame and center_x/center_y are guaranteed non-None by the needs_intersection/needs_grid_centers guards, but mypy cannot infer this across branches.
Codecov Report❌ Patch coverage is
☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Profile functions (
compute_density_profile,compute_speed_profile,compute_profiles) no longer require pre-computed grid intersections. Whengrid_intersections_areais not provided, intersections are computed per frame internally, avoiding the allocation of the full (grid_cells × pedestrians) matrix._compute_frame_grid_intersectionhelper for single-frame intersection_compute_grid_polygon_intersectionto process frame-by-frame using NumPy instead of a single broadcastgrid_intersections_areaoptional incompute_density_profileandcompute_speed_profile(on-the-fly when omitted)compute_profilesto delegate tocompute_density_profileandcompute_speed_profileper frame with shared intersectioncompute_grid_cell_polygon_intersection_area