Skip to content

Keep CDM cache warming batches whole on dialects without integer division - #2589

Open
DG47 wants to merge 1 commit into
OHDSI:masterfrom
DG47:fix/concept-count-batch-groups
Open

DG47 wants to merge 1 commit into
OHDSI:masterfrom
DG47:fix/concept-count-batch-groups

Conversation

@DG47

@DG47 DG47 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Fixes #2588

CDMCacheService.cacheRecordsBatch warms the record count cache by splitting achilles_result_concept_count into ranges of 100,000 concepts with getConceptsForBatch.sql:

select concept_id, ordinal / @batch_size as group_id

and grouping by group_id. That only produces whole ranges where / on two integers is integer division (PostgreSQL, SQL Server, Redshift). On Snowflake the quotient is a NUMBER with six decimals, and on Spark/Databricks and BigQuery it is a floating point value, so every concept gets its own group_id and the query returns one min_concept = max_concept row per concept. The warming job then runs one getBatchConceptRecordCount query per concept in the table, hundreds of thousands of round trips instead of a handful, and on those platforms the cache warming job effectively never finishes.

Change

  • getConceptsForBatch.sql: floor(ordinal / @batch_size) so the group id is integral on every dialect. The expression is unchanged where the division was already integral, and SqlRender passes floor through for Snowflake, Spark, BigQuery, SQL Server and Redshift.

Tests

  • CDMCacheServiceBatchTest (embedded Postgres): renders the batch query the way CDMCacheService does, runs it over 250,000 concepts with gaps between ids, and checks that three contiguous ranges come back that cover every concept; a second test checks that the Snowflake, Spark and BigQuery translations keep the floored group id (this one fails on master).
mvn -DskipUnitTests=false -DskipITtests=true test -Dtest=CDMCacheServiceBatchTest,CDMResultsServiceTest,CDMResultsCacheTest,CDMResultsVisitDurationTest,CDMResultsAnalysisRunnerTest,CDMResultsConditionByTypeTest,CDMResultsTreemapTest
Tests run: 22, Failures: 0, Errors: 0, Skipped: 0

…sion

CDMCacheService warms the record count cache by splitting
achilles_result_concept_count into ranges of 100,000 concepts with
getConceptsForBatch.sql, which computes the range id as
"ordinal / @batch_size" and groups by it. That only yields whole ranges
where "/" on two integers is integer division (PostgreSQL, SQL Server,
Redshift). On Snowflake the quotient is a NUMBER with six decimals and
on Spark and BigQuery it is a floating point value, so every concept
gets its own group id and min_concept = max_concept for each row. The
warming job then runs one getBatchConceptRecordCount query per concept
in the table, hundreds of thousands of round trips instead of a handful,
and on those platforms the cache warming job effectively never finishes.

Floor the quotient so the group id is integral on every dialect; the
expression is unchanged where the division was already integral.

Adds a test that runs the rendered batch query against the embedded
Postgres database with 250,000 concepts and checks that three
contiguous ranges come back, and that the Snowflake, Spark and BigQuery
translations keep the floored group id.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CDM cache warming issues one record-count query per concept on Snowflake, Spark and BigQuery (getConceptsForBatch.sql relies on integer division)

1 participant