Repository navigation
Conversation
…sion CDMCacheService warms the record count cache by splitting achilles_result_concept_count into ranges of 100,000 concepts with getConceptsForBatch.sql, which computes the range id as "ordinal / @batch_size" and groups by it. That only yields whole ranges where "/" on two integers is integer division (PostgreSQL, SQL Server, Redshift). On Snowflake the quotient is a NUMBER with six decimals and on Spark and BigQuery it is a floating point value, so every concept gets its own group id and min_concept = max_concept for each row. The warming job then runs one getBatchConceptRecordCount query per concept in the table, hundreds of thousands of round trips instead of a handful, and on those platforms the cache warming job effectively never finishes. Floor the quotient so the group id is integral on every dialect; the expression is unchanged where the division was already integral. Adds a test that runs the rendered batch query against the embedded Postgres database with 250,000 concepts and checks that three contiguous ranges come back, and that the Snowflake, Spark and BigQuery translations keep the floored group id.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2588
CDMCacheService.cacheRecordsBatchwarms the record count cache by splittingachilles_result_concept_countinto ranges of 100,000 concepts withgetConceptsForBatch.sql:and grouping by
group_id. That only produces whole ranges where/on two integers is integer division (PostgreSQL, SQL Server, Redshift). On Snowflake the quotient is aNUMBERwith six decimals, and on Spark/Databricks and BigQuery it is a floating point value, so every concept gets its owngroup_idand the query returns onemin_concept = max_conceptrow per concept. The warming job then runs onegetBatchConceptRecordCountquery per concept in the table, hundreds of thousands of round trips instead of a handful, and on those platforms the cache warming job effectively never finishes.Change
getConceptsForBatch.sql:floor(ordinal / @batch_size)so the group id is integral on every dialect. The expression is unchanged where the division was already integral, and SqlRender passesfloorthrough for Snowflake, Spark, BigQuery, SQL Server and Redshift.Tests
CDMCacheServiceBatchTest(embedded Postgres): renders the batch query the wayCDMCacheServicedoes, runs it over 250,000 concepts with gaps between ids, and checks that three contiguous ranges come back that cover every concept; a second test checks that the Snowflake, Spark and BigQuery translations keep the floored group id (this one fails onmaster).