diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d1d834..5400046 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,77 +4,47 @@ All notable changes to this project are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +## Where to find the mapping files + +- **Working version (use for ETL):** `redCapDataDictionaryWithExtension/value-set-mapping-common/_ai_readi_redcap_value_set_mapping_common.csv`. This file is actively maintained and may change after each release. +- **Frozen release version:** `manuscript-tvst2026/ai_readi_redcap_value_set_mapping_common.csv` (with frozen SSSOM and LinkML copies), matching release `v1.0-tvst` and the TVST 2026 manuscript. + ## [Unreleased] -- prior versions are used for development and testing ## [v1.0-tvst] - 2026-10-14 -First archived release, accompanying the manuscript *"A Generalizable Framework for Mapping Survey, Vision Care, and Cognitive Assessment Data to the OMOP Common Data Model in AI-READI"* (Translational Vision Science & Technology, 2026). +First archived release, accompanying the manuscript *"A Generalizable Framework for Mapping Survey, Vision Care, and Cognitive Assessment Data to the OMOP Common Data Model in AI-READI"* (Translational Vision Science & Technology, 2026). Archived on Zenodo: https://doi.org/10.5281/zenodo.23375723 ### Added -- `manuscript-tvst2026/`: frozen mapping file, DuckDB and Databricks count scripts, OMOP vocabulary validation script, Figures 1–3 and Supplementary Tables S1–S2. -- AI-READI custom concepts 2005200630 (PhenX DMS total score) and 2005200631 (dvenvlocn). +- `manuscript-tvst2026/`: frozen mapping file, frozen SSSOM and LinkML copies, DuckDB and Databricks count scripts (`count_concept_map*.sql`, `run_counts.py`), OMOP vocabulary validation script, Figures 1–3 and Supplementary Tables S1–S2. +- AI-READI custom concepts: + - 2005200631 "Location of environmental sensor within home" for `dvenvlocn`, with qualifier 45762337 (Environmental sensor). It was previously mapped to SNOMED 284349006 (4109238), which does not describe sensor location. + - 2005200630 "Discrimination in Medical Settings (DMS) total score [PhenX]" for `pxdhc_sum`, with qualifier 2005200575 (PhenX SDOH Survey). It was previously unmapped. - SSSOM mapping set and LinkML schema in `redCapDataDictionaryWithExtension/`. - `deprecated/` folder with a README describing superseded files. ### Changed -- Standard concepts replaced after validation against the OMOP Standardized Vocabularies from non-standard (e.g., heart rate 3027018, LOINC 63954-2, LOINC 52810-9, SNOMED 1148423001). line 864,867 - non-standard vs. standard - use standard concept -Heart rate (pulse_vsorres) -4239408 (SNOMED 364075005) -> 3027018 (LOINC 8867-4, Heart rate) Standard Measurement-domain concept for a recorded vital sign -- Insulin mapped to LOINC 3695-4 (Mass/volume) with ng/mL units; 24 missing lab unit concepts added. +- Non-standard targets replaced with standard concepts after validation against the OMOP Standardized Vocabularies (v5.0, 27-Aug-2025): + - Heart rate (`pulse_vsorres`): 4239408 (SNOMED 364075005) → 3027018 (LOINC 8867-4, Heart rate). + - MoCA total score: 37174522 (SNOMED 1255891005) → 606671 (SNOMED 1148423001, Montreal Cognitive Assessment version 8.1). +- Closer standard equivalents: + - Substance use (`subrdosfr`): 1010004 (LOINC 68518-0) → 40766674 (LOINC 63954-2), closeMatch, confidence 0.75. + - Current medications, frequency: 4126015 (SNOMED 260864003) → 3042548 (LOINC 52810-9, Current Medication, Frequency). + - Disposition reason: 4260904 (SNOMED 410666004) → 44810920 (SNOMED 871401000000109), relatedMatch, confidence 0.5; no exact standard concept, flagged for review. +- Insulin mapped to LOINC 3695-4 (Mass/volume) with ng/mL units. +- Count scripts renamed `count_mapping_concepts*.sql` → `count_concept_map*.sql`; `run_counts.py` now prints UTF-8 output on Windows. +- Supplementary Table S1 now links the MoCA to https://mocacognition.com/paper; the MoCA 8.1 test form PDF was removed (copyright). +- `_v4` and `_v5` copies of the common mapping file moved to `deprecated/`; map2omop and open-source public versions regenerated from the common file. - Top-level README, `CITATION.cff` and `codemeta.json` rewritten for this repository (previously template content). - Superseded mapping files and folders moved to `deprecated/`. -- New custom concept created to support the following concept -Line 436, dvenvlocn (“Location of the environmental sensor in the home”): **dvenvlocn** → 2005200631 (436) and PhenX DMS → 2005200630 (1408) -It was mapped to SNOMED 4109238 (284349006), which doesn’t describe sensor location. -No suitable standard concept exists, so it now uses the new custom concept 2005200631, “Location of environmental sensor within home”, with qualifier 45762337 (Environmental sensor). -- Line 1408, pxdhc_sum (PhenX Discrimination in Medical Settings summary score): -It was previously unmapped. -It now uses the new custom concept 2005200630, “Discrimination in Medical Settings (DMS) total score [PhenX]”, with qualifier 2005200575 (PhenX SDOH Survey). -- concept map changed for MoCA total score/ 37174522 (SNOMED 1255891005) -> 606671 (SNOMED 1148423001), class Staging / Scale - ### Fixed -- Swapped REDCap survey start and completion timestamp concepts. -- Excel-corrupted values: concept codes converted to scientific notation and answer ranges converted to dates. -- the swapped timestamp concepts (1098/1099) Survey timestamp concepts swapped back (lines 1098–1099) -In the PhenX racial and ethnic discrimination survey, the start and completion timestamps were mapped the wrong way round: -pxrdcmpts (completion) pointed to 2005200558 Survey Started Timestamp; -pxrdstartts (start) pointed to 2005200559 Survey Completed Timestamp. -- heart rate (864/867), MoCA total (647), line 1893, line 334, line 246 -- 25 lab unit rows that had no UCUM concept -- the Excel-corrupted codes found in the mapping csv, fixed -- line 647 - failed standard validation test (non-standard vs standard) - use the standard concept -- Line 1893 - found a better term -Substance use item (subrdosfr) -1010004 (LOINC 68518-0) -> 40766674 (LOINC 63954-2)closeMatch, confidence 0.75 Closer standard equivalent -- Line 334 - found a better term -Current medications, frequency/ 4126015 (SNOMED 260864003)-> 3042548 (LOINC 52810-9, “Current Medication, Frequency”) -Question-level LOINC concept matches the item -- Line 246 - found a better term -Disposition reason -4260904 (SNOMED 410666004) -> -44810920 (SNOMED 871401000000109), relatedMatch, confidence 0.5 -No exact standard concept; nearest related concept kept and was flagged for review - -- We used to have the lab results in the separate mapping file, but they are all merged. In the merged list the UCUM unit concept id was blank in the target_concept_id column. They are now all filled in. 25 rows of imported lab result unit (line range: 506-613) -- Values corrupted by excel- found few values that were turned into a scientific notation when the code value was long ( i.e. 434941000124101, 859981000000107, 449921000124105, 873751000000103, 368441000000106) -For example, case of 4-Mar when it should be 3-4 and case of 2-Jan when it should be 1-2 - -- line 862,865,863,866 - concept id was entered into the code field - corrected. - -- Concept names did not match the concept name in the OMOP vocabulary table - corrected - -- Laterality qualifiers added for eye and foot measurements - missing qualifier values added -Snellen visual acuity rows were missing the qualifier concept id for right eye or left eye -Monofilament foot sensation rows were missing the laterality qualifier - right or left - - -## Latest Release version tag v1.0-tvst2026 -- the latest release version of the concept map csv file is ai_readi_redcap_value_set_mapping_common.csv - tag v1.0-tvst-2026 -- fronzen release copy of the concept map csv file is located in the manuscript-tvst2026/ -- working version is located in the redCapDataDictionaryWithExtension/value-set-mapping-common/ -Please note, the working directory contains the ai_readi_redcap_value_set_mapping_common.csv, which should be used for ETL. This file is working file so it can change after each release. The fronzen release version is in the manuscript-tvst2026 folder. \ No newline at end of file +- REDCap survey start/completion timestamp concepts swapped back for the PhenX racial and ethnic discrimination survey (`pxrdstartts` → 2005200558 Survey Started Timestamp; `pxrdcmpts` → 2005200559 Survey Completed Timestamp). +- Lab result unit rows (25) that had a blank UCUM `TARGET_CONCEPT_ID` after the lab mappings were merged into the common file are now filled in. +- Excel-corrupted values restored: long concept codes converted to scientific notation (e.g., 434941000124101, 859981000000107, 449921000124105, 873751000000103, 368441000000106) and answer ranges converted to dates (e.g., "4-Mar" → "3-4", "2-Jan" → "1-2"). +- Concept IDs that had been entered in the concept code field corrected. +- Concept names that did not match the OMOP vocabulary concept name corrected. +- Missing laterality qualifiers added: Snellen visual acuity rows (Right eye / Left eye) and monofilament foot sensation rows (Structure of right foot / Structure of left foot). diff --git a/release_notes_v1.0-tvst.md b/release_notes_v1.0-tvst.md new file mode 100644 index 0000000..9216428 --- /dev/null +++ b/release_notes_v1.0-tvst.md @@ -0,0 +1,15 @@ +Archived on Zenodo: https://doi.org/10.5281/zenodo.23375723 + +First archived release, accompanying the manuscript *"A Generalizable Framework for Mapping Survey, Vision Care, and Cognitive Assessment Data to the OMOP Common Data Model in AI-READI"* (Translational Vision Science & Technology, 2026). + +**What's in this release** +- `manuscript-tvst2026/`: the frozen REDCap-to-OMOP mapping file (2,004 rows) used in the manuscript, frozen SSSOM and LinkML copies, count scripts that reproduce every number in the paper (`count_concept_map.sql`, `run_counts.py`), the OMOP vocabulary validation script, Figures 1–3 and Supplementary Tables S1–S2. +- AI-READI custom vocabularies (AIREADI, AIREADI-Vision, AIREADI-MOCA) submitted to the OHDSI Vocabulary Working Group. + +**Key numbers (Table 2):** 2,906 curated elements; 1,900 mapped (65.4%): 1,412 to 380 standard concepts and 488 to 343 AI-READI custom concepts. + +**Reproduce:** `cd manuscript-tvst2026 && pip install duckdb && python run_counts.py` + +The working mapping in `redCapDataDictionaryWithExtension/value-set-mapping-common/` may change after this release. Use this tag for the version cited in the paper. + +See `CHANGELOG.md` for the full list of changes.