You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Normalize localized sex labels across managed person datasets #1138
The managed person sampler accepts sex="Male" or sex="Female", but several prepared NGC datasets store localized labels. Both accepted filters match zero rows and generation raises PersonSamplerConstraintsError, even though the requested population exists.
This already affects existing locales and should be handled separately from the locale registration in #1136.
Japan and Brazil files match the published NGC 0.0.1 checksums. El Salvador and Vietnam files were downloaded from their 0.0.1 resources. Full-file sex distributions contain only the labels above. On 512-row excerpts, either accepted sex value reproduces the error through PersonSampler.sample(1). English-language India already stores Male/Female and both filters work.
To reproduce, download an affected managed dataset, configure a person sampler column with PersonSamplerParams(locale="ja_JP", sex="Female"), and generate a record. Repeat with sex="Male" or locale="pt_BR".
Expected: filtering selects records of the requested sex. Actual: zero matching records, followed by a constraints error.
ManagedDatasetGenerator.generate_samples currently translates German Weiblich/Männlich labels before filtering, but passes other locales through unchanged. The German regression test provides a pattern for a shared fix.
Suggested scope:
Normalize localized labels before filtering, either in prepared assets or the loader.
Keep unfiltered output consistent with the same Male/Female vocabulary.
Add regression coverage using actual labels for affected locales, retaining German and already-normalized cases.
Source inspected: PR #1136 head 8bd840b82a86c90eace87dee2b621ad65dc4a6c0; the loader behavior also exists in its base 453283a5b577ebab1f4a687b1c0dd0af5b4febcd.
Activity