Skip to content

Normalize localized sex labels across managed person datasets #1138

Open

Description

@andreatnvidia

The managed person sampler accepts sex="Male" or sex="Female", but several prepared NGC datasets store localized labels. Both accepted filters match zero rows and generation raises PersonSamplerConstraintsError, even though the requested population exists.

This already affects existing locales and should be handled separately from the locale registration in #1136.

Locale Labels in the prepared data
ja_JP 男, 女
pt_BR Masculino, Feminino
es_SV (added in #1136) Masculino, Femenino
vi_VN (added in #1136) Nam, Nữ

Japan and Brazil files match the published NGC 0.0.1 checksums. El Salvador and Vietnam files were downloaded from their 0.0.1 resources. Full-file sex distributions contain only the labels above. On 512-row excerpts, either accepted sex value reproduces the error through PersonSampler.sample(1). English-language India already stores Male/Female and both filters work.

To reproduce, download an affected managed dataset, configure a person sampler column with PersonSamplerParams(locale="ja_JP", sex="Female"), and generate a record. Repeat with sex="Male" or locale="pt_BR".

Expected: filtering selects records of the requested sex. Actual: zero matching records, followed by a constraints error.

ManagedDatasetGenerator.generate_samples currently translates German Weiblich/Männlich labels before filtering, but passes other locales through unchanged. The German regression test provides a pattern for a shared fix.

Suggested scope:

  • Normalize localized labels before filtering, either in prepared assets or the loader.
  • Keep unfiltered output consistent with the same Male/Female vocabulary.
  • Add regression coverage using actual labels for affected locales, retaining German and already-normalized cases.

Source inspected: PR #1136 head 8bd840b82a86c90eace87dee2b621ad65dc4a6c0; the loader behavior also exists in its base 453283a5b577ebab1f4a687b1c0dd0af5b4febcd.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions