[williamcaban]

📋 Summary

Fixes three cases where profiling failed or misreported because a statistic is undefined or a missing value is pd.NA: the output-token stddev of a single-record LLM column on pandas 3.1+, the distribution of a single-record numerical column, and the null and unique counts of nullable columns. Done as one profiling PR as discussed on #1144 and #1147.

🔗 Related Issue

Fixes #1144
Fixes #1147

🔄 Changes

🐛 Fixed

• calculate_output_token_stats computes on a float64 series. On pandas 3.1+ Series.apply keeps the Arrow backing of a structured LLM column and the sample stddev of one value is pd.NA, which llm_column_ensure_python_floats cannot convert (DataDesignerProfilingError on pandas 3.1.x when previewing a single record with an LLM column #1144).
• NumericalDistribution.from_series reports an undefined stddev as NaN instead of failing, so a single-record numerical column keeps its min, max, mean and median. This already failed on pandas 2.3.3 and 3.0.x.
• calculate_general_column_info counts nulls on the original column and drops them before the unique count. ensure_hashable turned pd.NA into the string "<NA>", so num_null was 0 and num_unique one too high for nullable numeric, string and boolean columns (Profiling miscounts nulls and unique values in nullable columns on pandas 3.1 #1147).

🧪 Testing

• make test passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)

make needs uv 0.13.4 or newer, so I ran the same suites with pytest directly (config 834, engine 2981, interface 1337 passed). They pass on the locked pandas 3.0.5 and on 3.1.1, and ruff format and ruff check are clean on the changed files.

New tests:

• Engine: single-record and multi-record Arrow-backed output stats, nine missing-value count cases (nullable int, float, string, boolean, Arrow string, all-null, object), and nested values that still need hashing for uniqueness.
• Config: NumericalDistribution.from_series for one value on NumPy, nullable and Arrow float series, and a multi-value Arrow series.
• Interface: preview and create for nullable seeds and for a single-record sampler.

On pandas 3.1.1 without the fix 13 tests fail, 11 of them new. The other two are existing tests (test_calculate_general_column_info and the profiler Jinja-engine test) that main also fails on 3.1.1.

I did not add a plan document because the change is small and limited to the statistics code. Happy to add one if you prefer.

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• If I am an external contributor, the linked issue is open and labeled triaged
• Architecture docs updated (if applicable)

🤖 Generated with Claude Code

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[williamcaban]

I have read the DCO document and I hereby sign the DCO.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable bugs or repository-rule violations were found.

Summary

Fixes profiling for nullable columns and single-record datasets.

• Single-record numerical profiles keep their defined statistics.
• Nullable columns report missing values separately from unique values.
• LLM output profiles keep token statistics for one-row columns.

_{Reviews (3) · Last reviewed commit: "Merge branch 'main' into williamcaban/fi..." · Reviewed by Greptile}

[williamcaban]

recheck

[andreatnvidia]

Thanks for tackling both issues together! This addresses #1144 and #1147, including preserving the defined statistics for single-record numerical samplers.

Looks good to merge.