Structured-output prompt schemas escape non-ASCII text as \uXXXX #1130

Priority Level

Medium (Annoying but has workaround)

Describe the bug

StructuredResponseRecipe.schema and PydanticResponseRecipe.schema serialize the JSON schema with json.dumps(...) and its default ensure_ascii=True:

• packages/data-designer-engine/src/data_designer/engine/models/recipes/response_recipes.py L113: return json.dumps(self.json_schema)
• same file L185: return json.dumps(self.data_type.model_json_schema())

That string is placed inside <response_schema> in the user prompt (L101 and L193). Any non-ASCII text in the schema (field descriptions, enum values, LLM-judge score descriptions and option labels) therefore reaches the model as \uXXXX escapes rather than as readable text. This affects LLMJudgeColumnConfig (score rubrics are turned into a pydantic model) and LLMStructuredColumnConfig (output_format as a dict or a pydantic model) whenever the schema text is not ASCII, for example Thai, CJK, Cyrillic or accented Latin.

The same file already uses ensure_ascii=False in StructuredResponseRecipe.serialize_output (L116). Parsing a Thai response works today; only the prompt side is affected.

Steps/Code to reproduce bug

Against main at ba00b05, CPU only, no model needed:

import re

from data_designer.config.column_configs import LLMJudgeColumnConfig, LLMStructuredColumnConfig, Score
from data_designer.engine.column_generators.utils.prompt_renderer import create_response_recipe

judge = LLMJudgeColumnConfig(
    name="j",
    model_alias="m",
    prompt="ให้คะแนน {{ x }}",
    scores=[Score(name="accuracy", description="ประเมินความถูกต้อง", options={1: "ผิดทั้งหมด", 3: "ถูกต้องครบถ้วน"})],
)
p = create_response_recipe(judge).apply_recipe_to_user_prompt("ให้คะแนนคำตอบนี้")
print(p.rsplit("<response_schema>", 1)[1][:200])
print("escaped:", len(re.findall(r"\\u0e[0-9a-f]{2}", p)), "literal:", len(re.findall(r"[฀-๿]", p)))

st = LLMStructuredColumnConfig(
    name="s",
    model_alias="m",
    prompt="x",
    output_format={"type": "object", "properties": {"region": {"type": "string", "description": "ภาค", "enum": ["ภาคเหนือ", "ภาคใต้"]}}},
)
print(create_response_recipe(st).schema)

Output on main:

{"$defs": {"accuracy": {"description": "\u0e1b\u0e23\u0e30\u0e40\u0e21\u0e34\u0e19\u0e04\u0e27\u0e32\u0e21\u0e16\u0e39\u0e01\u0e15\u0e49\u0e2d\u0e07", ...
escaped: 42 literal: 16
{"type": "object", "properties": {"region": {"type": "string", "description": "\u0e20\u0e32\u0e04", "enum": ["\u0e20\u0e32\u0e04\u0e40\u0e2b\u0e19\u0e37\u0e2d", "\u0e20\u0e32\u0e04\u0e43\u0e15\u0e49"]}}}

The 16 literal Thai characters are the user's own instruction. Every Thai character from the rubric and the schema is escaped.

Expected behavior

The schema in <response_schema> keeps non-ASCII text as written, the way serialize_output already does:

{"$defs": {"accuracy": {"description": "ประเมินความถูกต้อง", ... "* 1: ผิดทั้งหมด\n* 3: ถูกต้องครบถ้วน"}}, ...
{"type": "object", "properties": {"region": {"type": "string", "description": "ภาค", "enum": ["ภาคเหนือ", "ภาคใต้"]}}}

ASCII-only schemas should stay byte-identical.

Agent Diagnostic / Prior Investigation

I investigated this with Claude Code (Anthropic) on a clone of main at ba00b05. I reviewed the findings and reran them myself.

• Root cause: json.dumps defaults to ensure_ascii=True in StructuredResponseRecipe.schema (L113) and PydanticResponseRecipe.schema (L185). StructuredResponseRecipe.generate_response_example (L109) has the same default. It has no production callers, but it is the documented example format for that recipe.
• The schema property is used in only one place, the <response_schema> block of task_instructions. Nothing hashes or caches it, so changing its serialization changes only the prompt text.
• Size: for a three-option Thai judge rubric, the full user prompt is 623 prompt tokens on main and 315 with ensure_ascii=False, as reported by a local vLLM server (Qwen3.6-35B-A3B tokenizer). The difference comes entirely from the escapes.
• Live check: one request each way (temperature 0, thinking disabled). Both replies were valid fenced JSON and parsed with the recipe. One sample does not show a difference in judge quality, and I am not claiming one. What changes is what the model reads: \u0e1c\u0e34\u0e14... on main, ผิดทั้งหมด with the fix.
• Duplicate search (issues and PRs, open and closed: "ensure_ascii", "non-ASCII", "unicode", "Thai", "escape", "multilingual", "response_schema"): no match. feat: persist scheduler events to JSONL #800 is the only other ensure_ascii hit, and it is about JSONL scheduler events. Accept valid bare JSON in structured response recipes #971/fix: accept bare JSON in structured responses #972 changed parsing in the same recipes, not schema serialization. Add an opt-in native schema-enforcement mode to LLMStructuredColumnConfig #1125 (open feature request: opt-in native schema-enforcement mode for LLMStructuredColumnConfig) is a different feature. No open PR touches response_recipes.py.
• Proposed fix: pass ensure_ascii=False in the two schema properties and in generate_response_example, and add parametrized tests (Thai pydantic model and Thai dict schema) that fail on main and pass with the change. That is +30/-3 lines. I have it ready and can open a PR once this issue is triaged.

Additional context

• Workaround: write rubrics, descriptions and enum values in English. That does not help when the enum values themselves have to be in the target language.
• Possible follow-up, not part of this fix: engine/mcp/io.py L502 and L531 also json.dumps MCP tool results into text blocks with the default ensure_ascii=True. I have not checked how those blocks reach the model, so I am leaving them out of this issue's scope.
• Environment: Linux aarch64 (DGX Spark), Python 3.14.6 (uv sync --all-packages --group dev), CPU-only reproduction.

Checklist

• I reproduced this issue or provided a minimal example
• I searched the docs/issues myself, or had my agent do so
• If I used an agent, I included its diagnostics above