Skip to content

test(conformance): tag unversioned v0.9 cases in functions.yaml and actions.yaml and add v1.0 copies - #4242

Open
vojtech wants to merge 3 commits into
a2ui-project:mainfrom
vojtech:conformance/version-unversioned-cases
Open

vojtech wants to merge 3 commits into
a2ui-project:mainfrom
vojtech:conformance/version-unversioned-cases

Conversation

@vojtech

@vojtech vojtech commented Nov 15, 2026 •

Copy link
Copy Markdown

23 cases in conformance/core/ have no protocolVersion and expect v0.9 behaviour that the v1.0 spec changed. The Python, Dart and Swift core runners default an unversioned case to v0.9, so they pass today. That default is only in the runner code: conformance_schema.json and conformance/README.md do not state it, and conformance_harness_implementation_guide.md gives the opposite default ("1.0") for agent fixtures. A harness for a 1.0-only renderer has no documented way to know these cases are v0.9, and has to fail them.

functions.yaml (21 cases): bare boolean instead of ValidationResult. The cases expect true/false (for example test_evaluate_function_required_string_empty: expect: false). The v1.0 spec says these functions return a ValidationResult:

validation functions (required, email, regex, length, numeric, which return ValidationResult)
— specification/v1_0/docs/a2ui_protocol.md, "DynamicBoolean and CheckRule Coercion"

result.valid is extracted only where the value is used as a DynamicBoolean, not for a direct call. The regex_v10_* cases in the same file already set protocolVersion: "1.0" and expect {valid: …}.

Affected: required string_valid, string_empty, null, array_empty, array_valid, object_empty; regex match, mismatch; email valid, invalid_no_domain, invalid_single_char_tld; length string_valid, string_too_short, string_too_long, array_count_valid, array_count_too_short; numeric valid_float, valid_string, too_small, too_large, unparseable_string.

actions.yaml (2 cases): bare path instead of @path. test_action_event_with_data_binding and test_action_event_with_relative_scope bind event context with {path: …}. In v1.0 a DataBinding is {"@path": …} (common_types.json), and reserved_keys.yaml keeps a bare path key as literal data. The 1.0 cases in the same file already bind with "@path".

Change (against main 3ed9601862):

  • Tag each of the 23 cases protocolVersion: "v0.9". This matches what the runners already do, so v0.9 coverage is unchanged. functions.yaml already tags test_evaluate_function_length_null_value and …_too_short this way.
  • Add 21 _v10 copies with protocolVersion: "1.0" that expect {valid: <bool>} (19 in functions.yaml) or bind with "@path" (2 in actions.yaml). Before this, functions.yaml had no direct v1.0 evaluate_function case for required, email, length or numeric.
  • The two failing email cases get no _v10 copy. In v1.0 a failed email call also returns code, message and severity, and validation_result.yaml already covers that result.

Testing

  • Python: pytest tests/test_conformance.py in python/a2ui_core (853 passed).
  • Swift: swift test --filter "BasicCatalogConformanceTests|ActionsConformanceTests" (both suites pass).
  • pytest conformance/tests/test_conformance_yaml.py test_feature_registry.py test_check_executed_cases.py (172 passed), and Prettier reports both files formatted.
  • Dart: not run locally.
  • A 1.0-only renderer (AnySDUI) skips the v0.9 cases and passes the _v10 copies. The two regex copies were not run against it.

🤖 Generated with Claude Code

@google-cla

google-cla Bot commented Nov 15, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@github-actions github-actions Bot added the status: needs-triage auto-managed: https://github.com/a2ui-project/a2ui/blob/main/scripts/triage.mjs label Nov 15, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the conformance tests in actions.yaml and functions.yaml to support protocol version 1.0 alongside v0.9 by explicitly marking existing tests with protocolVersion: "v0.9" and adding corresponding v10 test cases with updated expectation formats. The review feedback highlights three instances in functions.yaml where new _v10 test cases were placed after section header comments, resulting in incorrect grouping and duplicated header comments. Moving these test cases before the section headers will keep the test suite properly organized.

Comment thread conformance/core/functions.yaml Outdated
Comment on lines 299 to 311
# --- RegexFunction (D21) ---

- name: test_evaluate_function_required_object_empty_v10
description: Verifies required returns true for empty object (D20).
protocolVersion: "1.0"
action: evaluate_function
function: required
args:
value: {}
expect:
valid: true

# --- RegexFunction (D21) ---

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The new test_evaluate_function_required_object_empty_v10 test case is placed after the # --- RegexFunction (D21) --- section header comment, which incorrectly groups it under the Regex section instead of the Required section. Additionally, this results in a duplicated # --- RegexFunction (D21) --- comment. Let's move the test case before the section header to keep the tests correctly grouped.

- name: test_evaluate_function_required_object_empty_v10
  description: Verifies required returns true for empty object (D20).
  protocolVersion: "1.0"
  action: evaluate_function
  function: required
  args:
    value: {}
  expect:
    valid: true

# --- RegexFunction (D21) ---

Comment thread conformance/core/functions.yaml Outdated
Comment on lines 747 to 759
# --- LengthFunction (D24) ---

- name: test_evaluate_function_email_invalid_single_char_tld_v10
description: Verifies email returns false on single-character TLD (D22).
protocolVersion: "1.0"
action: evaluate_function
function: email
args:
value: "test@test.c"
expect:
valid: false

# --- LengthFunction (D24) ---

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The new test_evaluate_function_email_invalid_single_char_tld_v10 test case is placed after the # --- LengthFunction (D24) --- section header comment, which incorrectly groups it under the Length section instead of the Email section. Additionally, this results in a duplicated # --- LengthFunction (D24) --- comment. Let's move the test case before the section header to keep the tests correctly grouped.

- name: test_evaluate_function_email_invalid_single_char_tld_v10
  description: Verifies email returns false on single-character TLD (D22).
  protocolVersion: "1.0"
  action: evaluate_function
  function: email
  args:
    value: "test@test.c"
  expect:
    valid: false

# --- LengthFunction (D24) ---

Comment thread conformance/core/functions.yaml Outdated
Comment on lines 991 to 1003
# --- AndFunction (D25, D26) ---

- name: test_evaluate_function_numeric_unparseable_string_v10
description: Verifies numeric returns false for unparseable string without raising (D23).
protocolVersion: "1.0"
action: evaluate_function
function: numeric
args:
value: "not_a_number"
expect:
valid: false

# --- AndFunction (D25, D26) ---

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The new test_evaluate_function_numeric_unparseable_string_v10 test case is placed after the # --- AndFunction (D25, D26) --- section header comment, which incorrectly groups it under the And section instead of the Numeric section. Additionally, this results in a duplicated # --- AndFunction (D25, D26) --- comment. Let's move the test case before the section header to keep the tests correctly grouped.

- name: test_evaluate_function_numeric_unparseable_string_v10
  description: Verifies numeric returns false for unparseable string without raising (D23).
  protocolVersion: "1.0"
  action: evaluate_function
  function: numeric
  args:
    value: "not_a_number"
  expect:
    valid: false

# --- AndFunction (D25, D26) ---

@vojtech vojtech changed the title conformance: 21 unversioned cases in functions.yaml and actions.yaml expect v0.9 behaviour test(conformance): tag unversioned v0.9 cases in functions.yaml and actions.yaml and add v1.0 copies Nov 17, 2026
@vojtech
vojtech force-pushed the conformance/version-unversioned-cases branch from 5d3b171 to f5f4932 Compare November 17, 2026 17:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status: needs-triage auto-managed: https://github.com/a2ui-project/a2ui/blob/main/scripts/triage.mjs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant