Skip to content

test(distributed): xfail TestTraceableCollectives Neuron tests that copy across devices - #1030

Closed
tdalmia-aws wants to merge 1 commit into
aws-neuron:mainfrom
tdalmia-aws:fix/cross-device-copy-1021
Closed

tdalmia-aws wants to merge 1 commit into
aws-neuron:mainfrom
tdalmia-aws:fix/cross-device-copy-1021

Conversation

@tdalmia-aws

@tdalmia-aws tdalmia-aws commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Description

Marks the 7 TestTraceableCollectives Neuron tests that fail in the nightly distributed job as xfail.

Related Issue

Related to #1021

Motivation and Context

The threaded process group in these tests copies tensors between Neuron devices, and a cross-device copy currently keeps the result on the source device (#1021). The tests fail with Inconsistent device IDs found in kernel outputs: expected device 0 but found device 1, which keeps the nightly red: https://github.com/aws-neuron/torch-neuronx/actions/runs/41279502191

The copy fix will come with the next release. Until then these are xfail (not strict), so they report as xpass once the fix lands.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)

Test-only change: skip registry entries.

Testing

Ran locally (no Neuron hardware needed):

pre-commit run --files tests/pytorch_tests_native/skip_registry/distributed_xfails_registry.py

All hooks pass.

Ran on Neuron hardware:

python -m pytest tests/pytorch_tests_native/test/distributed/test_functional_api.py -k TestTraceableCollectives

On main these 7 fail with the error above, the same as the nightly. The xfail entries cover exactly those 7 tests.

Checklist

  • Added or updated tests for this change
  • I've performed a self-review for my code
  • Ran the tests, or explained under Testing why I could not
  • Updated the corresponding documentation
  • My changes generate no new warnings
  • Said under Description if an AI assistant helped with this change (see AI_POLICY.md)

@tdalmia-aws
tdalmia-aws marked this pull request as ready for review November 2, 2026 21:18
@tdalmia-aws
tdalmia-aws requested review from a team as code owners November 2, 2026 21:18
@github-actions github-actions Bot added module:runtime-core Touches core runtime — full suite module:upstream-test-infra Touches upstream test runner/dist config module:ci Touches GitHub Actions config labels Nov 2, 2026
…opy across devices

The threaded process group copies tensors between Neuron devices. A
cross-device copy keeps the result on the source device (aws-neuron#1021), so 7
TestTraceableCollectives tests fail in the nightly distributed job.
Mark them as xfail until the copy fix ships.
@tdalmia-aws tdalmia-aws changed the title fix(dispatch): move cross-device copy results to the target Neuron device test(distributed): xfail TestTraceableCollectives Neuron tests that copy across devices Nov 2, 2026
@tdalmia-aws
tdalmia-aws force-pushed the fix/cross-device-copy-1021 branch from 9fc4dd7 to 4859dd7 Compare November 2, 2026 21:55
@github-actions github-actions Bot added ciflow:skip-upstream Docs/license only — skip upstream tests and removed module:runtime-core Touches core runtime — full suite module:upstream-test-infra Touches upstream test runner/dist config labels Nov 2, 2026
@tdalmia-aws tdalmia-aws closed this Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow:skip-upstream Docs/license only — skip upstream tests module:ci Touches GitHub Actions config

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant