Skip to content

fix: import torch before spmd_types to avoid circular import - #16

Open
bananighosh wants to merge 1 commit into
aws-neuron:release_2.33.0from
bananighosh:fix/spmd-types-circular-import
Open

bananighosh wants to merge 1 commit into
aws-neuron:release_2.33.0from
bananighosh:fix/spmd-types-circular-import

Conversation

@bananighosh

Copy link
Copy Markdown

Description of changes:
On the Neuron SDK 2.33 DLAMI, importing spmd_types before torch triggers the torch_neuronx backend autoload, which loads torch.distributed.fsdp; _fsdp_param_group.py then re-imports the partially initialized spmd_types and fails with "has no attribute 'register_local_autograd_function'".

The Qwen3, DeepSeek-V3 and GPT-OSS recipes import spmd_types at module top, so their config_registry fails to import and ConfigManager reports "Config function ... not found" (e.g. the Qwen3-32B command in docs/dense-tutorials.md).

Import torch in torchtitan_neuron/init.py so it is loaded before any recipe module.

Tested on trn3-dev1.48xlarge, DLAMI ami-0dea3e9e3e3a16119 (torch 2.14.0, torch-neuronx 2.14.3.0.1, spmd_types 0.2.1): the four model config_registry modules import, and the dense-tutorials Qwen3-32B eager command runs 10/10 steps (rc=0).

error:

W1107 21:06:24.372000 2218974 torch/distributed/run.py:982] *****************************************
W1107 21:06:24.372000 2218974 torch/distributed/run.py:982] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W1107 21:06:24.372000 2218974 torch/distributed/run.py:982] *****************************************
[titan] 2026-11-07 21:06:28,423 - root - INFO - torchtitan version: 0.2.2+neuron.1 (0.0.0 means __version__ is not defined correctly).
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/nl/exp/torchtitan-neuron/torchtitan/train.py", line 79, in <module>
    main()
  File "/home/ubuntu/nl/exp/torchtitan-neuron/torchtitan/train.py", line 29, in main
    config = config_manager.parse_args()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/nl/exp/torchtitan-neuron/torchtitan/config/manager.py", line 37, in parse_args
    loaded_config, args = self._load_config(args)
                          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/nl/exp/torchtitan-neuron/torchtitan/config/manager.py", line 153, in _load_config
    raise ValueError(
ValueError: Config function 'qwen3_32b_tp4_fsdp16_fullac_eager' not found in torchtitan_neuron.models.qwen3. Available config functions: ['core_model_registry', 'model_registry', 'replace', 'validate_converter_order']
W1107 21:06:29.340000 2218974 torch/distributed/elastic/multiprocessing/api.py:1028] Sending process 2219042 closing signal SIGTERM
W1107 21:06:29.341000 2218974 torch/distributed/elastic/multiprocessing/api.py:1028] Sending process 2219043 closing signal SIGTERM

Issue #, if available: - N/A

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

On the Neuron SDK 2.33 DLAMI, importing spmd_types before torch triggers
the torch_neuronx backend autoload, which loads torch.distributed.fsdp;
_fsdp_param_group.py then re-imports the partially initialized spmd_types
and fails with "has no attribute 'register_local_autograd_function'".

The Qwen3, DeepSeek-V3 and GPT-OSS recipes import spmd_types at module
top, so their config_registry fails to import and ConfigManager reports
"Config function ... not found" (e.g. the Qwen3-32B command in
docs/dense-tutorials.md).

Import torch in torchtitan_neuron/__init__.py so it is loaded before any
recipe module.

Tested on trn3-dev1.48xlarge, DLAMI ami-0dea3e9e3e3a16119 (torch 2.14.0,
torch-neuronx 2.14.3.0.1, spmd_types 0.2.1): the four model
config_registry modules import, and the dense-tutorials Qwen3-32B eager
command runs 10/10 steps (rc=0).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant