diff --git a/blog.html b/blog.html index 15138a1..5ebb566 100644 --- a/blog.html +++ b/blog.html @@ -77,6 +77,7 @@ diff --git a/blog.js b/blog.js index 7a0b76f..3780825 100644 --- a/blog.js +++ b/blog.js @@ -6,6 +6,14 @@ desc: "发布时间：2026-2-26 | 作者：张洪滨", descEn: "Published: 2026-2-26 | Author: Hongbin Zhang", markdownUrl: "blog/triton-on-riscv.md" + }, + "pytorch-riscv-ci": { + title: "PyTorch RISC‑V CI 实践", + titleEn: "PyTorch CI on RISC-V", + desc: "", + descEn: "", + markdownUrl: "blog/pytorch-riscv-ci.md", + bodyOnly: true } }; diff --git a/blog/pytorch-riscv-ci.en.md b/blog/pytorch-riscv-ci.en.md new file mode 100644 index 0000000..900db20 --- /dev/null +++ b/blog/pytorch-riscv-ci.en.md @@ -0,0 +1,117 @@ +# PyTorch CI on RISC-V: Implementation and Practice + +Native RISC-V support is a critical foundation for bringing the PyTorch ecosystem to RISC-V platforms. The RuyiAI team at the Institute of Software, Chinese Academy of Sciences (ISCAS), together with the XuanTie team, the RISE AI/ML Working Group, and broader community partners, has been working to strengthen PyTorch support for RISC-V through native builds, testing on real RISC-V hardware, compatibility validation, continuous integration, and upstream collaboration. + +We have established a CI infrastructure for PyTorch on RISC-V and integrated it into the [RuyiAI fork of PyTorch](https://github.com/RuyiAI-Stack/pytorch). The core test suite now runs reliably on RISC-V, while test sharding, blocklist management, and continuous regression testing are progressively expanding coverage across the broader PyTorch test suite. Architecture-specific issues identified through CI are systematically tracked and reported, with corresponding fixes and improvements contributed back to the upstream PyTorch community. + +## Background: Why Does RISC-V Need Dedicated PyTorch CI? + +PyTorch's official CI infrastructure provides broad coverage across x86, Arm, and mainstream GPU platforms, but validation on physical RISC-V hardware remains limited. As a result, earlier RISC-V enablement efforts relied heavily on cross-compilation and simulator-based testing. While these approaches are valuable for initial bring-up, they cannot fully expose runtime issues that emerge only on real hardware, including problems related to floating-point behavior, atomic operations, timing, complex-number computation, and other architecture-specific execution details. + +As RISC-V processors, particularly those supporting RVV 1.0, continue to mature, simply verifying that PyTorch can be successfully compiled for RISC-V is no longer sufficient. Robust ecosystem support requires continuous validation of actual PyTorch workloads on physical RISC-V systems. To address this gap, the RuyiAI team, the XuanTie team, and community partners are building native CI pipelines on real RISC-V hardware to continuously track upstream PyTorch development, detect architecture-specific regressions, identify compatibility issues, and drive corresponding fixes and upstream contributions. + +This effort originated from the [RFC for RISC-V support](https://github.com/pytorch/pytorch/issues/171659) initiated by the XuanTie team in the PyTorch community. The RuyiAI team subsequently joined the effort and has been collaborating with the XuanTie team and broader community to advance native RISC-V support. As part of this collaboration, a dedicated [tracking issue](https://github.com/pytorch/pytorch/issues/180975) was established to coordinate ongoing work across native builds, CI infrastructure, operator optimization, compiler support, and other RISC-V-specific enablement efforts. + +## PyTorch CI Development in the RuyiAI Community + +### 2.1 Enabling Native RISC-V Builds + +The first step was to establish a stable and reproducible workflow for building PyTorch natively on riscv64. Our work has focused on three main areas: + +1. **Separating the build and test stages:** We adopt a "build once, test on multiple machines" strategy. PyTorch wheels are built on a high-performance RISC-V system and then distributed to multiple RISC-V machines for validation. This avoids redundant builds and allows the same binary artifact to be tested consistently across different hardware platforms. + +2. **Standardizing the build environment:** We use Debian Trixie as the baseline operating system, with Python 3.13, GCC 14, and OpenBLAS 0.3.33 as key components of the toolchain. The environment is kept as closely aligned with upstream PyTorch as possible, while remaining dependency differences are being addressed incrementally as RISC-V software support matures. + +3. **Improving the availability of RISC-V dependencies:** [RuyiRepo](https://github.com/RuyiRepo/ruyirepo), an open-source software repository for RISC-V, provides riscv64 packages and dependencies required by the PyTorch build and test environments. This reduces the effort required to provision native RISC-V systems and improves the availability and reproducibility of software artifacts across the broader RISC-V ecosystem. + +We are also contributing fixes identified during the adaptation of the native build environment back to the upstream PyTorch community, including: + +- [https://github.com/pytorch/pytorch/pull/173663](https://github.com/pytorch/pytorch/pull/173663) + +- [https://github.com/pytorch/pytorch/pull/178778](https://github.com/pytorch/pytorch/pull/178778) + +### 2.2 Building a Continuous Test Pipeline on Physical RISC-V Hardware + +With a reproducible native build workflow in place, we established a continuous testing pipeline on physical RISC-V hardware. The pipeline supports both frequent core-test validation and broader full-suite regression testing against upstream PyTorch development. + +1. **Parallel sharding of the test suite:** Jenkins distributes PyTorch tests across multiple physical RISC-V nodes and executes them in parallel. A core test run currently takes approximately three hours, while a full test run takes approximately five hours. This makes routine large-scale regression testing on real RISC-V hardware practical. + +2. **Dynamic sharding based on historical execution times:** Upstream PyTorch CI uses internal test timing data to balance workloads across workers. Because this data is not directly available to external CI systems, our pipeline automatically derives `test_times.json` from historical RISC-V test runs. Tests are then partitioned according to their observed execution times, reducing load imbalance and minimizing idle time across CI nodes. + +3. **Managing known failures through a blocklist:** A unified blocklist explicitly records tests that are currently unsupported or affected by RISC-V ISA behavior, third-party dependencies, or limitations in the runtime environment. New failures are never added automatically. Each failure must first be investigated and classified before being added to the blocklist. This ensures that known limitations remain distinguishable from newly introduced regressions and keeps CI results actionable. + +4. **Automatically aggregating failures and test logs:** Each CI run summarizes test outcomes and generates a `failures.log` file containing the failing test cases together with their corresponding raw logs. Test results are [archived centrally](https://community-ci.openruyi.cn/misc/pytorch/), allowing developers to inspect failures, compare regressions across runs, and reproduce issues more efficiently. + +5. **Continuously tracking upstream PyTorch `main`:** RuyiAI SyncBots continuously synchronize changes from `pytorch:main` into the RISC-V working branch and automatically trigger core tests. This allows compatibility regressions introduced by upstream development to be detected early, rather than postponing RISC-V adaptation until a release cycle is complete. At present, the RISC-V core test suite is passing consistently. + +6. **Contributing test-driven fixes upstream:** When RISC-V-specific issues are uncovered by core or full-suite testing, we first diagnose the root cause, implement a fix, and validate it through regression testing in the RuyiAI PyTorch fork. Once the fix has demonstrated sufficient stability, it is submitted upstream. Through this process, the CI infrastructure serves not only as a validation system, but also as a continuous feedback loop for progressively improving RISC-V support in upstream PyTorch. + +**PRs submitted upstream:** + +- [https://github.com/pytorch/cpuinfo/pull/388](https://github.com/pytorch/cpuinfo/pull/388) + +- [https://github.com/pytorch/pytorch/pull/193334](https://github.com/pytorch/pytorch/pull/193334) + +**Work currently under development and validation:** + +- RISC-V support for PyTorch quantization engines. + +## Positioning, Goals, and Related Work + +### 3.1 Positioning and Goals of RuyiAI PyTorch CI + +RuyiAI PyTorch CI is intended to provide a stable and reusable validation foundation for RuyiAI's ongoing support of diverse RISC-V processors and accelerator backends. It enables faster iteration as PyTorch is adapted to new hardware platforms, instruction-set extensions, compiler optimizations, and architecture-specific runtime capabilities. Its main goals include: + +- **Cross-validation across hardware platforms:** Validate a common PyTorch build artifact and a consistent test baseline across multiple RISC-V platforms, making it easier to identify compatibility differences introduced by processor implementations, instruction-set extensions, toolchains, system software, and runtime environments. + +- **Supporting rapid iteration of RISC-V optimizations:** RuyiAI continues to advance operator-level optimizations, compiler enablement, and support for emerging RISC-V instructions and extensions. A unified CI infrastructure provides a stable regression baseline for these efforts, enabling rapid development and continuous validation while preserving functional correctness and compatibility before and after performance-oriented changes. + +- **Preparing for emerging RISC-V accelerator hardware:** As RISC-V vector, matrix, and custom AI acceleration capabilities increasingly become available in physical hardware, the existing CI infrastructure can be extended to integrate new hardware backends and continuously validate their correctness, compatibility, and performance benefits. This provides an important foundation for building a unified software framework for RISC-V AI hardware enablement. + +In this sense, RuyiAI PyTorch CI is not only a testing system for current RISC-V platforms, but also an infrastructure layer for continuously integrating future RISC-V AI hardware into the PyTorch ecosystem. + +### 3.2 Relationship and Collaboration with RISE RISC-V Runners + +ISCAS is a member of RISE, and a member of the RuyiAI team serves as Co-Chair of the RISE AI/ML Working Group. Within this broader ecosystem, RISE RISC-V Runners provides general-purpose RISC-V CI runner infrastructure for open-source projects, while RuyiAI PyTorch CI focuses specifically on PyTorch-native builds, test orchestration, regression testing, issue diagnosis, and upstream enablement on RISC-V. + +The two efforts are therefore complementary rather than overlapping. RISE RISC-V Runners provides broadly reusable RISC-V CI resources, while RuyiAI PyTorch CI builds the PyTorch-specific validation and engineering workflow on top of physical RISC-V infrastructure. Through coordination of hardware resources, CI capacity, and upstream development efforts, the two initiatives jointly help strengthen PyTorch support for RISC-V and improve collaboration with the wider PyTorch and RISC-V communities. + +## Next Steps + +Going forward, we will continue to advance two complementary priorities: strengthening RISC-V support in upstream PyTorch and expanding RuyiAI's own hardware enablement and validation capabilities. + +- **Strengthening upstream PyTorch RISC-V CI and optimization:** We will continue collaborating with the XuanTie team, RISE, and other community partners to explore a more unified CI capability for RISC-V in upstream PyTorch. In parallel, we will continue contributing architecture enablement, operator optimizations, compatibility fixes, and performance improvements upstream, progressively improving the completeness, stability, and performance of PyTorch on RISC-V. + +- **Expanding hardware coverage in RuyiAI PyTorch CI:** We will continue integrating RISC-V hardware platforms from RuyiAI partners and establishing stable workflows for hardware adaptation, validation, and continuous regression testing. This infrastructure will provide long-term support for a growing range of RISC-V processor and accelerator platforms, while also serving as a unified validation foundation for rapid iteration across operators, compilers, runtime components, and hardware backends. + +## Contact Us + +If you are interested in RuyiAI or would like to contribute to the RISC-V AI ecosystem, please follow our [GitHub repositories](https://github.com/RuyiAI-Stack) and [official website](https://www.ruyiai.org/) for the latest project updates, technical documentation, and community activities. + +Building a thriving RISC-V AI software and hardware ecosystem requires broad and sustained collaboration across the community. The RuyiAI team welcomes contributions from developers, researchers, hardware vendors, and ecosystem partners. We also welcome internship applications on an ongoing basis, as well as opportunities for research, engineering, and project collaboration. + +For inquiries, please contact Hongbin Zhang at [hongbin2019@iscas.ac.cn](mailto:hongbin2019@iscas.ac.cn). + +## Related Links + +- [RuyiAI official website](https://www.ruyiai.org/) + +- [RFC for PyTorch RISC-V support](https://github.com/pytorch/pytorch/issues/171659) + +- [Tracking issue](https://github.com/pytorch/pytorch/issues/180975) + +- [RuyiAI-Stack/pytorch fork](https://github.com/RuyiAI-Stack/pytorch) + +- [RuyiRepo](https://github.com/RuyiRepo/ruyirepo) + +- [Disable CUDA bindings on riscv64](https://github.com/pytorch/pytorch/pull/173663) + +- [Restrict MKL to x86](https://github.com/pytorch/pytorch/pull/178778) + +- [PyTorch RISC-V CI results archive](https://community-ci.openruyi.cn/misc/pytorch/) + +- [Read L1/L2 cache sizes in cpuinfo](https://github.com/pytorch/cpuinfo/pull/388) + +- [Improve NaN sign-bit handling](https://github.com/pytorch/pytorch/pull/193334) + +- [Add RISC-V blacklist (draft)](https://github.com/pytorch/pytorch/pull/195345) diff --git a/blog/pytorch-riscv-ci.md b/blog/pytorch-riscv-ci.md new file mode 100644 index 0000000..d56856c --- /dev/null +++ b/blog/pytorch-riscv-ci.md @@ -0,0 +1,87 @@ +# PyTorch RISC‑V CI 实践 + +RISC-V 原生适配是推动 PyTorch 生态在 RISC-V 平台落地的重要基础。中国科学院软件研究所 RuyiAI 团队联合玄铁团队、RISE AI/ML 工作组及社区伙伴，持续推进 PyTorch 在 RISC-V 架构上的原生构建、真机测试、兼容性验证与上游社区协作。 + +目前，我们已搭建面向 RISC-V 的 PyTorch CI 基础设施，应用于 [RuyiAI fork 的 PyTorch 仓库](https://github.com/RuyiAI-Stack/pytorch)，实现核心测试集的稳定运行，并通过测试分片、Blocklist 管理和持续回归机制逐步扩大全量测试覆盖范围。同时，CI 过程中发现的架构适配问题、构建环境问题及相关修复，也在持续反馈并贡献至 PyTorch 上游社区。 + +## 背景：为什么要搭建 RISC‑V 专属 PyTorch CI + +PyTorch 官方 CI 已覆盖 x86、ARM 及主流 GPU 平台，但长期缺少 RISC-V 真机测试能力。此前 RISC-V 主要依赖交叉编译进行适配验证，虽然能够发现编译问题，却难以暴露真实运行环境中的浮点精度、计时、原子操作以及高精度复数算子等兼容性问题。 + +随着支持 RVV 1.0 的 RISC-V 处理器逐步成熟，仅验证“能否编译”已无法满足 PyTorch 生态适配需求。为此，RuyiAI 联合玄铁团队及社区伙伴，重点推进 riscv64 原生构建与真机测试、持续跟随上游的 CI 流水线，以及基于测试结果的问题修复与上游贡献。 + +相关工作始于玄铁团队在 PyTorch 社区发起的 [RISC-V 支持 RFC](https://github.com/pytorch/pytorch/issues/171659)。RuyiAI 团队随后积极参与讨论与协作，并与玄铁团队共同建立 [Tracking Issue](https://github.com/pytorch/pytorch/issues/180975)，持续跟踪和推进 RISC-V 原生构建、CI、算子优化及编译器等相关工作。 + +## 如意社区 PyTorch CI 核心工作 + +### 打通 RISC‑V 原生构建能力 + +此前 PyTorch 在 riscv64 上主要依赖交叉编译，缺少稳定的原生构建与测试流程。针对这一问题，我们逐步完善构建环境与依赖体系，打通 PyTorch 在 RISC-V 真机上的原生构建流程： + +1. **构建与测试解耦**：采用“一次构建、多机测试”方案，在高性能 RISC-V 设备上完成 Wheel 构建，并将同一安装包分发至多款 RISC-V 真机执行测试，实现原生构建、原生测试。 +2. **标准化构建环境**：以 Debian Trixie 为基线，统一 Python 3.13、GCC 14、OpenBLAS 0.3.33 等核心依赖。目前主要构建环境已基本与上游对齐，并持续推进其他依赖版本的适配。 +3. **完善 RISC-V 依赖供给**：依托 RISC-V 开源制品仓库 [RuyiRepo](https://github.com/RuyiRepo/ruyirepo) 提供 PyTorch 构建与测试所需的 riscv64 软件包和依赖，持续完善 RISC-V 生态的软件制品供给。 + +与此同时，我们也持续将构建环境适配过程中形成的修复贡献至上游社区，包括： + +- https://github.com/pytorch/pytorch/pull/173663 +- https://github.com/pytorch/pytorch/pull/178778 + +### 搭建可持续迭代的真机测试流水线 + +在完成原生构建后，我们进一步搭建面向 RISC-V 真机的持续测试流水线，使 PyTorch 上游代码能够在 RISC-V 平台上持续进行核心测试和全量回归。 + +1. **全量测试并行分片**：基于 Jenkins 将 PyTorch 测试集拆分到多个 RISC-V 节点并行执行。目前核心测试约 3 小时完成一轮，全量测试约 5 小时完成一轮，使真机全量回归能够进入常态化运行。 +2. **基于历史耗时动态分片**：PyTorch 上游 CI 使用内部测试耗时数据进行任务均衡。针对外部 CI 无法直接复用这一数据的问题，我们根据历史运行结果自动生成 `test_times.json`，按照各测试用例实际耗时进行分片，减少不同节点之间因任务量不均造成的等待。 +3. **Blocklist 管理已知失败**：对于当前尚未支持或受 RISC-V ISA、第三方依赖及运行环境影响的测试用例，通过统一 Blocklist 明确管理。新增失败不会被自动忽略，必须完成原因确认后才能进入 Blocklist，从而区分“已知问题”和“新增回归”，保证测试结果具有明确含义。 +4. **自动汇总失败与测试日志**：每轮 CI 自动统计测试结果并生成 `failures.log`，记录具体失败用例及对应原始日志，相关结果进行 [统一归档](https://community-ci.openruyi.cn/misc/pytorch/)，便于开发者直接定位和复现问题。 +5. **持续跟踪 PyTorch 上游主干**：通过 RuyiAI SyncBots 持续将 `pytorch:main` 同步至 RISC-V 工作分支，并自动触发核心测试。这样，上游代码变化导致的 RISC-V 兼容性问题可以在主干演进过程中及时发现，而不是等到版本发布后再集中适配。目前 RISC-V core 测试已保持稳定通过。 +6. **推动测试问题修复上游化**：对于核心测试和全量测试中发现的 RISC-V 专项问题，我们首先在 RuyiAI 的 PyTorch fork 中完成定位、修复和回归验证，确认稳定后再向上游提交 PR，逐步将 RISC-V 适配能力合入 PyTorch 主线。 + +已提交上游的 PR： + +- https://github.com/pytorch/cpuinfo/pull/388 +- https://github.com/pytorch/pytorch/pull/193334 + +正在开发与验证： + +- PyTorch 量化引擎的 RISC-V 支持 + +## RuyiAI PyTorch CI 的定位与目标 + +RuyiAI PyTorch CI 的核心定位，是为 RuyiAI 持续支持不同类型的 RISC-V 加速硬件后端提供稳定的验证基础，使 PyTorch 在新硬件、新指令扩展和新优化能力上的适配能够快速迭代。主要目标包括： + +- **多硬件交叉验证**：在多款 RISC-V 硬件上使用统一的 PyTorch 构建产物和测试基线进行验证，快速发现不同处理器、指令扩展、工具链和运行环境之间的兼容性差异。 +- **支撑 RISC-V 优化快速迭代**：RuyiAI 在持续推进算子优化、编译器适配和新指令支持。统一 CI 可以为这些改动提供稳定的回归基线，使优化能够快速验证、持续迭代，同时通过持续验证保障性能优化前后的功能正确性和兼容性。 +- **为定制 RISC-V 加速硬件接入做准备**：随着更多 RISC-V 向量、矩阵及定制 AI 加速能力进入实际硬件，RuyiAI PyTorch 可以在现有 CI 基础上快速接入新的硬件后端，对其功能正确性、兼容性和优化效果进行持续验证，为后续形成统一的 RISC-V AI 硬件支持体系奠定基础。 + +## 与 RISE RISC-V Runners 的关系与合作 + +中国科学院软件研究所是 RISE 的成员单位，RuyiAI 团队成员在 RISE AI/ML WG 中担任 Co-Chair。RISE RISC-V Runners 面向开源社区提供通用的 RISC-V CI Runner 服务，RuyiAI PyTorch CI 聚焦 PyTorch 在 RISC-V 平台上的专项构建、测试与持续回归。双方通过硬件资源和 CI 能力协同，共同推进 PyTorch RISC-V 支持与上游社区衔接。 + +## 后续规划 + +后续，我们将围绕 PyTorch RISC-V 上游生态建设和 RuyiAI 自身硬件支持能力两条主线持续推进： + +- **推进 PyTorch RISC-V 上游 CI 与优化能力建设**：持续与玄铁团队、RISE 及其他社区合作伙伴协作，探索建设面向 PyTorch 上游的统一 RISC-V CI 能力，并推动 RISC-V 架构适配、算子优化和性能优化成果持续贡献至上游社区，逐步完善 PyTorch 对 RISC-V 的原生支持。 +- **扩展 RuyiAI PyTorch CI 硬件覆盖**：持续接入 RuyiAI 合作伙伴的 RISC-V 硬件平台，建立稳定的适配、验证与持续回归机制，为不同 RISC-V 加速硬件提供长期支持，同时为 RuyiAI 在算子、编译器和硬件后端等方向的优化与快速迭代提供统一的测试基础。 + +## 联系我们 + +对 RuyiAI 生态建设感兴趣的伙伴们，可关注我们的 GitHub 仓库或官方主页获取最新的技术文档与项目进展。 + +RISC-V AI 软硬件生态的繁荣离不开广泛合作，RuyiAI 团队长期开放实习岗位，并诚挚欢迎各类项目合作，联系邮箱：hongbin2019@iscas.ac.cn（张洪滨） + +## 参考链接 + +1. [RuyiAI 官方主页](https://www.ruyiai.org/) +2. [PyTorch RISC-V 支持 RFC](https://github.com/pytorch/pytorch/issues/171659) +3. [Tracking Issue](https://github.com/pytorch/pytorch/issues/180975) +4. [RuyiAI-Stack/pytorch fork](https://github.com/RuyiAI-Stack/pytorch) +5. [RuyiRepo](https://github.com/RuyiRepo/ruyirepo) +6. [禁用 riscv64 的 cuda 绑定](https://github.com/pytorch/pytorch/pull/173663) +7. [mkl 仅用于 x86](https://github.com/pytorch/pytorch/pull/178778) +8. [PyTorch RISC-V CI 结果归档](https://community-ci.openruyi.cn/misc/pytorch/) +9. [cpuinfo 读 L1/L2 cache size](https://github.com/pytorch/cpuinfo/pull/388) +10. [优化 NaN 符号位处理](https://github.com/pytorch/pytorch/pull/193334) +11. [Add RISC-V blacklist (draft)](https://github.com/pytorch/pytorch/pull/195345) diff --git a/i18n.js b/i18n.js index ad9c3cc..411b060 100644 --- a/i18n.js +++ b/i18n.js @@ -45,6 +45,7 @@ "doc.addingOperators": "增加算子支持", "doc.operatorLang": "RuyiAI 算子编程语言", "doc.tritonOnRiscv": "Triton 的 RISC-V 适配", + "blog.pytorchRiscvCi": "PyTorch RISC‑V CI 实践", "doc.debug": "调试", "doc.irOutput": "中间表示输出", "doc.tritonRiscvOverview": "triton-riscv 总览", @@ -103,6 +104,7 @@ "doc.addingOperators": "Adding Operators", "doc.operatorLang": "RuyiAI Operator Language", "doc.tritonOnRiscv": "Triton on RISC-V", + "blog.pytorchRiscvCi": "PyTorch CI on RISC-V", "doc.debug": "Debug", "doc.irOutput": "Intermediate Representation (IR) Output", "doc.tritonRiscvOverview": "Triton RISC-V Overview",