feat(stubgen): Add core stub generation library - ast based solution [1/3]

[Rayahhhmed]

Summary

Have added an AST-based .pyi stub generation from Python source.

• stubgen/emit.rs: Emits PEP484 stub syntax from ruff AST nodes, preserving source annotations verbatim via byte-range slicing (will improve on this heuristic in a separate PR)
  • stubgen/visibility.rs: Filters exported names using __all__ or public-name heuristics. The purpose of this file is to control if a function / method should be exposed in the stubs with this __all__ scheme for public visibility.
• stubgen/mod.rs: generate_stub() entry point with 34 unit tests covering functions, classes, imports, overloads, type aliases, generics, and __all__ filtering.

Disclaimer: Used AI to generate the tests and to understand the structure of the codebase. AI was also used to understand the library details and help plan possible path ways with available API in the codebase so I don't have to go around looking for them.

No type inference yet! Unannotated items get Any or bare signatures. The inference layer will be added in a follow-up PR 👍

Flow visualised for AST based parsing only (look below for code example):

flowchart LR
    A["source.py"] --> B["Ast::parse()"]
    B --> C["Definitions (resolve __all__)"]
    C --> D["VisibilityFilter"]
    D --> E["emit_stmt() per statement"]
    E --> F["stub.pyi"]
    style A fill:#2d3748,stroke:#e2e8f0,color:#e2e8f0
    style F fill:#2d3748,stroke:#e2e8f0,color:#e2e8f0

Loading

Test plan

• cargo test -p pyrefly stubgen passes
• python3 test.py --no-test --no-conformance passes

Output diff:

---

Example python code (this is not commited, just a play example)

from typing import Optional, TypeVar

T = TypeVar("T")

__all__ = ["greet", "Config", "process_items"]

VERSION = "1.0.0"

_CACHE = {}

class Config:
    DEFAULT_TTL = 300
    name: str
    debug: bool

    def __init__(self, name: str, debug: bool = False) -> None:
        self.name = name
        self.debug = debug

    def is_production(self):
        return not self.debug

    @property
    def display_name(self) -> str:
        return self.name.upper()

def greet(name):
    return f"Hello, {name}!"

def process_items(items, transformer, default=None):
    results = []
    for item in items:
        try:
            results.append(transformer(item))
        except Exception:
            results.append(default)
    return results

def _internal_helper(x):
    return x * 2

• Pyrefly stubgen:

from typing import Any
__all__ = ['greet', 'Config', 'process_items']
VERSION: Any = ...
class Config:
    DEFAULT_TTL: Any = ...
    name: str
    debug: bool
    def __init__(self, name: str, debug: bool = ...) -> None: ...
    def is_production(self): ...
    @property
    def display_name(self) -> str: ...
def greet(name): ...
def process_items(items, transformer, default=...): ...

• MyPy stubgen

from _typeshed import Incomplete
__all__ = ["greet", "Config", "process_items"]
VERSION: str
_CACHE: Incomplete
class Config:
    DEFAULT_TTL: int
    name: str
    debug: bool
    def __init__(self, name: str, debug: bool = False) -> None: ...
    def is_production(self): ...
    @property
    def display_name(self) -> str: ...
def greet(name: Incomplete) -> Incomplete: ...
def process_items(items: Incomplete, transformer: Incomplete, default: Incomplete = None) -> Incomplete: ...
def _internal_helper(x: Incomplete) -> Incomplete: ...

Issue

#2133

PR Train (If there is a better way to break this PR pls lmk 🙏) :

• AST parsing solution: feat(stubgen): Add core stub generation library - ast based solution [1/3] #2721
• CLI commands for ast: feat(stubgen): Wire up pyrefly stubgen CLI command [2/3] Rayahhhmed/pyrefly#4
• Inference for unanotated vars: feat(stubgen): Add type inference for unannotated items [3/3] Rayahhhmed/pyrefly#5

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pip (https://github.com/pypa/pip)
-  WARN iterative_resolve_scc: SCC CalcId(pip._vendor.msgpack.fallback, /home/runner/work/pyrefly/pyrefly/primer_base/projects/pip/src/pip/_vendor/msgpack/fallback.py, KeyClassField(class1, _unpack)) exceeded 5 iterations; committing last answers

cloud-init (https://github.com/canonical/cloud-init)
+  WARN iterative_resolve_scc: SCC CalcId(cloudinit.sources.helpers.aliyun, /home/runner/work/pyrefly/pyrefly/primer_base/projects/cloud-init/cloudinit/sources/helpers/aliyun.py, Key::Definition(_process_dict_values 85:9-29)) exceeded 5 iterations; committing last answers

[meta-codesync]

@yangdanny97 has imported this pull request. If you are a Meta employee, you can view this in D95793377.