Argument parsing for functions

[emmatyping]

To make writing Rust functions that take Python arguments ergonomic, we need an easy way to describe how to parse arguments, and ideally convert to Rust object abstractions. The C API for argument parsing is described here: https://docs.python.org/3/c-api/arg.html

C code in CPython uses a DSL to pre-generate C argument parsing code called argument clinic

We have a few different ways we could go here. PyO3 currently uses proc-macros to provide very ergonomic argument parsing. We could also modify argument clinic to generate Rust code and include!() the generated files, which might have better compilation times.

I am still undecided about which approach I prefer.

[youknowone]

Maybe using clinic + proc-macro will is also possible.

#[clinic]
/// useful DSL
fn xyz_impl(...) 
``

Then #[clinic] can generate everything inside the macro. No `include!()` will be required.
Cons: The generated code will not be searched 

If saving generated code in the repository is important, generate the code and including it using `#[clinic]` will be also possible though.

[davidhewitt]

I have no strong opinion about the build system integration, anything is possible. I think what matters most is going to be what the Rust code we write looks like. Does it use normal Rust attributes, or something more like argument clinic special syntax?

[Eclips4]

I’m pretty much in favor of any approach except updating the Tools/clinic one. The basic problem is that CPython's clinic isn't guaranteed to work with third-party C modules. I think we can use a proc-macro instead, so we could make it usable for third-party modules that way.

[emmatyping]

I think we can use a proc-macro instead, so we could make it usable for third-party modules that way.

Yeah I think this is a good idea. The APIs we design now should eventually be stablized and made usable by the public, so proc-macros seem like a natural use case for something we'll want to make ergonomic for users. It also makes a lot of sense as we will likely be iterating on function signatures initially a lot.

If compile times are an issue, we can probably re-use the proc-macro code to generate argument parsing code that gets checked in and used via a build script hook (similar to how bindgen can generate bindings ahead of time).

[cmaloney]

Second not updating Tools/clinic. Recently I have been adding a new decorator and what I need to change and why isn't really obvious in updating AC internals. The code is effective but, to me, hard to extend.

To me there are a couple core parts to the problems AC solves today:

1. Specifying what the accepted Python function signature(s) are.
2. Performing argument type / value checking and normalization
3. Mapping the interpreter call representation (tp_call, vectorcall) to the implementation function ABI (currently C ABI) ideally in a way the compiler can see and potentially optimize.

To me if 1 can be exactly Python syntax that's actually better than the current DSL. This is very close to type stubs but hand-annotations don't always match actual implementation. For example I/O .write() stub says it returns an int but it also returns None for some layers of the I/O stack. Doesn't come up in most cases but does exist.

For 2 and 3 need something which exactly matches behavior to start at least which is harder than making something new. Ideally that can be seen/worked with by the JIT to eliminate interpreter overhead for more cases.