C API Working Group and plan to get a Python C API compatible with alternative Python implementations?

Core Development C API
1

The C API Working Group produced a PEP (PEP 733 – An Evaluation of Python’s Public C API | peps.python.org) entitled “An Evaluation of Python’s Public C API”.

Part of this PEP is dedicated to discuss issues related to the CPython C API. The issues are grouped in different categories. The category API Specification and Abstraction is especially important for long-term potential Python performance improvements.

I won’t try to summarize the PEP and associated issues in GitHub - capi-workgroup/problems: Discussions about problems with the current C Api, but basically this is about the fact the CPython C API is incompatible with a lot of strategies to improve performance of interpreters (strategies used for different dynamic languages).

Without a proper solution to these issues, I’m a bit pessimistic about the possibilities that:

  • alternative Python implementations like PyPy and GraalPy become more usable and useful in practice for a lot of Python users.
  • the Faster CPython project leads to major performance improvements, especially when Python extensions are used (i.e. for basically all Python codes related to science and data).

I try to write a text on this subject but here my question is more specific.

I observed the huge work done by the C API Working Group to produce PEP 733.

I know that HPy could be an answer to these problems. I aware of GitHub - markshannon/New-C-API-for-Python: Design and discussion for the new C-API for Python. Unfortunately, these projects are no longer very active.

What are the next steps for the C API Working Group?

Is there a long-term plan to get these issues fixed in few years? Is there a plan to make the swift from a Python ecosystem using CPython for nearly anything to a Python ecosystem compatible with advanced and faster Python implementations?

2

(Speaking for myself without having checked with the rest of the WG.)

A “few” years is short-term for Python. We’re looking more at the 10+ year scale for genuine fixes, and have been focusing for now on stemming the growth in random directions of the current API.

You can see some of the more forward-looking discussion right here in this same category - just click on the “C API” link at the top of this page. There’s also our repository where we discuss specific designs (if not clear, the WG is mostly reactive to proposals, and while some of the members are also actively proposing improvements, anyone is allowed to bring them - the WG exists to answer the question of “who should review it”), and the issue trackers on other repositories in that GitHub org have various proposals and discussions for long-term designs.

That said, as our priority is to not break existing code without a good reason, and also to not increase our maintenance burden without the prospect of later reducing it (for example, by deprecating/removing something whenever we add something else), you aren’t going to notice a lot of changes in the code base.

I think I can summarise our current vision/guiding goal/north star right now as:

  • incrementally lead native extension modules to use stable and reliable APIs[1] that do not prevent innovation on the core interpreter loop

It turns out there’s plenty to worry about here! But there are a few things that this overarching goal does exclude or deprioritise:

  • we aren’t treating native extension performance as the top concern, but Python performance and extension stability (this doesn’t mean we try to slow down native code, but the choices are usually like “we can give extension modules a 1% speedup but never modify the interpreter again”, and we have decided that the interpreter wins)
  • we aren’t focusing on embedders (I personally am, but the assumption is that most embedders are going to be spending most time in native modules anyway, and so they get the benefits there)
  • we aren’t actively improving the core runtime ourselves (other groups are)
  • we are more concerned with (C or non-C) extension modules being loaded into CPython than with CPython modules being loaded into non-CPython runtimes (but we try not to do API designs that will only work for CPython or from C)

And probably more examples of things that people might be hoping to see but aren’t getting.

I’ll let other WG members chime in as they want. Hopefully that goes some way to answering your question, but if you have any specific queries, this is as good a place as any to bring them up.

  1. I’d have said “Stable ABI” or “Limited API” if I’d meant either of those specific things. I’m just using “stable” as an adjective here. ↩︎

8 Likes
3

There is a slow on-going effort to hide implementation details. Replace code accessing directly structure members with new getter functions. Add new functions which don’t expose implementation details. It should be easier to implement these functions on PyPy and GraalPython, and it should make it possible (in the long term at least) to evolve Python internals to develop new optimizations.

There were multiple atttemps to replace the C API with a new one, but it seems like we have to support the C API in the long term (to support the great diversity of existing C extensions). So we can only make tiny changes following the slow PEP 387 deprecation process to evolve the C API.

Maybe things will change and prove me wrong, but so far changes are incremental and slow. Steadily the C API is evolving to become better :wink:

A problem with adding new functions is that projects don’t want to use them until they are available for the oldest Python version that they support. I wrote the pythoncapi-compat project to make new functions available on older Python versions, so it becomes possible to use new functions as soon as possible. For example, it provides Py_HashBuffer() function for Python 3.13 and older, wheras the function was only added to Python 3.14. The pythoncapi-compat project also supports PyPy.

There is a work-in-progress to add more public C API for Cython (example). Currently, Cython has to consume the internal C API which some practical issues. This topic is being discussed for a few years, but it’s complicated so it’s not solved yet.

Good progress was made to make the limited C API more complete and more usable in practice.

On my side, I would like to replace direct access to PyThreadState structure with getter and setter functions, but I didn’t start to work on this topic yet. Projects like greenlet and Cython access directly to PyThreadState.

You might need to elaborate on which problem should be solved here. From my point of view, PyPy already solved the problem: it supports the C API with cpyext. The main issue is that the emulation is slow and that’s why HPy was created.

HPy has a nicer API and ABI, and is faster on PyPy, but it takes time to port existing C extensions. Python continues to support its C API in the long term.

3 Likes
4

First, I’d like to thank you @vstinner and @steve.dower for your answers. Is it remarkable that a small question from a random guy get (so quickly) so interesting and informed answers. This is very much appreciated.

I’m going to add few remarks from my point of view of user of the scientific Python stack (and dev of few packages in this field, and Python teacher).

What would change my life in terms of Python usage would be to be able for applications needing some packages of the scientific Python stack to use PyPy and GraalPy (nearly) as easily as I use CPython today and to get the benefit in terms of speed of these alternative implementations even when extensions are used.

Currently,

  • alternative Python implementations are very slow as soon as extensions are used (even a bit, like one poor Numpy float in pure Python code)
  • Most projects do not upload wheels for PyPy and GraalPy (famous examples: Scipy and Pandas)

A proper solution to get a Python stack adapted for modern interpreters is clearly something like HPy. I write “something like HPy” since I don’t know if it should be HPy or an alternative project inspired by HPy. But I think the “universal wheels” of HPy are a key aspect that should be taken into account. Also gradual porting and better debugging. HPy has a lot of nice qualities.

Of course, the question of when this could be possible is important. It has strong practical implications if one can get that in something like 3 years or in something like 10 years. In 10 years, it is probable that Mojo will be properly open-source and quite mature. I take this example (Mojo) but I guess you see what I mean. Will it still be reasonable in few years to use Python for new scientific software projects needing performance? If Python users needing extensions have no other choice than using interpreters using or emulating the memory layout used in old CPython and ref counting, it might not be the case.

So it seems to me that we should try to find a fast path towards a Python stack adapted for modern interpreters.

The process that you describe seems really reasonable from the point of view of CPython (and thus the whole Python ecosystem) but it is also important that some projects (in particular Numpy) can go faster.

Of course, the changes in the CPython C API going in the direction of a C API adapted for modern interpreters seem very useful, especially if they can be used for older Python versions with pythoncapi-compat.

However, I’d like to point out that the current situation and plan of the C API working group introduce an uncertainty which does not help to start now an adaptation of some packages.

Let’s take the point of view of Numpy (which is really the key project to go towards a Python stack adapted for modern interpreters):

  • Should we invest on HPy?
  • Should we wait for another new CPython C API officially supported by the C API WG?
  • Should we just slowly follow CPython using pythoncapi-compat?

It is even worth than that! Numpy 1 has been ported to HPy (at least nearly, I don’t know if all tests passed and there is also the question of the Numpy C API) but now the project is stalled for different reasons, but I guess in particular because of uncertainties about the direction to be taken regarding the Python C API. One could find some founding for such important and impactful tasks, but it is difficult if the direction is not clear.

It seems to me that to get “universal Numpy wheels” in few years, the C API WG should officially support HPy or something like HPy, i.e. a layer above the CPython C API that could be used now to port C extensions while supporting a reasonable range of Python versions. I realize that it is a very big thing and that it would need a specific PEP.

I’m going to finish by warning that all this is only a modest point of view of a Python user without experience of the CPython C API for anything serious!

3 Likes
5

I think you’re making a very important point here. Even I as a CPython core developer wouldn’t be able to answer these questions, and that’s a problem.

2 Likes
6

Great questions, and some very important context none of us have mentioned yet is that libraries that want to move faster can move faster, but CPython has to change at the rate of the slowest moving ones. If everyone were as responsive as numpy, we could dramatically accelerate any improvements, but alas, they are not.

What you’re asking about here is which level of abstraction should libraries adopt. There are some great choices out there, but they all come with tradeoffs, mostly (in this space) between speed and maintenance. To offer a more complete list (in order):

  • internal C API (highest performance, most maintenance, least portable)
  • CPython C API
  • pythoncapi-compat
  • HPy[1]
  • pybind11, nanobind, PyO3[2]
  • Cython[3]
  • limited C API/stable ABI
  • Python code (lowest performance, least maintenance, most portable)

Each individual library will also have their own constraints on which options are viable, but ultimately, this is how you choose which one to use.

If you choose the CPython C API, you’ll be able to achieve better performance (though we’re working to close that gap by providing better APIs for code generators), but you’ll spend more time chasing our changes. If you choose, say, nanobind[4], then once you’ve adapted to it you’ll get to use new CPython APIs automatically, potentially at a slight cost to performance, and should (eventually) be able to recompile for other runtimes and get peak integration there, too. HPy is in between these two levels, but already offers the recompiling (and no doubt some level of new API adoption, though I haven’t checked that out).

And of course, you can totally mix and match. Most projects do, by combining Python code and some form of native code, but if there’s value in using Cython for a lot, HPy for a few parts and Python for the rest then there’s no reason you can’t do it.

What we as the C API WG can’t do is to force people into this. As I said, we have to work at the pace of the slowest adopters, not the fastest. But we can and do encourage fast adopters to use something other than our oldest APIs.

Very minor point of clarification:

another new CPython C API officially supported by the C API WG

I’d say the C API WG would endorse a new CPython C API, which is one legitimate interpretation of “supported by”. But the other interpretations (cared for, maintained by, etc.) are by the entire maintainer team (core developers). The role of the WG is to review changes/proposals and ensure consistency across the codebase, not necessarily to be the only ones who work on the API.

  1. Ranked separately from the other bindings due to manual reference counting and also the relative thinness of the bridge. ↩︎

  2. I’ve no doubt missed some, please don’t get mad. These are representative examples not a census. ↩︎

  3. Ranked separately from the other bindings because it’s typically used for complete implementations, not just bridging languages/interfaces. ↩︎

  4. Which seems to me to be the “highest” reasonable choice for numpy. ↩︎

3 Likes
7

I don’t want to answer for @paugier here, but I think that part (the level of abstraction) is already understood by interested third-parties. The problem is more “which alternative is future-proof” and “which timeline can I be confident on for the CPython’s C API evolution”.

For example, HPy is a tremendous idea but it doesn’t seem to receive much maintenance or development anymore, and without any official endorsement/support by the CPython team or the PSF it may simply die in the next few years. Nobody wants to start rebasing their library on a dying API.

For now the implicit message seems to be “wait for us to fix things at the CPython C API level”, there’s no clear timeline or roadmap, which leads people to just… wait indeed (same for the stable ABI vs. free-threading, incidentally).

4 Likes
8

I think the other aspect that has been mentioned, but not really picked up on, is “which alternative(s) offer the best hope for good performance across other Python implementations?”

That’s not necessarily something the CPython core developers can comment on, but it is a reasonable question for a developer wanting to write an extension to ask. And it’s something that is relevant to the CPython C API WG, in the sense that portability vs performance is often a trade-off, and if portability isn’t a priority for the WG, then using the C API may be a poor choice for developers who value that.

3 Likes
9

Thanks @steve.dower for your answer and @pitrou and @pf_moore for your reactions.

I agree that it is not just a question of “level of abstraction” and tradeoff between “speed and maintenance”.

  • We are considering important points for the relatively long-term future for Python.
  • The current direction is not satisfactory.
  • The C API WG, the PSF and CPython core devs have some responsibilities.

First, there is a big project that should involve a lot of people working at different places on Python: adapt the Python ecosystem (at least the most important pillars) to be:

  • compatible with modern methods to get good performance for Python interpreters. Some of these methods are currently used in alternative Python implementations and some could be used in the future in CPython.
  • focused on the possibility for users to use different Python implementations depending on their needs. For this, we in particular need universal extensions (like HPy ones) and a tool (something like HPy) that is able to produce standard extensions AND universal extensions from the same code.

First questions can be “Is this program interesting?”, “Is it technically doable?” and “Is it the right thing to do?”

It seems to me that the HPy project demonstrated that this is technically doable and an attractive strategy.

It seems to me that some issues of this program/project are that (i) it might not be so attractive (or at least so urgent) for most CPython core devs and (ii) CPython core devs are of course very important to decide where Python goes (This is not at all a criticism.).

We kind of see an issue by comparing the points of view written in PEP 731 – C API Working Group Charter | peps.python.org and GitHub - hpyproject/hpy: HPy: a better API for Python and in particular the meaning associated with “Python API”. For the PEP, the “Python API” clearly actually refers to the CPython C API whereas HPy proposes to design “a better API for Python”.

PEP 731 writes

a small committee of Python core developers

and I guess that this means more precisely CPython core developers, meaning that the point of view of alternative Python implementations is not directly represented. Again, it is not a criticism and just a factual (I hope) observation.

Anyway, it seems that the C API WG is focused on improving the CPython C API and fixing the different issues from CPython, which makes sense from the CPython point of view. HPy is clearly considered as a side project “below” the CPython C API, which is true from the CPython point of view but not from the point of view of alternative Python implementations (PyPy and GraalPy).

I can see three bad things about the current situations:

  • It seems to me that the alternative strategy to fix the problems from an alternative project (new API and co, HPy or something similar), has not been seriously and publicly discussed. (I might be completely wrong about that.)

  • There is no clear time schedule for the strategy “gradual fixes from CPython and pythoncapi-compat to speed up the possible consequences for users”.

  • I didn’t see discussions on how one could get the equivalent of universal extensions with the strategy “gradual fixes from CPython”.

Another point is that the C API WG (with the Steering Council) has authority. When the C API WG states (even implicitly) that the right strategy for Python is “gradual fixes from CPython”, this has in practice a strong negative effect for HPy. People and projects just wait.

I’m afraid that if there is no changes in how it is organized, Python (the language and the community) will never in practice enjoy the goodness of fast implementations. There is a time schedule in GitHub - markshannon/New-C-API-for-Python: Design and discussion for the new C-API for Python which describes a “fast path”. There is for example written “2025 (3.14): New C API implementation in CPython complete.” I would love to see a reasonable time schedule for the “slow path” (“gradual fixes from CPython”) until a satisfactory state (for example just a C API that does not require emulating fixed objects in memory and ref counting + Numpy usable with good perf on different Python implementations + universal wheels). I fear that with this slow strategy it will be too slow to be really impactful.

I have to add that I don’t think that HPy is perfect and I don’t know if this is the right solution. I know it has a lot of very good qualities. I think seeing it as a wrapper around the CPython C API (more portable but slower on CPython) is a bit unfair. A mature HPy can be much more than that and one should be able to get very good performance even on CPython for things as advanced as Numpy. Nevertheless, I also know that it has issues, for example it seems to me that PyObject and PyArrayObject are both represented by handles (HPy), which has drawbacks.

1 Like
10

As have numerous others, yes. SWIG, pybind11, Cython, nanobind, and PyO3 all come immediately to mind. HPy hasn’t really demonstrated anything new beyond those, apart from it being possible at a lower abstraction level.

Python has core developers, and those of us who contribute implementations contribute to the reference implementation, CPython. Historically, maintainers of other implementations have found themselves able to become Python core developers very easily, though as it offers very few benefits unless you actively want to work on the reference implementation, I understand why many wouldn’t bother.

Yeah, I’m afraid you are wrong about it. But more to the point, anyone can write a project to solve it from outside (hence why there are so many of them), but only the core team can prevent our core APIs from drifting to the point where they are unusable by those tools or by humans.

Correct. We can’t actually enforce a schedule on our users, so we have to move at a pace that lets us see how much we are hurting them, and be able to back up when we do. As I said, if every project was highly responsive, we would move much quicker.

This is likely true, I imagine these discussions very quickly reached the point of “it requires extension maintainers to do a lot of work, and if they want to do it then they can already do it today, so we don’t need to focus on them.”

It’s certainly not our intent to discourage the use of libraries that generate bindings. It’s possible that HPy suffers unfairly compared to the others, because it doesn’t have the ability to adapt to new APIs (e.g. Cython has transparently adopted many of our new, faster APIs, so their users automatically get the benefits, just by recompiling - this may not be true for HPy).

Actually, the great value of HPy is that the handles are purely for bookkeeping, and when you compile specifically for CPython they will turn into the minimal amount of reference counting required to keep objects safely active.

Conceptually, there’s overhead, especially if you are already used to manual reference counting and are having to learn a new model, but ultimately it’s probably a better model of object ownership than what CPython currently has.

1 Like
11

I don’t think this is true.

HPy is natively supported by PyPy and GraalPy. This is actually their preferred C API.

Except from HPy, PyPy and GraalPy do not have another C API adapted for them so without HPy other projects like those you cited have no other choices than using the CPython C API (with PyObject*, Py_INCREF and Py_DECREF) through cpyext (again cpyext is very slow). The generated code is adapted to use things that are known to work with cpyext but it is not magic.

Moreover, I don’t think there is anything like HPy universal ABI with SWIG, pybind11, Cython, nanobind and PyO3. The closest would be usage by these tools of the Limited API and Stable ABI but this is clearly much less good than the HPy universal extensions!

If HPy is successful, other projects can emit HPy code (this is the plan) and then produce universal extensions compatible with fast interpreters.

Finally the level of abstraction is not a small thing. With HPy, one can implement things like Numpy with the same performance as with the historical Python API and you can then build standard extensions (same perf on CPython) and universal extensions (much much faster on PyPy and GraalPy, since it does not use cpyext or the equivalent for GraalPy). Unfortunately, it is not possible to implement Numpy with for example Cython.

HPy is really something intrinsically different than the other tools that you cited. Without HPy (or something similar), the other tools are completely unable to improve the situation in terms of good compatibility of the ecosystem with fast Python implementations without bad solutions like cpyext.

I don’t understand what you mean. What can they already do today? How can they create something like HPy universal extensions with something officially provided by CPython (even in future)?

3 Likes
12

The pace is dictated by the whole Python ecosystem which takes time to adapt to deprecate APIs, new APIs, and other API changes. Some projects such as Cython and numpy are quick to adapt to new APIs, whereas some projects are simply unmaintained and so not updated.

It took more than a year to adapt the ecosystem to my big cleanup of Python 3.13 C API when I removed private functions. We discovered how many C extensions rely on private functions: well, way more than expected! I’m still working on adding new public C API to replace private functions.

Also, there is no clear goals or “DONE definition” for this project. It’s more like: we discover an issue, we are working on fixing it, it takes a few months to a few years to fix it. Then iterate on the next issue. Issues have different severity, we are trying to address the most impactful issues first.

5 Likes
13

The pithy answer is “implement HPy”. But more relevant is that nobody can make your extension into a universal extension solely by changing CPython - you have to change the extension module. But we have to change CPython under the assumption that almost nobody is going to change their extension module - we can’t simply cause everyone whose code worked on 3.13 to suddenly break on 3.14.

Your focus seems to be on “officially provided by CPython”, but this is an error. Don’t assume that just because some open-source project is not maintained by the Python core developers that you shouldn’t use it, and please tell other people the same thing. Non “official” projects are very important to the Python ecosystem, and anyone who thinks that a non “official” project is automatically less useful is wrong.

1 Like
14

Oh great! Could you please help me to find some links?

Excuse me but this is something that I still don’t understand. I perfectly understand that CPython (and other implementations by the way) have to continue to support the historical C API (with slow evolutions). But why would it be a problem to introduce a new better API, with another header (like with hpy.h) that could be used sooner by projects that want to and can evolve as soon as possible?

1 Like
15

Sure, pyvideo is a good way to find the public discussions: PyVideo.org

There are plenty of references to HPy right here that mostly seem related to discussing it in context of alternative or new C APIs.

I expect most other discussions have been on GitHub or in more private forums that don’t have links.

It’s been introduced - it’s called HPy. You can acquire it and integrate it into your project whenever you like.

What is it you really want? Because you’re starting to sound like the people who come asking for free maintenance and support, which the open-source community is not equipped to provide. So you may need to clarify your request.

1 Like
16

The projects I care about are not my projects. These are mainly Numpy and other things using Cython and Pythran for extensions. This is not a personal problem but an ecosystem problem. The next step is really Numpy.

I’m sorry if it sounds like that. I’m also not sure if I should interpret this as being very nice. But anyway. I think I contribute as I can and at my level to open-source and in particular to Python in different ways.

I like to understand where Python goes and how we could get more usable faster Python. With the experience I have with CPython, PyPy and GraalPy, I see how impactful it would be for many people to be able to use more PyPy and GraalPy. I see that HPy is currently kind of stopped and I try to better understand why, in particular to try to find solutions to get a result in a reasonable time.

It seems to me that it is really a collective/community problem. Many people would have to focus and invest on this issue, which would be good for the whole Python community.

It seems to me that there is in particular an issue with the position of the C API WG, which does not help so I tell you as I can. Again I might be wrong and it’s already interesting to hear that you don’t say that HPy would be a bad solution for something like Numpy.

Another step for me will be to communicate on the possible need for a transition to a scientific Python stack towards a state more adapted to alternative and faster Python implementations. And why not to be able to build projects to find founding for HPy (or something similar and better in some way).

“Official” endorsement or some kind of support by the C API WG or some Python core devs, even just moral, would be useful in that respect. But that’s true that it is not needed and things can be done without.

Since you asked, one thing that I “really want” could be an answer from the C API WG to the question “what is better for this global project (in particular for Numpy), HPy or something else?”

17

Apologies if it comes across that way, but I have to tell you how it sounds so you can help me understand how it’s meant to sound :slight_smile:

As it’s a separate project, you’ll need to talk to the maintainers. I’m pretty sure that most of them are still generally active, but they all have multiple projects and no doubt their focus moves to wherever the most activity exists.

Yeah, this still doesn’t help, and I think it’s the point that needs better understanding. The C API WG are certainly not against projects using binding generators - we prefer it! It means that we can coordinate with just a few projects who actively care about the C APIs, rather than trying to coordinate with every single project, most of whom don’t really care about the C API, they just want to get their job done. The more projects using any of the (actively maintained) binding libraries, the better!

We get a bit stuck here, because yes, we love when people use those projects. Officially, we love when people use those projects. But that doesn’t excuse us from also making the C API as usable as we can for people who choose to use it directly. That doesn’t mean we’re against people using binding generators, it just means we must do a good C API, regardless of any alternatives.

A good parallel is the Windows and macOS binaries we distribute. We have to distribute something, because we can’t responsibly outsource it entirely. But we’re not against other groups (even for-profit companies!) doing it as well, and we don’t mind when people use them (though if they have problems, we ask them to go to the people they paid before they rely on the volunteers). It’s the same with the C API - we have to responsibly provide something that’s good and usable, but we don’t do it to compete with other projects, and we often do things to make the other projects lives easier.

Ultimately, this is a highly opinionated question that we can’t answer.

Personally, virtually all of my own projects that have native code either use pybind11 or Cython. If I personally were going to build numpy, I’d probably use one of those. (I also have two projects that are essentially binding generators, and one uses the C API directly, while the other uses pybind11.)

Within CPython, we only use the C API directly, because we want to be able to build and test without 3rd party dependencies. That ensures that projects can safely make changes to adapt to CPython, without worrying that their changes will cause more things to break in CPython.

Whether numpy thinks it’s best to go with HPy, nanobind, Cython, or directly to the C API (which I think are probably the most viable choices) is entirely up to the people who will convert and maintain it. The C API WG cannot have any kind of official recommendation here, because it’s entirely outside of the WG’s scope. The numpy maintainers have to decide. We can provide suggestions or pros/cons, as people who are more familiar with the field than most, but it’s not our decision.

Does that help to clarify things?

5 Likes
18

Yes, your position is very clear. Thank you for your replies.

2 Likes
19

Hi, I am an HPy and GraalPy developer. Commenting as myself, no “official” HPy/GraalPy stance.

What is HPy, really?

It seems to me that HPy is often put into the same bucket as tools like pybind11 or Cython. The goal of HPy, in my view, is different. It is to design an API between the Python VM and 3rd party native extensions for Python.

The fact that the current reference implementation of HPy for CPython happens to be implemented on top of the CPython C API is an implementation detail that, I suspect, misleads people to see HPy like “other bindings”. To me such “bindings” are projects that provide some abstraction on top of the Python VM API (for the time being the CPython C API), but are not meant to be provided by the Python VM itself (now or in the future).

Why is this difference important? Because the goal of HPy is to enable the Python VM to hide and freely evolve its internals and still provide a performant API and stable ABI, which in the end should benefit also the packages and ecosystem as a whole.

We’ve shown that this works: in our experiments, HPy-based extensions performed significantly better on alternative Pythons (GraalPy and PyPy), because HPy allowed them to avoid the costly emulation of the CPython C API tailored for what happens to be CPython internals at this point. At the same time, HPy extensions can be binary compatible between CPython, GraalPy and PyPy. I think that this is a strong evidence that the HPy API (and ABI) would work reasonably well for any Python VM with whatever internal structure it may have.

HPy and CPython

CPython (and alternative Pythons) will have to support the CPython C API for the foreseeable future (and it is worthwhile to incrementally improve it in any case), but it doesn’t mean that CPython cannot provide an alternative API to the VM. HPy has shown that “legacy” C API and some new API can be mixed to provide a gradual migration path. If you port a single extension method to HPy, its invocations will already benefit from HPy; for example, the VM can pass tagged integers to it (like GraalPy does).

Over time, important packages would migrate (at least their most important parts) to the new API not just because it’s nice and shiny, but because it would provide real benefits on CPython (mainly performance and ABI stability once fully migrated). At that point, CPython can do what GraalPy/PyPy do: emulate the “old” API—it will be slower, but it won’t matter (so much) anymore, and CPython would be free to do almost any internal changes without breaking the ecosystem.

Current state of HPy

Right now HPy development is not very active. AFAIK it is because most of the maintainers have other commitments, but we are still keen advocates of the HPy ideas. We’ve given multiple presentations and shared our results and experiences. I believe that some of those inspired some CPython C API changes and more may inspire future changes. We are watching this space and try to contribute (these days mainly to design discussions about the CPython C API) where we see fit and think our expertise can be useful.

At this point, HPy is, in our view, a successful demonstration of the goals that drove its initial development. Now the question is: what next?

Future of HPy, Take 1

HPy can stay as a completely separate project from CPython. In such a case, the benefits of HPy for package authors would be: binary stability like stable ABI, but even for GraalPy and PyPy (and possibly any new Python VM), and potentially better performance on GraalPy and PyPy. Admittedly, for HPy to succeed on this front, we’d need more (active) contributors. The current contributors cannot allocate enough resources for HPy to fully push this forward, but I think we’ll be happy to mentor and help.

Future of HPy, Take 2

Imagine for a while that HPy or some API inspired by it would be declared as the Python VM API for 3rd party C extensions. It could be governed by the C API work group or even better by some work group consisting of representatives of CPython core devs, other Python VM devs, package authors and the community. In my opinion, in the long run, CPython would benefit and the ecosystem would benefit.

One ABI breakage is already on the horizon (Stable ABI/Limited API for free-threaded builds) and there are discussions about providing a new parallel API (Add a `PyThreadState *` parameter (almost) everywhere · Issue #132312 · python/cpython · GitHub). If it is going to happen sooner or later, why delay the effort? The migration of the ecosystem will take some time. I believe that, also thanks to HPy, we have good enough understanding of the problem to start solving it. Both linked issues (exposed ob_refcount in old stable ABI) and explicit “context” parameter would have been solved with HPy.

If such a parallel API would not simply be included in Python.h, but instead in a new PythonAPI.h, the difference could be very clear and an incremental pathway to getting people off the old Python.h APIs and onto a better designed API specifically for 3rd party native extensions might be clearer.

One approach to such a new Python VM API could be to gradually implement parts of the HPy API in CPython (possibly adjust/polish/rethink some parts, but lots of work has been already done!) and at the same time already take advantage of it for better performance (I think there are already some situations where CPython must revert some optimizations when PyObject leaks to an extension). It can start as an API for extensions that just have simple module-level methods, then it may add support for basic heap types, then module state, then meta types, etc.

This HPy future take 2 would suffer from the same problem with lack of contributors, but I think it would be in a much better position for attracting new contributors (or re-attracting the old ones :-)) and getting adoption from extension authors, because it would provide benefits (performance) over the existing stable ABI even for extensions that do not care (so much) about alternative Pythons. Hopefully this would kick off the positive feedback loop between development and adoption.

What would be the next step? I think a prototype in CPython fork/branch that would show the potential benefits can help move this forward. This would be no small project. I don’t think there is anyone in HPy right now, who would be able to allocate enough time such such project, but again if there are people willing to push this effort, I believe the current HPy maintainers would be happy to mentor and help as much as their other commitments allow.

15 Likes
20

Hello:

Thank you for the pointer to HPy. First time I hearing about it.

To me a lot of this discussion reminds me of the device driver ABI for the Linux kernel. I understand this is user space, different requirements etc.

But my fundamental point is that we could aim for source level compatibility instead of ABI compatibility where extensions can be loaded into multiple python implementations.

Why does this matter? Because of python’s popularity in the AI world and ease of machine assisted translation to compiled languages, transpilers are becoming increasingly important.

Instead of writing extensions in C, we could write them in a statically typed subset of python and transpile. The “many implementations” aspect would come from which particular compiled language you transpile to.