Bump supported cpython version to 3.14 for testing

[clin1234]

This is to mainly silence this abominable error:

        error: the configured Python interpreter version (3.14) is newer than PyO3's maximum supported version (3.13)
        = help: please check if an updated version of PyO3 is available. Current version: 0.23.1

[davidhewitt]

Thanks for the PR. I am not comfortable bumping this check at this point. If we did, unsuspecting users will likely install software built with the next PyO3 version (call it 0.24) when 3.14 stable lands... and they will likely have a terrible experience with random crashes and other frustrating hard-to-debug and dangerous issues.

At the same time, I appreciate the desire to get testing with 3.14 and that PyO3 should not be a blocker for this. See #4662 (comment)
for my latest suggestion on what to do here.

We also have an undocumented environment variable UNSAFE_PYO3_SKIP_VERSION_CHECK=1 which can be used for development / testing. We could document that option, though with the heavy caveat that it's highly likely any unsupported version of Python will crash pretty quickly.

If you are testing PyO3 with 3.14 you may need to make patches to pyo3-ffi to match any changes in the current alphas. I think I'm ok with having such patches in PyO3 to allow testing to make progress, though of course we will not guarantee any stability until at least the beta versions land.

[clin1234]

To investigate:

• Following tests fail in 3.14 with Rust stable in the form akin to

internal error: entered unreachable code
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869\library/std\src\panicking.rs:665
   1: core::panicking::panic_fmt
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869\library/core\src\panicking.rs:76
   2: core::panicking::panic
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869\library/core\src\panicking.rs:148
   3: pyo3::instance::Borrowed<pyo3::types::string::PyString>::data
             at .\src\types\string.rs:408
   4: pyo3::types::string::impl$2::data
             at .\src\types\string.rs:313
   5: pyo3::types::string::tests::test_string_data_ucs1::closure$0
             at .\src\types\string.rs:676
   6: pyo3::marker::Python::with_gil<pyo3::types::string::tests::test_string_data_ucs1::closure_env$0,tuple$<> >
             at .\src\marker.rs:412
   7: pyo3::types::string::tests::test_string_data_ucs1
             at .\src\types\string.rs:674
   8: pyo3::types::string::tests::test_string_data_ucs1::closure$0
             at .\src\types\string.rs:673
   9: core::ops::function::FnOnce::call_once<pyo3::types::string::tests::test_string_data_ucs1::closure_env$0,tuple$<> >
             at C:\Users\runneradmin\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ops\function.rs:250
  10: core::ops::function::FnOnce::call_once
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869\library/core\src\ops\function.rs:250
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

• types::string::tests::test_string_data_ucs1
• types::string::tests::test_string_data_ucs1_invalid
• types::string::tests::test_string_data_ucs2
• types::string::tests::test_string_data_ucs2_invalid
• types::string::tests::test_string_data_ucs4
• types::string::tests::test_string_data_ucs4_invalid
• clippy failing on beta Rust toolchains with

   error: this looks like a formatting argument but it is not part of a formatting macro
   --> src/lib.rs:3:22
    |
  3 |       feature = "nightly",
    |  ______________________^
  4 | |     feature(auto_traits, negative_impls, try_trait_v2)
    | |_^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#literal_string_with_formatting_args
    = note: `-D clippy::literal-string-with-formatting-args` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(clippy::literal_string_with_formatting_args)]`

[clin1234]

@davidhewitt Might need your set of eyeballs, something has changed significantly in regards to 3.14 with strings representation(?)

[davidhewitt]

This is a great example of the kind of churn which premature 3.14 support will burden us with.

The layout of PyUnicodeObject will have changed internally, so I suspect that PyUnicode::data is now UB.

Personally I'm disinterested in fixing it for an alpha which might yet churn again, I suggest just disabling this API on 3.14 with a note that we would prefer not to reintroduce it.

[ZeroIntensity]

It's great to test in alphas, but PyO3 probably isn't one of the projects that should do that, at least for the non-stable APIs; we mess with the internals all the time in alphas. I'd wait until the beta freeze (or to at least get close to it) before trying to test 3.14.

I also personally don't think PyO3 support for it would help catch too many bugs:

• Bugs tend to be in the new features, not the old ones that have already been tested. Unless you're planning on trying to also add PyO3 support for all the new C API functions (which I really don't recommend), then it's unlikely that the early birds will catch that much. (And for the minority that do catch regressions, it's probably not worth the burden on PyO3.)
• The other issue is that there's a bit of a programming language barrier between Rust developers and C(Python) developers. There have been several Rust-related reports in the past where it's difficult to tell whether something is a user-error, a PyO3 issue, or an actual core problem. I suspect introducing PyO3 support for the 3.14 alphas will cause quite a bit of noise, and we just don't have enough Rust experts on our end to efficiently triage all the issues.
• Tangentially related to that point, I don't know how much PyO3 exposes implementation details. If you guys expose/use some private APIs on your end, we're bound to get reports about it if we break them.

All that said, it might be worth trying to support 3.14 early for the limited API only, because we have to ensure stability for that anyway. I don't know how feasible that is for you guys, though.

[clin1234]

CI passes, might need to consolidate commits

[codspeed-hq]

CodSpeed Performance Report

Merging #4811 will not alter performance

_{Comparing clin1234:main (d924563) with main (241080a)}

Summary

✅ 87 untouched benchmarks

[ngoldbaum]

Good news! I was able to fix ffi-check on the GIL-enabled build. There are still a ton of FFI updates that need to be done that aren't checked by the ffi-check macros, but at least there aren't any ABI mismatches any more.

I got part way through updating the FFI for the GIL-disabled build but there have been some pretty substantial changes to refcounting (at least, there might be more) and it's requiring a bit more of a fine-toothed perusal of the headers to make sure everything is right.

I'll go ahead and push just the updates for the GIL-enabled build and will come back another day with the GIL-disabled updates as well as some substantial refactoring to reflect upstream refactoring and changes.

[ngoldbaum]

btw, you may want to redo this PR so it isn't from the main branch on your fork, unless you don't mind me pushing to main to update this PR.

[ngoldbaum]

but there have been some pretty substantial changes to refcounting

Looking again, I think it's mostly just a refactor? So maybe not so bad.

[ngoldbaum]

(accidentally posted this on the issue instead of here, sorry for the double notification)

The last pushes fix ffi-check on the free-threaded build but unfortunately it breaks it on the GIL-enabled build.

@davidhewitt it looks like CPython made a change that breaks ffi-check, similar to how it was broken in Python 3.12 - ob_refcnt changed representation once again. Now it's a union that contains an inline anonymous struct and I'm not sure how to modify the ffi-check macro to account for this.

See here inside CPython for what I'm talking about.

In ffi-check there's this code you wrote in 2023:

pyo3/pyo3-ffi-check/macro/src/lib.rs

Lines 148 to 155 in 241080a

	let bindgen_field_ident = if (pyo3_build_config::get().version >= PY_3_12)
	&& struct_name == "PyObject"
	&& field_name == "ob_refcnt"
	{
	// PyObject since 3.12 implements ob_refcnt as a union; bindgen creates
	// an anonymous name for the field
	Ident::new("__bindgen_anon_1", Span::call_site())
	} else if struct_name == "PyMemberDef" && field_name == "type_code" {

Do you happen to know how I should I modify it now that there's an inline struct in the union as well?

[clin1234]

btw, you may want to redo this PR so it isn't from the main branch on your fork, unless you don't mind me pushing to main to update this PR.

Go ahead