Stability-focused test infra strategy

[webknjaz]

What's the problem this feature will solve?

The CI has been prone to external influence for a long time. This is due to unpinned deps. Pip specifically. Historically, we've been running the tests against a pip version that was pinned to a minor release, and the latest release, plus the main branch of the pip repo.

This turned out to be unsustainable since it causes PR runs to become unpredictable over time. Frankly, this is at odds with the reproducibility promise that pip-tools itself seeks to implement for its users.

It also contributes to other CI problems accumulating over time and blocks progress in many other aspects.

Describe the solution you'd like

I want to make sure that the CI runs in PRs and pushes use pinned pip, short term. With the future possibility to have more lock files, long-term.

With that in mind, I want to let the nightly runs keep hitting newer pip versions.

Alternative Solutions

…suffering the status quo?

Additional context

N/A

[webknjaz]

One of the problems found while looking into fixing the CI: #2141.

N.B. I'm going to prepare a series of separate PRs and then a PR merging-in all those branches into one. The combined one is what will need to be approved and merged.

[webknjaz]

Found out that we're hitting a problem with the conditionals relying on ${{ github.job_workflow_sha }} which would have to be rewritten as a hack: actions/runner#2417 (comment).