Add CI to ensure all `requires_` tests are run at least once

[VeckoTheGecko]

What is your issue?

I was looking at #11251 and realised that we have tests marked as requires_cupy yet in CI have no environment providing cupy meaning these tests are (I assume) always skipped.

I think that all tests that are skipped due to a requires_ condition should either:

• pass once in one of the environments, or
• be skipped for another reason that isn't a requires_ reason

I think this can be achieved by using the existing pytest-reportlog extension, saving the resulting files, and then having a job afterwards that combines all these artifacts together again to ensure that these conditions are held. This will have informative error messaging to mention which test is never run in CI.

This will change our testing graph structure

flowchart LR
    Lock --> T1["Test (env 1)"]
    Lock --> T2["Test (env 2)"]
    Lock --> T3["Test (env ...)"]
    T1 --> c["Combine results and report status"]
    T2 --> c
    T3 --> c

Loading

which also means we can have a single required check on "Combine results and report status"

Other concerns:

• Can we use Pixi to define environments that only work on certain platforms? (e.g., cupy isn't available on Mac, but we shouldn't drop mac from the pixi.toml:workspace.platforms
• Are there certain things that we want to test against that aren't possible due to GHA not having compatible runners?

cc @keewis (since I've mainly worked with you on ci tooling)

[VeckoTheGecko]

I'm happy to work on this. Just leaving for now so it can be discussed

[VeckoTheGecko]

Hmm, looking at https://github.com/xarray-contrib/cupy-xarray/actions I don't see any GHA workflows for testing O_o

perhaps due to incompatibility between GHA's platform with cupy dependencies? cc @dcherian

[keewis]

the problem with requires_cupy specifically is that cupy requires specialized hardware (CUDA compatible GPUs) to run, which I don't think are available for free on GHA. AFAIR cupy-xarray has scheduled jobs run on NVIDIAs hardware somewhere (but I may be wrong). cc @jacobtomlinson, since I believe you were the one who set this up?

Can we use Pixi to define environments that only work on certain platforms?

Yes, see xarray's nightly environment: the nightly feature only exists for unix-likes. Not sure if we can do that without duplicating the dependency list between win-64 and linux-64 targets.

In general, it may be nice to have some way of pointing out tests that don't ever run in CI but are not explicitly skipped (i.e. any tests marked with pytest.mark.skipif, but not those with pytest.mark.skip).

[jacobtomlinson]

@keewis there is no CI on cupy-xarray for this reason 😅. You can get NVIDIA hardware on GitHub Actions now, but it costs money. Although you can cap it. I know scikit-learn has theirs configured for $50/month and if CI runs too much and they hit that limit they just don't get GPU CI until the next month.