ENH: Add an output parameter for ``coverage``

[ganesh-k13]

ENH: Add an output parameter for coverage

Today spin test --coverage hardcodes the output to HTML in a default location:

spin/spin/cmds/meson.py

Lines 622 to 626 in 1713846

	pytest_args = [
	*pytest_args,
	"--cov-report=term",
	f"--cov-report=html:{coverage_dir}",
	f"--cov={package}",

Proposal

Enhance the coverage flag to take <format>:<location> as input. So some of the options will look like:

spin test --coverage # Default behavior for backward compatibility
spin test --coverage xml # Only format, default output to `{coverage_dir}`
spin test --coverage xml:/var/tmp/cov.xml # Format+location

This will mainly help in CI builds to upload artifacts

[ganesh-k13]

As per numpy/numpy#30911 (comment), we can introduce a new coverage command as well (cc: @mattip)

Both looks good to me, I can implement based on what everyone thinks: 1. make --coverage an extended param of spin test or 2. Introduce spin coverage

[ngoldbaum]

I like the idea of a spin coverage command that wraps spin test.

[stefanv]

What do we gain by adding a coverage command, with the test command already supporting coverage? Not necessarily against it, but trying to preserve the simplest design possible that supports given features.

[ganesh-k13]

It's a good point. In it's current form in #316, coverage only runs for python and makes more sense to extend test itself.

But if we were to extend it to other languages, mainly C, it might be helpful. One workflow I was thinking of was:

spin build --gcov # Gets .gcno files
# Run some manual tests to see which functions are hit
# This generates .gcda files
spin coverage --gcov-format html

This is entirely possible today with spin test, but it overloads the test command.

[ngoldbaum]

What do we gain by adding a coverage command, with the test command already supporting coverage?

One is that it becomes a little easier for downstream projects to add new options specifically for dealing with coverage without polluting spin test --help with more options unrelated to coverage.