directives_type_checking.py: do not assert that type checkers must ignore all type errors in if not TYPE_CHECKING blocks

[AlexWaygood]

This test currently asserts that type checkers should not emit errors for clearly incorrect code in if not TYPE_CHECKING blocks. Maybe that's reasonable behaviour, and maybe it's not, but it's never been specified in any PEP and it's not currently mandated by the spec. If we want to enforce that type checkers have to have this behaviour, then I think it should go via the normal process for modifications to the spec. In the meantime, I don't think it should be something asserted by the conformance suite.

This PR changes directives_type_checking.py so that it still asserts that type checkers should understand the TYPE_CHECKING constant as always-true, but so that it no longer asserts that type checkers should silence all diagnostics in these code regions, which goes far beyond what the spec mandates.

[JelleZijlstra]

It's true that the spec doesn't say anything here, but I do feel that ty is in the wrong for emitting errors within if not TYPE_CHECKING. In my view, the point of doing if not TYPE_CHECKING is to do crazy runtime things that the type checker may not understand.

[AlexWaygood]

In my view, the point of doing if not TYPE_CHECKING is to do crazy runtime things that the type checker may not understand.

I agree that that's one of the things it's often used for. And for this very reason, we deliberately suppress many errors in unreachable regions that we would normally emit in regions we infer as being reachable. We just don't suppress literally all errors in these regions — but that's what the test here currently mandates.

[rchen152]

Hmm. I agree with Jelle that I don't think type checkers should emit errors in if not TYPE_CHECKING blocks. It's true that the spec doesn't say anything about it, so I wouldn't object too strenuously to this being taken out of the conformance tests, but it seems like it might be better to change the spec.

[AlexWaygood]

I'm open to a spec change here if somebody wants to propose that. I agree that there's an argument for doing so. Until somebody makes such a proposal, however, I still don't personally think that ty should be marked down on its conformance score due to behaviour that is not mentioned in the spec or any accepted PEP

[davidhalter]

I'm open to a spec change here if somebody wants to propose that. I agree that there's an argument for doing so. Until somebody makes such a proposal, however, I still don't personally think that ty should be marked down on its conformance score due to behaviour that is not mentioned in the spec or any accepted PEP

Can I make the same argument for the type: ignore[...] issue that affects Zuban and Mypy (#2153)? It is currently also not specified. I'm open to having the spec being the single source of truth, but in that case I would like to be consistent. I'm a bit torn on this issue, because it feels like the spec is kind of incomplete.

It feels like it's very hard to get spec changes merged and I therefore doubt that these kind of things are going to be added, since it takes a lot of effort to get even tiny changes through. I don't think this is necessarily a bad thing, for the record.

[AlexWaygood]

We recently changed our behaviour in ty to suppress all diagnostics in unreachable code (except for some very special-cased ones such as reveal_type diagnostics). So this PR is no longer necessary for ty's conformance score.

I still personally feel inclined to land it regardless, on the grounds that the conformance suite shouldn't be asserting things not mentioned by the spec; this limits the room for type checkers to experiment with behaviour and hurts innovation. But I don't feel too strongly anymore either way, especially if folks are interested in changing the spec on this point to more clearly state that all type-checking errors should be ignored in if not TYPE_CHECKING blocks.

[AlexWaygood]

(I also pushed some updates so that, rather than deleting the old tests outright, it just makes them optional assertions. That way we still document whether the tested type checkers suppress diagnostics in if not TYPE_CHECKING blocks -- as they currently all do -- without mandating that they must.)

[carljm]

I agree with @JelleZijlstra and @rchen152 that type checkers should not emit type diagnostics in if not TYPE_CHECKING blocks, and that it would make sense to just specify this. I posted on Discuss, since that's part of the process for a spec change: https://discuss.python.org/t/clarifying-that-type-checkers-should-not-emit-diagnostics-in-if-not-type-checking/106902

I would favor waiting the required week for any discussion there, then updating this PR to remove the # E? and update the spec to require silencing diagnostics in if not TYPE_CHECKING.

@davidhalter regarding #2153, I certainly support that change in some form, just haven't taken a close yet since the PR is still in draft.