Bot comment on build failure

[mmarchini]

(not when the job fails to start, but when the tests or build fails, so it wouldn't revert #238)

node-core-utils has this great tool ncu-ci pr, which can be used to get a list of failing tests. We could use the GitHub Bot to comment the result of ncu-ci pr back on the Pull Request when a build fails, so it's easier for folks to see which errors happened. Or the bot could edit it's own comment, to avoid notifications, but personally I would like to get notified (I understand folks might be opposed to it though).

[phillipj]

To ensure I understand you correctly and having in mind I haven't used ncu-ci before; in addition to the inline PR statuses shown below, the Github Bot would also post a comment providing more details about the actual test(s) that failed? Meaning not just the fact that a job failed like the PR statuses says?

[mmarchini]

Exactly. It would look like this: nodejs/node#32116 (comment)

Edit: I updated all Bot comments on nodejs/node#32116 to also include the failures, if anyone wants to see how "the bot could edit it's own comment" would look like.

[mmarchini]

I just thought of two other huge benefits of this:

1. We'll keep a history of CI failures (since Jenkins deletes old CI runs)
2. When searching for a flaky test, every PR where the test failed will show on the results (which will make it easier to see if the test is flaky than navigating Jenkins to find each occurrence of that test failing).

[phillipj]

Well said!

I definitively see the value of trying what you're proposing. Learning from well intentioned changes we've introduced before, that ended up getting reverted later primarily because we didn't reach out to core collaborators beforehand, I'm wondering if we could do some kind of collaborator vote or similar before jumping onto implementing this?

[mmarchini]

That might be a good idea. I also think the "edit comment" idea would be more likely to be well received. I'll start a poll, thank you for the suggestion.