-
Notifications
You must be signed in to change notification settings - Fork 950
Pinbump cron #19034
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
JacobSzwejbka
wants to merge
3
commits into
main
Choose a base branch
from
pinbump-cron
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+210
−0
Open
Pinbump cron #19034
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| name: Pin Bump CI Handler | ||
|
|
||
| on: | ||
| workflow_run: | ||
| workflows: ["trunk"] | ||
| types: [completed] | ||
|
|
||
| jobs: | ||
| handle-ci-result: | ||
| if: github.repository_owner == 'pytorch' | ||
| runs-on: ubuntu-latest | ||
| environment: update-commit-hash | ||
| permissions: | ||
| pull-requests: write | ||
| issues: write | ||
| steps: | ||
| - uses: actions/github-script@v7 | ||
| with: | ||
| github-token: ${{ secrets.UPDATEBOT_TOKEN }} | ||
| script: | | ||
| const { owner, repo } = context.repo; | ||
| const workflowRun = context.payload.workflow_run; | ||
| const conclusion = workflowRun.conclusion; | ||
| const runUrl = workflowRun.html_url; | ||
|
|
||
| const prs = workflowRun.pull_requests; | ||
| if (!prs || prs.length === 0) { | ||
| console.log('No PRs associated with this workflow run. Skipping.'); | ||
| return; | ||
| } | ||
|
|
||
| const prNumber = prs[0].number; | ||
| const pr = await github.rest.pulls.get({ owner, repo, pull_number: prNumber }); | ||
|
|
||
| const isPinBump = pr.data.labels.some(l => l.name === 'ci/pytorch-pin-bump'); | ||
| if (!isPinBump) { | ||
| console.log(`PR #${prNumber} is not a pin bump PR. Skipping.`); | ||
| return; | ||
| } | ||
|
|
||
| const allowedAuthors = new Set(['pytorchbot', 'pytorchupdatebot', 'facebook-github-bot']); | ||
| if (!allowedAuthors.has(pr.data.user.login)) { | ||
| console.log(`PR #${prNumber} was created by ${pr.data.user.login}, not an allowed automation account. Skipping.`); | ||
| return; | ||
| } | ||
|
|
||
| console.log(`Pin bump PR #${prNumber}, trunk concluded: ${conclusion}`); | ||
|
|
||
| const comments = await github.rest.issues.listComments({ | ||
| owner, repo, issue_number: prNumber, per_page: 100 | ||
| }); | ||
| const fixAttempts = comments.data.filter( | ||
| c => c.body && c.body.startsWith('@claude [ci-fix-attempt') | ||
| ).length; | ||
|
|
||
| if (conclusion === 'success') { | ||
| const note = fixAttempts > 0 | ||
| ? `Claude fixed CI failures in ${fixAttempts} attempt(s).` | ||
| : 'CI passed on the first try.'; | ||
|
|
||
| await github.rest.issues.createComment({ | ||
| owner, repo, issue_number: prNumber, | ||
| body: `## CI Passed\n\nAll trunk CI checks have passed on this pin bump PR. ${note}\n\n**This PR is ready for human review and merge.**\n\ncc @jakeszwe` | ||
| }); | ||
| return; | ||
| } | ||
|
|
||
| if (conclusion !== 'failure') { | ||
| console.log(`Trunk concluded with "${conclusion}" (not failure). Skipping.`); | ||
| return; | ||
| } | ||
|
|
||
| if (fixAttempts >= 3) { | ||
| await github.rest.issues.createComment({ | ||
| owner, repo, issue_number: prNumber, | ||
| body: [ | ||
| '## Automated Fix Attempts Exhausted', | ||
| '', | ||
| `CI is still failing after ${fixAttempts} automated fix attempt(s).`, | ||
| `Failed trunk run: ${runUrl}`, | ||
| '', | ||
| 'This pin bump likely requires human intervention. Common causes:', | ||
| '- BC-breaking API changes in PyTorch that need design discussion', | ||
| '- New dependencies or build system changes', | ||
| '- Test infrastructure issues unrelated to the pin bump', | ||
| '', | ||
| 'cc @jakeszwe' | ||
| ].join('\n') | ||
| }); | ||
| return; | ||
| } | ||
|
|
||
| const attemptNum = fixAttempts + 1; | ||
| await github.rest.issues.createComment({ | ||
| owner, repo, issue_number: prNumber, | ||
| body: [ | ||
| `@claude [ci-fix-attempt ${attemptNum}/3]`, | ||
| '', | ||
| `The \`trunk\` CI workflow has failed on this automated PyTorch pin bump PR.`, | ||
| `Failed run: ${runUrl}`, | ||
| '', | ||
| 'Please:', | ||
| '1. Read the Dr. CI comment on this PR for a summary of which jobs failed and whether they are flaky. Ignore failures marked as FLAKY.', | ||
| '2. Use your CI tools to download the failure logs for the non-flaky failing jobs', | ||
| '3. Identify the root cause of the failure', | ||
| '4. If this is a build or test failure caused by PyTorch API changes, fix the ExecuTorch code to be compatible with the new PyTorch version', | ||
| '5. If this is a c10 header sync issue, the headers have already been synced by the pin bump script — the issue is likely in ExecuTorch code that uses those headers', | ||
| '6. Run `lintrunner -a` on any files you change', | ||
| '7. Push your fix as a new commit to this PR branch', | ||
| '', | ||
| 'Important constraints:', | ||
| '- Do NOT modify torch_pin.py or .ci/docker/ci_commit_pins/pytorch.txt — the pin itself is correct', | ||
| '- Do NOT modify files under runtime/core/portable_type/c10/ unless the sync introduced a new API that ExecuTorch code needs to adapt to', | ||
| '- Focus on fixing ExecuTorch code to be compatible with the new PyTorch APIs', | ||
| '- If this is a major BC-breaking change that requires architectural discussion, say so clearly and stop — do not attempt a fix' | ||
| ].join('\n') | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| name: Weekly PyTorch Pin Bump | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: '0 9 * * 1' # Monday 9:00 UTC | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| create-pin-bump-pr: | ||
| if: github.repository_owner == 'pytorch' | ||
| runs-on: ubuntu-latest | ||
| environment: update-commit-hash | ||
| permissions: | ||
| contents: write | ||
| pull-requests: write | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| token: ${{ secrets.UPDATEBOT_TOKEN }} | ||
|
|
||
| - uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.11' | ||
|
|
||
| - name: Determine nightly version | ||
| id: nightly | ||
| run: | | ||
| NIGHTLY_DATE=$(date -u -d 'yesterday' '+%Y%m%d') | ||
| NIGHTLY_VERSION="dev${NIGHTLY_DATE}" | ||
| echo "version=${NIGHTLY_VERSION}" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Read current TORCH_VERSION | ||
| id: torch | ||
| run: | | ||
| TORCH_VERSION=$(python -c "exec(open('torch_pin.py').read()); print(TORCH_VERSION)") | ||
| echo "version=${TORCH_VERSION}" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Update torch_pin.py with new NIGHTLY_VERSION | ||
| run: | | ||
| printf 'TORCH_VERSION = "%s"\nNIGHTLY_VERSION = "%s"\n' \ | ||
| "${{ steps.torch.outputs.version }}" \ | ||
| "${{ steps.nightly.outputs.version }}" > torch_pin.py | ||
|
|
||
| - name: Run pin bump script | ||
| run: python .github/scripts/update_pytorch_pin.py | ||
|
|
||
| - name: Create branch and PR | ||
| env: | ||
| GH_TOKEN: ${{ secrets.UPDATEBOT_TOKEN }} | ||
| run: | | ||
| BRANCH="automated/pytorch-pin-bump-${{ steps.nightly.outputs.version }}" | ||
|
|
||
| git config user.name "pytorchbot" | ||
| git config user.email "pytorchbot@users.noreply.github.com" | ||
| git checkout -b "${BRANCH}" | ||
| git add torch_pin.py | ||
| git add .ci/docker/ci_commit_pins/pytorch.txt | ||
| git add runtime/core/portable_type/c10/ | ||
|
|
||
| if git diff --cached --quiet; then | ||
| echo "No changes to commit. Pin is already up to date." | ||
| exit 0 | ||
| fi | ||
|
|
||
| git commit -m "Bump PyTorch pin to nightly ${{ steps.nightly.outputs.version }}" | ||
| git push -u origin "${BRANCH}" | ||
|
|
||
| EXISTING=$(gh pr list --label "ci/pytorch-pin-bump" --state open --json number --jq '.[0].number') | ||
| if [ -n "${EXISTING}" ]; then | ||
| echo "Closing existing pin bump PR #${EXISTING} in favor of new one" | ||
| gh pr close "${EXISTING}" --comment "Superseded by newer pin bump." | ||
| fi | ||
|
|
||
| NIGHTLY="${{ steps.nightly.outputs.version }}" | ||
| read -r -d '' PR_BODY <<EOF || true | ||
| ## Summary | ||
|
|
||
| Automated weekly PyTorch pin bump. | ||
|
|
||
| - Updates \`NIGHTLY_VERSION\` in \`torch_pin.py\` to \`${NIGHTLY}\` | ||
| - Updates \`.ci/docker/ci_commit_pins/pytorch.txt\` to the corresponding nightly commit hash | ||
| - Syncs c10 headers from PyTorch into \`runtime/core/portable_type/c10/\` | ||
|
|
||
| This PR was created automatically. If CI fails, Claude will attempt to fix issues (up to 3 attempts). If CI still fails, human review will be requested. | ||
|
|
||
| cc @jakeszwe | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ideally this should be oncall but ok for now |
||
| EOF | ||
| PR_BODY=$(echo "${PR_BODY}" | sed 's/^ //') | ||
|
|
||
| gh pr create \ | ||
| --title "Bump PyTorch pin to nightly ${NIGHTLY}" \ | ||
| --body "${PR_BODY}" \ | ||
| --label "ci/pytorch-pin-bump" | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what if something which is not flaky but was broken on main before the pin bump for unrelated reasons, should we ask it to look at main? And try to see if they are failing similarly after the bump?