Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 117 additions & 0 deletions .github/workflows/pin-bump-ci-handler.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
name: Pin Bump CI Handler

on:
workflow_run:
workflows: ["trunk"]
types: [completed]

jobs:
handle-ci-result:
if: github.repository_owner == 'pytorch'
runs-on: ubuntu-latest
environment: update-commit-hash
permissions:
pull-requests: write
issues: write
steps:
- uses: actions/github-script@v7
with:
github-token: ${{ secrets.UPDATEBOT_TOKEN }}
script: |
const { owner, repo } = context.repo;
const workflowRun = context.payload.workflow_run;
const conclusion = workflowRun.conclusion;
const runUrl = workflowRun.html_url;

const prs = workflowRun.pull_requests;
if (!prs || prs.length === 0) {
console.log('No PRs associated with this workflow run. Skipping.');
return;
}

const prNumber = prs[0].number;
const pr = await github.rest.pulls.get({ owner, repo, pull_number: prNumber });

const isPinBump = pr.data.labels.some(l => l.name === 'ci/pytorch-pin-bump');
if (!isPinBump) {
console.log(`PR #${prNumber} is not a pin bump PR. Skipping.`);
return;
}

const allowedAuthors = new Set(['pytorchbot', 'pytorchupdatebot', 'facebook-github-bot']);
if (!allowedAuthors.has(pr.data.user.login)) {
console.log(`PR #${prNumber} was created by ${pr.data.user.login}, not an allowed automation account. Skipping.`);
return;
}

console.log(`Pin bump PR #${prNumber}, trunk concluded: ${conclusion}`);

const comments = await github.rest.issues.listComments({
owner, repo, issue_number: prNumber, per_page: 100
});
const fixAttempts = comments.data.filter(
c => c.body && c.body.startsWith('@claude [ci-fix-attempt')
).length;

if (conclusion === 'success') {
const note = fixAttempts > 0
? `Claude fixed CI failures in ${fixAttempts} attempt(s).`
: 'CI passed on the first try.';

await github.rest.issues.createComment({
owner, repo, issue_number: prNumber,
body: `## CI Passed\n\nAll trunk CI checks have passed on this pin bump PR. ${note}\n\n**This PR is ready for human review and merge.**\n\ncc @jakeszwe`
});
return;
}

if (conclusion !== 'failure') {
console.log(`Trunk concluded with "${conclusion}" (not failure). Skipping.`);
return;
}

if (fixAttempts >= 3) {
await github.rest.issues.createComment({
owner, repo, issue_number: prNumber,
body: [
'## Automated Fix Attempts Exhausted',
'',
`CI is still failing after ${fixAttempts} automated fix attempt(s).`,
`Failed trunk run: ${runUrl}`,
'',
'This pin bump likely requires human intervention. Common causes:',
'- BC-breaking API changes in PyTorch that need design discussion',
'- New dependencies or build system changes',
'- Test infrastructure issues unrelated to the pin bump',
'',
'cc @jakeszwe'
].join('\n')
});
return;
}

const attemptNum = fixAttempts + 1;
await github.rest.issues.createComment({
owner, repo, issue_number: prNumber,
body: [
`@claude [ci-fix-attempt ${attemptNum}/3]`,
'',
`The \`trunk\` CI workflow has failed on this automated PyTorch pin bump PR.`,
`Failed run: ${runUrl}`,
'',
'Please:',
'1. Read the Dr. CI comment on this PR for a summary of which jobs failed and whether they are flaky. Ignore failures marked as FLAKY.',
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if something which is not flaky but was broken on main before the pin bump for unrelated reasons, should we ask it to look at main? And try to see if they are failing similarly after the bump?

'2. Use your CI tools to download the failure logs for the non-flaky failing jobs',
'3. Identify the root cause of the failure',
'4. If this is a build or test failure caused by PyTorch API changes, fix the ExecuTorch code to be compatible with the new PyTorch version',
'5. If this is a c10 header sync issue, the headers have already been synced by the pin bump script — the issue is likely in ExecuTorch code that uses those headers',
'6. Run `lintrunner -a` on any files you change',
'7. Push your fix as a new commit to this PR branch',
'',
'Important constraints:',
'- Do NOT modify torch_pin.py or .ci/docker/ci_commit_pins/pytorch.txt — the pin itself is correct',
'- Do NOT modify files under runtime/core/portable_type/c10/ unless the sync introduced a new API that ExecuTorch code needs to adapt to',
'- Focus on fixing ExecuTorch code to be compatible with the new PyTorch APIs',
'- If this is a major BC-breaking change that requires architectural discussion, say so clearly and stop — do not attempt a fix'
].join('\n')
});
93 changes: 93 additions & 0 deletions .github/workflows/weekly-pytorch-pin-bump.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
name: Weekly PyTorch Pin Bump

on:
schedule:
- cron: '0 9 * * 1' # Monday 9:00 UTC
workflow_dispatch:

jobs:
create-pin-bump-pr:
if: github.repository_owner == 'pytorch'
runs-on: ubuntu-latest
environment: update-commit-hash
permissions:
contents: write
pull-requests: write
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.UPDATEBOT_TOKEN }}

- uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Determine nightly version
id: nightly
run: |
NIGHTLY_DATE=$(date -u -d 'yesterday' '+%Y%m%d')
NIGHTLY_VERSION="dev${NIGHTLY_DATE}"
echo "version=${NIGHTLY_VERSION}" >> "$GITHUB_OUTPUT"

- name: Read current TORCH_VERSION
id: torch
run: |
TORCH_VERSION=$(python -c "exec(open('torch_pin.py').read()); print(TORCH_VERSION)")
echo "version=${TORCH_VERSION}" >> "$GITHUB_OUTPUT"

- name: Update torch_pin.py with new NIGHTLY_VERSION
run: |
printf 'TORCH_VERSION = "%s"\nNIGHTLY_VERSION = "%s"\n' \
"${{ steps.torch.outputs.version }}" \
"${{ steps.nightly.outputs.version }}" > torch_pin.py

- name: Run pin bump script
run: python .github/scripts/update_pytorch_pin.py

- name: Create branch and PR
env:
GH_TOKEN: ${{ secrets.UPDATEBOT_TOKEN }}
run: |
BRANCH="automated/pytorch-pin-bump-${{ steps.nightly.outputs.version }}"

git config user.name "pytorchbot"
git config user.email "pytorchbot@users.noreply.github.com"
git checkout -b "${BRANCH}"
git add torch_pin.py
git add .ci/docker/ci_commit_pins/pytorch.txt
git add runtime/core/portable_type/c10/

if git diff --cached --quiet; then
echo "No changes to commit. Pin is already up to date."
exit 0
fi

git commit -m "Bump PyTorch pin to nightly ${{ steps.nightly.outputs.version }}"
git push -u origin "${BRANCH}"

EXISTING=$(gh pr list --label "ci/pytorch-pin-bump" --state open --json number --jq '.[0].number')
if [ -n "${EXISTING}" ]; then
echo "Closing existing pin bump PR #${EXISTING} in favor of new one"
gh pr close "${EXISTING}" --comment "Superseded by newer pin bump."
fi

NIGHTLY="${{ steps.nightly.outputs.version }}"
read -r -d '' PR_BODY <<EOF || true
## Summary

Automated weekly PyTorch pin bump.

- Updates \`NIGHTLY_VERSION\` in \`torch_pin.py\` to \`${NIGHTLY}\`
- Updates \`.ci/docker/ci_commit_pins/pytorch.txt\` to the corresponding nightly commit hash
- Syncs c10 headers from PyTorch into \`runtime/core/portable_type/c10/\`

This PR was created automatically. If CI fails, Claude will attempt to fix issues (up to 3 attempts). If CI still fails, human review will be requested.

cc @jakeszwe
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this should be oncall but ok for now

EOF
PR_BODY=$(echo "${PR_BODY}" | sed 's/^ //')

gh pr create \
--title "Bump PyTorch pin to nightly ${NIGHTLY}" \
--body "${PR_BODY}" \
--label "ci/pytorch-pin-bump"
Loading