Skip to content

OCPBUGS-81568: fix(storage): wait for CSI driver daemonset rollout to prevent pathological event failures#30974

Open
RomanBednar wants to merge 1 commit intoopenshift:mainfrom
RomanBednar:OCPBUGS-81568-pathological-events-vsphere
Open

OCPBUGS-81568: fix(storage): wait for CSI driver daemonset rollout to prevent pathological event failures#30974
RomanBednar wants to merge 1 commit intoopenshift:mainfrom
RomanBednar:OCPBUGS-81568-pathological-events-vsphere

Conversation

@RomanBednar
Copy link
Copy Markdown
Contributor

@RomanBednar RomanBednar commented Apr 7, 2026

No description provided.

@RomanBednar @claude
fix(storage): wait for CSI driver daemonset rollout to prevent pathol…
e94c5f9 
…ogical event failures

The vSphere CSI Driver Configuration tests modify clusterCSIDriver which
triggers a daemonset rollout across all nodes. Previously, the tests only
waited for the storage operator to report Progressing=False, but this only
means the operator updated the daemonset spec. The daemonset controller
then asynchronously recreates pods on all nodes over 5-12 minutes.

These trailing SuccessfulCreate events were emitted after the test interval
ended, causing client-go to aggregate them with lastTimestamp outside the
test window. The pathological event matcher's ±10min padding couldn't
reliably cover this nondeterministic gap, leading to CI failures like:
"event happened 27 times, something is wrong: namespace/openshift-cluster-csi-drivers
daemonset/vmware-vsphere-csi-driver-node - reason/SuccessfulCreate"

The fix adds waitForCSIDriverDaemonSetRollout() which polls the daemonset
status until NumberReady == DesiredNumberScheduled && NumberUnavailable == 0.
This is called in both the test body and AfterEach after Progressing=False,
ensuring all pod creation events fall within the test's time interval.

This follows the same pattern used by networking tests
(test/extended/networking/util.go:isDaemonSetRunningOnGeneration) and
DRA tests (test/extended/node/dra/nvidia/prerequisites_installer.go:waitForDaemonSet).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented Apr 7, 2026

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Apr 7, 2026
@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented Apr 7, 2026

@RomanBednar: This pull request references Jira Issue OCPBUGS-81568, which is invalid:

  • expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Apr 7, 2026 •
edited
Loading

Caution

Review failed

An error occurred during the review process. Please try again later.

Walkthrough

A helper function waitForCSIDriverDaemonSetRollout() was added to poll the vmware-vsphere-csi-driver-node DaemonSet until rollout completion. Related constants were introduced, and the test flow was updated to invoke this helper in two locations after storage operator status transitions.

Changes

Cohort / File(s) Summary
CSI Driver DaemonSet Rollout Helper
test/extended/storage/driver_configuration.go		 Added waitForCSIDriverDaemonSetRollout() helper function to poll DaemonSet readiness using new constants (csiDriverDSName, csiDriverDSNs, dsRolloutTimeout). Integrated into test control flow at two points: within AfterEach and main It paths when operatorShouldProgress is true. Updated comments to reflect DaemonSet rollout requirement.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 7, 2026

@RomanBednar: trigger 0 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

@openshift-ci openshift-ci bot requested review from bertinatto and dobsonj April 7, 2026 13:38
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 7, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: RomanBednar
Once this PR has been reviewed and has the lgtm label, please assign dobsonj for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@RomanBednar
Copy link
Copy Markdown
Contributor Author

RomanBednar commented Apr 7, 2026

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Apr 7, 2026

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@RomanBednar
Copy link
Copy Markdown
Contributor Author

RomanBednar commented Apr 7, 2026

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-ovn-serial

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 7, 2026

@RomanBednar: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-ovn-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/46d25800-3287-11f1-92a5-166f459ebe91-0

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 7, 2026

@RomanBednar: trigger 0 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented Apr 7, 2026

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 7, 2026

@RomanBednar: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-serial-1of2 e94c5f9 link true /test e2e-aws-ovn-serial-1of2
ci/prow/e2e-metal-ipi-ovn-ipv6 e94c5f9 link true /test e2e-metal-ipi-ovn-ipv6
ci/prow/e2e-aws-ovn-fips e94c5f9 link true /test e2e-aws-ovn-fips

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bertinatto bertinatto Awaiting requested review from bertinatto

@dobsonj dobsonj Awaiting requested review from dobsonj

Assignees

No one assigned

Labels

jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@RomanBednar @openshift-ci-robot