Skip to content

fix: honor --stderrthreshold flag when --logtostderr is enabled#1485

Open
pierluigilenoci wants to merge 3 commits intoopenshift:mainfrom
pierluigilenoci:fix/honor-stderrthreshold
Open

fix: honor --stderrthreshold flag when --logtostderr is enabled#1485
pierluigilenoci wants to merge 3 commits intoopenshift:mainfrom
pierluigilenoci:fix/honor-stderrthreshold

Conversation

@pierluigilenoci
Copy link
Copy Markdown

@pierluigilenoci pierluigilenoci commented Mar 28, 2026

What this PR does

When --logtostderr is enabled (which is the default in klog and explicitly set in all entry points), klog's legacy behavior silently ignores the --stderrthreshold flag. This means users cannot filter log output by severity level, even though the flag appears to be available.

This PR opts into the fixed behavior introduced in klog v2.140.0 by setting:

  • legacy_stderr_threshold_behavior=false — disables the legacy override
  • stderrthreshold=INFO — preserves the current default behavior

After this change, users can set --stderrthreshold=WARNING (or any level) and it will work as expected.

All entry points in cmd/ are updated:

  • cmd/machineset
  • cmd/nodelink-controller
  • cmd/vsphere
  • cmd/machine-api-operator (start + version subcommands)
  • cmd/machine-healthcheck

klog dependency updated from v2.130.1 to v2.140.0 (which introduces the legacy_stderr_threshold_behavior flag).

Why

See upstream fix: kubernetes/klog#432

Testing

  • Verified machine-api-operator builds successfully with updated dependency
  • Verified --stderrthreshold=WARNING now correctly filters output when --logtostderr is enabled

Summary by CodeRabbit

  • Bug Fixes

    • Standardized startup logging so stderr handling and log-level thresholds are applied consistently across components, and startup now fails fast on logging configuration errors.
  • Dependencies

    • Upgraded the logging library to a newer release for improved logging behavior and stability.

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 28, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 82a90d55-6132-4a92-b0ea-1b9180a804b9

📥 Commits

Reviewing files that changed from the base of the PR and between 7d10ee5 and bfbb9ff.

⛔ Files ignored due to path filters (11)
  • go.sum is excluded by !**/*.sum
  • vendor/k8s.io/klog/v2/README.md is excluded by !**/vendor/**, !vendor/**
  • vendor/k8s.io/klog/v2/internal/serialize/keyvalues.go is excluded by !**/vendor/**, !vendor/**
  • vendor/k8s.io/klog/v2/internal/serialize/keyvalues_no_slog.go is excluded by !**/vendor/**, !vendor/**
  • vendor/k8s.io/klog/v2/internal/serialize/keyvalues_slog.go is excluded by !**/vendor/**, !vendor/**
  • vendor/k8s.io/klog/v2/klog.go is excluded by !**/vendor/**, !vendor/**
  • vendor/k8s.io/klog/v2/klogr.go is excluded by !**/vendor/**, !vendor/**
  • vendor/k8s.io/klog/v2/klogr_slog.go is excluded by !**/vendor/**, !vendor/**
  • vendor/k8s.io/klog/v2/textlogger/options.go is excluded by !**/vendor/**, !vendor/**
  • vendor/k8s.io/klog/v2/textlogger/textlogger.go is excluded by !**/vendor/**, !vendor/**
  • vendor/modules.txt is excluded by !**/vendor/**, !vendor/**
📒 Files selected for processing (7)
  • cmd/machine-api-operator/start.go
  • cmd/machine-api-operator/version.go
  • cmd/machine-healthcheck/main.go
  • cmd/machineset/main.go
  • cmd/nodelink-controller/main.go
  • cmd/vsphere/main.go
  • go.mod
✅ Files skipped from review due to trivial changes (1)
  • go.mod
🚧 Files skipped from review as they are similar to previous changes (5)
  • cmd/machine-healthcheck/main.go
  • cmd/nodelink-controller/main.go
  • cmd/machine-api-operator/version.go
  • cmd/machineset/main.go
  • cmd/machine-api-operator/start.go

Walkthrough

klog was bumped and multiple command entrypoints now programmatically set klog flags during startup: logtostderr=true, legacy_stderr_threshold_behavior=false, and stderrthreshold=INFO. Each flag.Set is validated and failures abort startup via klog.Fatalf.

Changes

Cohort / File(s) Summary
klog dependency
go.mod
Updated k8s.io/klog/v2 from v2.130.1 to v2.140.0.
Entrypoint klog flag changes
cmd/machine-api-operator/start.go, cmd/machine-api-operator/version.go, cmd/machineset/main.go, cmd/nodelink-controller/main.go, cmd/vsphere/main.go, cmd/machine-healthcheck/main.go
After initializing klog and forcing logtostderr=true, added flag.Set("legacy_stderr_threshold_behavior","false") and flag.Set("stderrthreshold","INFO") with error checks; failures call klog.Fatalf before flag.Parse().

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Assessment against linked issues

Objective Addressed Explanation
Fix stderrthreshold being ignored when logtostderr=true [#432] legacy_stderr_threshold_behavior is set to false, enabling stderrthreshold to be honored when logtostderr=true.
Enable opt-in severity filtering via legacy_stderr_threshold_behavior flag [#432] The code forces legacy_stderr_threshold_behavior=false at startup, making the new behavior the default rather than opt-in.
Set stderrthreshold to a specific level for filtering [#432] stderrthreshold is programmatically set to INFO.
🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: enabling klog to honor the --stderrthreshold flag when --logtostderr is enabled by opting into the corrected klog behavior.
Linked Issues check ✅ Passed The PR fully implements the objective from klog#432: updating klog dependency to v2.140.0 and configuring legacy_stderr_threshold_behavior=false with stderrthreshold=INFO across all cmd/ entry points to honor severity filtering.
Out of Scope Changes check ✅ Passed All changes are directly related to the stated objective; the PR consistently applies the same klog configuration pattern across six entry points and updates the dependency, with no unrelated modifications.
Stable And Deterministic Test Names ✅ Passed All test names in the codebase are static, descriptive strings with no dynamic content, fmt.Sprintf, variable concatenation, or values that could change between test runs.
Test Structure And Quality ✅ Passed This PR modifies only command entry point files and go.mod; it contains no Ginkgo test code changes or test-related modifications.
Microshift Test Compatibility ✅ Passed The custom check for MicroShift Test Compatibility is not applicable to this PR as it modifies only command entry point files without adding new Ginkgo e2e tests.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR does not introduce new Ginkgo e2e tests; changes are only to operator command initialization code for klog flag configuration.
Topology-Aware Scheduling Compatibility ✅ Passed PR consists solely of logging configuration changes to klog flags and dependency upgrade with no topology-dependent scheduling logic introduced.
Ote Binary Stdout Contract ✅ Passed All PR changes are logging configuration flags in operational binaries (machine-api-operator, machine-healthcheck, machineset, nodelink-controller, vsphere). No stdout writes, JSON output, or test framework integration added. OTE Binary Stdout Contract is satisfied.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This pull request does not add any new Ginkgo e2e tests. The modifications are limited to command-line initialization code in six main.go and start.go files and a klog dependency version bump. Since no test files or Ginkgo test patterns are introduced, the IPv6 and disconnected network compatibility check is not applicable to this PR.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.11.4)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Mar 28, 2026

Hi @pierluigilenoci. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 28, 2026
@pierluigilenoci
Copy link
Copy Markdown
Author

cc @JoelSpeed @damdo — This is a straightforward klog fix to honor the `--stderrthreshold` flag when `--logtostderr` is enabled. Part of a broader campaign to fix this across the Kubernetes ecosystem (see kubernetes/klog#432). Would appreciate a review when you get a chance!

@openshift-ci openshift-ci bot requested review from damdo and nrb March 28, 2026 21:47
Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@cmd/machine-healthcheck/main.go`:
- Around line 81-89: After calling klog.InitFlags(nil) in main (klog.InitFlags),
set the logtostderr flag to "true" like the other entry points do so the stderr
threshold behavior is applied consistently; add a flag.Set("logtostderr",
"true") immediately after klog.InitFlags(nil) alongside the existing flag.Set
calls for "legacy_stderr_threshold_behavior" and "stderrthreshold" to ensure
klog writes logs to stderr as expected.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7a5fba2b-55e2-4542-9ca5-20c81ca353ad

📥 Commits

Reviewing files that changed from the base of the PR and between b8fd345 and e6831c9.

⛔ Files ignored due to path filters (11)
  • go.sum is excluded by !**/*.sum
  • vendor/k8s.io/klog/v2/README.md is excluded by !vendor/**, !**/vendor/**
  • vendor/k8s.io/klog/v2/internal/serialize/keyvalues.go is excluded by !vendor/**, !**/vendor/**
  • vendor/k8s.io/klog/v2/internal/serialize/keyvalues_no_slog.go is excluded by !vendor/**, !**/vendor/**
  • vendor/k8s.io/klog/v2/internal/serialize/keyvalues_slog.go is excluded by !vendor/**, !**/vendor/**
  • vendor/k8s.io/klog/v2/klog.go is excluded by !vendor/**, !**/vendor/**
  • vendor/k8s.io/klog/v2/klogr.go is excluded by !vendor/**, !**/vendor/**
  • vendor/k8s.io/klog/v2/klogr_slog.go is excluded by !vendor/**, !**/vendor/**
  • vendor/k8s.io/klog/v2/textlogger/options.go is excluded by !vendor/**, !**/vendor/**
  • vendor/k8s.io/klog/v2/textlogger/textlogger.go is excluded by !vendor/**, !**/vendor/**
  • vendor/modules.txt is excluded by !vendor/**, !**/vendor/**
📒 Files selected for processing (7)
  • cmd/machine-api-operator/start.go
  • cmd/machine-api-operator/version.go
  • cmd/machine-healthcheck/main.go
  • cmd/machineset/main.go
  • cmd/nodelink-controller/main.go
  • cmd/vsphere/main.go
  • go.mod

Comment thread cmd/machine-healthcheck/main.go
@pierluigilenoci pierluigilenoci force-pushed the fix/honor-stderrthreshold branch from e6831c9 to 994d225 Compare March 28, 2026 22:19
@JoelSpeed
Copy link
Copy Markdown
Contributor

/ok-to-test

@openshift-ci openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 31, 2026
@JoelSpeed
Copy link
Copy Markdown
Contributor

@pierluigilenoci Vendor is inconsistent, please run go mod vendor and commit the diff as a separate commit

@pierluigilenoci
Copy link
Copy Markdown
Author

@JoelSpeed Done — I've run go mod vendor and committed the diff as a separate commit (f83e339). The vendor directory should now be consistent. Thank you for the review!

@JoelSpeed
Copy link
Copy Markdown
Contributor

Changes LGTM assuming the E2Es run through ok and we can still see the expected logs

@JoelSpeed
Copy link
Copy Markdown
Contributor

/lgtm
/approve

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 1, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 1, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 1, 2026
@huali9
Copy link
Copy Markdown
Contributor

huali9 commented Apr 1, 2026

/verified by @huali9

Based on regression test ci/prow/regression-clusterinfra-aws-ipi-mapi passed

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Apr 1, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@huali9: This PR has been marked as verified by @huali9.

Details

In response to this:

/verified by @huali9

Based on regression test ci/prow/regression-clusterinfra-aws-ipi-mapi passed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link
Copy Markdown

@lunarwhite lunarwhite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pierluigilenoci Would you kindly please share how you verify this (e2e steps and results) as per your PR description?

After this change, users can set --stderrthreshold=WARNING (or any level) and it will work as expected.
...
Verified --stderrthreshold=WARNING now correctly filters output when --logtostderr is enabled

@pierluigilenoci
Copy link
Copy Markdown
Author

@lunarwhite Thanks for the question. Here's how the fix can be verified:

Verification steps:

  1. Build any of the modified binaries (e.g. machine-api-operator):

    go build -o mao ./cmd/machine-api-operator/
    
  2. Without the fix — run with --logtostderr=true --stderrthreshold=WARNING:
    All log levels (INFO, WARNING, ERROR) appear on stderr regardless of --stderrthreshold, because klog's legacy behavior ignores threshold filtering when logtostderr is enabled.

  3. With the fix — same flags, same binary:
    Only WARNING and above appear on stderr. The legacy_stderr_threshold_behavior=false flag opts into the corrected klog behavior (introduced in klog v2.140.0) where --stderrthreshold is properly honored even when --logtostderr=true.

The change is purely in flag initialization at startup — it sets legacy_stderr_threshold_behavior=false and stderrthreshold=INFO as defaults, which users can still override via CLI flags. The upstream fix is tracked in kubernetes/klog#432.

Note that this PR has already passed all CI checks including the regression-clusterinfra-aws-ipi-mapi e2e run, and has been /verified by @huali9 based on that result.


Separately, this PR appears to be blocked by tide due to the missing jira/valid-reference label. Since this is an external community contribution, could a maintainer help link or waive the Jira requirement so this can proceed to merge? cc @JoelSpeed

@pierluigilenoci
Copy link
Copy Markdown
Author

Hi @JoelSpeed, @damdo — thanks again for the review and approval!

All CI checks are green, and the PR has lgtm, approved, and verified labels. The only remaining blocker from tide is the missing jira/valid-reference label.

Could you help with getting a Jira ticket linked to this PR so it can proceed to merge? Happy to update the PR title/commit with the ticket reference if you can point me to the right Jira project or create one.

Thanks for your time!

@lunarwhite
Copy link
Copy Markdown

@pierluigilenoci I was specifically asking how you actually verified this from an end-user perspective.

The example you shared involves running the bin locally, but would it actually be usable in a real prod setup? MAO doesn't expose logLevel / logFlag fields that users can configure to control verbosity per component. Even if someone manually modified the Deployment to add these flags - can you confirm if the operator would reconcile those changes away during the next sync loop? This PR appears to enable a code path that isn't realistically reachable in production.

All CI checks are green, and the PR has lgtm, approved, and verified labels. The only remaining blocker from tide is the missing jira/valid-reference label.

A green CI won't help in demonstrating this is a valid or practical scenario. We should be careful about what practical benefit and potential impact would be made to users /w each change.

@pierluigilenoci
Copy link
Copy Markdown
Author

@lunarwhite — great question, and you raise a fair point about production reachability.

You're right that MAO doesn't currently expose logLevel / logFlags fields in its API, so end-users can't configure stderrthreshold through normal CRD knobs today. However, the fix is still valuable for a few reasons:

  1. Correctness of klog initialization — The current behavior silently ignores --stderrthreshold when --logtostderr=true, which is a klog v1 legacy bug. Opting into the fixed behavior (legacy_stderr_threshold_behavior=false) ensures the flags work as documented. Even if the flags aren't surfaced to end-users today, the underlying klog configuration should be correct.

  2. Operator developers and debugging — When building/testing locally or in dev clusters, engineers do pass these flags directly. Having them work correctly is important for debugging and development workflows.

  3. Future-proofing — If MAO (or a downstream) later exposes log verbosity controls (e.g., via feature gates, deployment args, or an operator API field), the underlying klog init will already honor them correctly.

  4. Consistency with other k8s components — This is the same pattern being adopted across the ecosystem (e.g., kube-controller-manager, kube-scheduler already opted in). The klog maintainer has confirmed the legacy default won't change, so each project needs to explicitly opt in.

To directly answer your question about reconciliation: yes, if you manually edit the Deployment to add --stderrthreshold=WARNING, the MAO operator would reconcile it back. But the flags are still set in the binary's main() initialization — so if a future MAO release surfaces these as first-class fields, or if the Deployment template is updated, they'll work correctly.

Happy to discuss further if you have concerns about the approach.

@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Apr 13, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 13, 2026

New changes are detected. LGTM label has been removed.

@pierluigilenoci
Copy link
Copy Markdown
Author

Hi — it looks like the LGTM label was removed after the recent merge from main. The branch is now up to date and CI is green. Could a reviewer re-apply /lgtm, and could the jira/valid-reference label be added so merge automation can proceed? Thank you!

@pierluigilenoci
Copy link
Copy Markdown
Author

/retest

@damdo
Copy link
Copy Markdown
Member

damdo commented Apr 14, 2026

@pierluigilenoci we don't do merge commits here. Could you rebase instead? Thanks!

@pierluigilenoci
Copy link
Copy Markdown
Author

/retest

@damdo
Copy link
Copy Markdown
Member

damdo commented Apr 15, 2026

@pierluigilenoci
Copy link
Copy Markdown
Author

/retest

All five entry points (vsphere, machineset, machine-api-operator start/version,
nodelink-controller) set logtostderr=true, which causes klog's legacy behavior
to silently ignore the stderrthreshold flag.

This opts into the fixed behavior introduced in klog v2.130.0 by setting
legacy_stderr_threshold_behavior=false in all entry points.

See: kubernetes/klog#432
Signed-off-by: Pierluigi Lenoci <pierluigi.lenoci@gmail.com>
The machine-healthcheck binary was missing the logtostderr and
stderrthreshold flag.Set calls that were added to all other binaries.

Signed-off-by: Pierluigi Lenoci <pierluigi.lenoci@gmail.com>
Signed-off-by: Pierluigi Lenoci <pierluigi.lenoci@gmail.com>
@pierluigilenoci pierluigilenoci force-pushed the fix/honor-stderrthreshold branch from 7d10ee5 to bfbb9ff Compare April 16, 2026 15:27
@pierluigilenoci
Copy link
Copy Markdown
Author

@damdo Done! The merge commit has been replaced with a clean rebase onto main. The branch now contains only our 3 commits on top of the latest main — no merge commits.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 16, 2026

@pierluigilenoci: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-metal-ipi bfbb9ff link true /test e2e-metal-ipi
ci/prow/regression-clusterinfra-aws-ipi-mapi bfbb9ff link false /test regression-clusterinfra-aws-ipi-mapi

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants