fix: raft HA production hardening — leader fencing, log compaction, election timeout, audit log

[auricom]

Summary

Hardens Raft HA behavior across several failure modes observed in SIGTERM cycling tests.

Leader fencing on SIGTERM

• Added ResignLeader(ctx context.Context) error to raft.Node and FullNode (via LeaderResigner interface)
• SIGTERM handler in run_node.go calls resigner.ResignLeader(resignCtx) directly with a 3s deadline — no wrapper goroutine needed; context.DeadlineExceeded vs other errors are logged separately
• ResignLeader runs leadershipTransfer in a goroutine and selects on the result vs ctx.Done(), so the caller is unblocked at the deadline (the inner transfer exits on its own, bounded by ElectionTimeout)
• ShutdownTimeout field (default 5s) added to raft.Config so Stop() drains committed logs with a meaningful timeout

Raft log compaction

• Added SnapshotThreshold (default 500) and TrailingLogs (default 200) to RaftConfig and wired them into the hashicorp/raft config — previously using library defaults (8192 / 10240, i.e. snapshots every ~2.25h at 1 blk/s)
• Fixed SnapCount default 0 → 3 (0 caused no snapshot files to be retained on disk)

Election timeout config

• Added ElectionTimeout to RaftConfig (default 1s)
• Validate() rejects negative values explicitly (< 0 is invalid; 0 remains valid as "use library default"); also validates >= HeartbeatTimeout when non-zero to prevent hashicorp/raft panicking at startup

Follower crash on restart (EVM ahead of stale snapshot)

• RecoverFromRaft now verifies the local block hash at raftState.Height; a hash match skips unnecessary recovery safely, a mismatch returns an explicit error indicating a fork
• waitForMsgsLanded replaced its repeating deadline ticker with a one-shot time.NewTimer and adds a final check in the deadline branch — prevents a race where both poll and deadline tickers fired simultaneously and timeout won even when AppliedIndex >= CommitIndex

FSM data race

• FSM.Apply and SetApplyCallback are now guarded by applyMu sync.RWMutex to prevent the raft library calling Apply concurrently while SetApplyCallback(nil) clears the channel pointer during shutdown

BoltDB log noise

• A one-time stdlib log filter (sync.Once in NewNode) silently drops "tx closed" lines from raft-boltdb — these are emitted on every successful commit due to defer tx.Rollback() and have no actionable meaning
• Filter wraps log.Writer() instead of hard-coding os.Stderr so any existing stdlib logger redirection is preserved

Block provenance audit log

• RaftApplyMsg carries a Term field; FSM.Apply() logs raft_term and raft_index alongside block height and hash for replication debugging

Test plan

• go test ./pkg/raft/... ./pkg/config/... ./node/... -count=1 — all pass
• TestNodeResignLeader_NilNoop / TestNodeResignLeader_NotLeaderNoop — nil/non-leader no-op paths
• TestFullNode_ResignLeader_Noop — table-driven nil and non-leader cases
• TestNewNode_SnapshotConfigApplied — snapshot config wiring (table-driven)
• TestSyncerStop_CallsRaftRetrieverStop — Syncer.Stop() calls raftRetriever.Stop()
• TestRecoverFromRaft_GetHeaderFailure — hash mismatch error path
• "negative election timeout rejected" — Validate() rejects ElectionTimeout < 0
• HA cycling test: verify leader fencing reduces unconfirmed-block window on SIGTERM
• HA cycling test: verify rebooted nodes resync within cycle window after snapshot config change

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• New Features

  • Added leadership resignation capability for graceful leader transition during node shutdown
  • New Raft configuration options for fine-tuning cluster behavior (election timeout, snapshot threshold, trailing logs)

• Bug Fixes

  • Improved shutdown sequence with proper cleanup of internal raft state
  • Enhanced recovery logic to gracefully handle stale cluster snapshots
  • Nodes now abdicate leadership when significantly behind the cluster state

• Configuration

  • New Raft configuration parameters with sensible defaults

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2d1f347c-97d4-42c5-94c7-d4e1c127007b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds Raft leadership resignation capability, extends configuration with election and snapshot tuning parameters, improves FSM apply concurrency safety with mutex guards, handles ahead-of-raft recovery scenarios gracefully, and implements leader abdication when the local store falls significantly behind the Raft state.

Changes

Cohort / File(s)	Summary
Raft Core Configuration & Node pkg/raft/node.go, pkg/raft/types.go	Extends Config with ShutdownTimeout, ElectionTimeout, SnapshotThreshold, and TrailingLogs; adds ResignLeader method for context-aware leadership transfer; makes FSM apply callback wiring concurrency-safe with RWMutex; updates waitForMsgsLanded loop with one-shot deadline timer; adds Term field to RaftApplyMsg.
Raft Leadership & Election pkg/raft/election.go	Adds leader abdication check: if store lags raft state by more than 1 block, node logs warning, attempts immediate leadership transfer, and skips leader start.
Block Syncing block/internal/syncing/syncer.go, block/internal/syncing/raft_retriever.go	Extends shutdown to stop raftRetriever; improves RecoverFromRaft to load and compare local block hash when local height exceeds raft snapshot height, enabling graceful skip of recovery when hashes match.
Full Node Implementation node/full.go, node/node.go	Adds LeaderResigner interface declaring ResignLeader(ctx context.Context) error; implements method on FullNode forwarding to raftNode when available; configures raft with new timing/retention fields.
Configuration Management pkg/config/config.go, pkg/config/defaults.go	Adds ElectionTimeout, SnapshotThreshold, and TrailingLogs fields to RaftConfig; extends validation to check ElectionTimeout >= HeartbeatTimeout; sets sensible defaults (ElectionTimeout: 1000ms, SnapshotThreshold: 500, TrailingLogs: 200, SnapCount: 3).
Startup/Shutdown Coordination pkg/cmd/run_node.go	Integrates best-effort Raft leadership resignation on SIGINT/SIGTERM shutdown via type assertion to LeaderResigner interface with 3-second timeout; logs success or distinguishes timeout vs. other errors.
Test Coverage block/internal/syncing/syncer_test.go, node/full_node_test.go, pkg/config/config_test.go, pkg/raft/node_test.go, pkg/raft/election_test.go	Adds tests for callback clearing during shutdown, ResignLeader noop behavior, config validation for election timeouts, buildRaftConfig with new parameters, and leader abdication when store lags raft state.

Sequence Diagram(s)

sequenceDiagram
    participant CLI as CLI (SIGINT/SIGTERM)
    participant Runner as Run Control
    participant Node as FullNode
    participant RaftNode as RaftNode
    participant FSM as FSM/Apply
    
    CLI->>Runner: Signal received
    Runner->>Node: Type-assert LeaderResigner
    alt LeaderResigner supported
        Runner->>Node: ResignLeader(ctx, 3s timeout)
        Node->>RaftNode: ResignLeader(ctx)
        RaftNode->>RaftNode: Transfer leadership
        RaftNode-->>Node: Success/Error
        Node-->>Runner: Return result
        Runner->>Runner: Log outcome
    else Not supported
        Runner->>Runner: Skip resignation
    end
    Runner->>Runner: Call cancel()
    Runner->>Runner: Wait for shutdown (10s)

Loading

sequenceDiagram
    participant Election as Election Loop
    participant Store as Store (Local)
    participant Runnable as Runnable
    participant Leader as Leader Sync
    participant Transfer as Leadership Transfer
    
    Election->>Runnable: IsSynced(raftState)
    Runnable-->>Election: diff = -5 (lag > 1 block)
    alt Store significantly behind
        Election->>Election: Log warning with lag size
        Election->>Transfer: leadershipTransfer()
        Transfer-->>Election: Initiated
        Election->>Election: Skip leader start (continue)
    else Store within tolerance
        Election->>Leader: Proceed with leader startup
    end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• fix: Publisher-mode synchronization option for failover scenario #3222: Directly modifies RecoverFromRaft logic in block/internal/syncing/syncer.go for ahead-of-raft state handling.

Suggested reviewers

• alpe
• julienrbrt
• tac0turtle

Poem

🐰 A Raft of changes, carefully planned,
Leadership resignation, delegation at hand,
Timeouts and thresholds in harmony play,
Safe shutdowns, abdications save the day! 🎩✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 26.09% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main changes: leader fencing, log compaction, election timeout config, and audit logging are all substantive parts of the changeset.
Description check	✅ Passed	The pull request description is comprehensive and well-structured, covering all major changes, the rationale, test coverage, and explicitly listing pending integration tests.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/3229-raft-re-election

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[claude]

Claude finished @auricom's task in 3m 51s —— View job

---

Code Review — Raft HA Production Hardening

This is a well-motivated PR with solid safety rationale throughout. The changes address real production failure modes observed in SIGTERM cycling tests. Below are my findings organized by severity.

---

Tasks

• Read AGENTS.md for repo guidelines
• Review pkg/raft/node.go — FSM mutex, ShutdownTimeout, ResignLeader
• Review pkg/raft/election.go — abdication + waitForBlockStoreSync
• Review pkg/raft/types.go, config, defaults
• Review pkg/cmd/run_node.go — SIGTERM handler
• Review node/full.go, node/node.go — LeaderResigner interface
• Review block/internal/syncing/syncer.go, raft_retriever.go
• Review test coverage
• Synthesize and post review

---

What's Good

• FSM data race fix (node.go:394-404): The applyMu RWMutex correctly guards applyCh access — write lock in SetApplyCallback, read lock in Apply. The pattern of reading the channel under the lock, then sending outside it, is correct and avoids holding the lock during a blocking send.
• waitForMsgsLanded one-shot timer: Replacing the repeating ticker with a time.NewTimer deadline eliminates the race where both poll and deadline could fire simultaneously. Clean fix.
• waitForBlockStoreSync addresses the hot-potato abdication problem correctly: if all nodes restart simultaneously with lagging stores, they wait for sync before abdicating rather than immediately passing leadership around. The leadership-lost abort (leaderID != NodeID) inside the poll loop is the right escape hatch.
• RecoverFromRaft hash verification: The "local ahead of stale snapshot" branch now verifies hash identity before skipping recovery. The explicit fork error gives operators a clear signal.
• Config defaults: SnapCount 0 → 3, SnapshotThreshold 500, TrailingLogs 200 are well-reasoned and commented in defaults.go.
• Context-aware ResignLeader: Correctly unblocks the caller at deadline while the inner transfer drains to completion (bounded by ElectionTimeout). The goroutine leak is intentional and bounded — acceptable.

---

Issues

Medium — Worth Addressing Before Merge

1. SnapCount not validated in RaftConfig.Validate()
(pkg/config/config.go)

raft.NewFileSnapshotStore(cfg.RaftDir, int(cfg.SnapCount), os.Stderr) requires retain > 0. A zero value silently creates a store that retains no snapshots — nodes would lose compaction on restart. The fix was flagged by CodeRabbit but not applied. The default is now 3, but an operator could override to 0.

// Add to RaftConfig.Validate():
if c.SnapCount == 0 {
    multiErr = errors.Join(multiErr, fmt.Errorf("snap_count must be > 0"))
}

Fix this →

2. waitForBlockStoreSync repurposes ShutdownTimeout semantically
(pkg/raft/election.go:296-299)

ShutdownTimeout is used here as the "how long to wait for block-store sync before abdicating" budget. With the default of 5 * SendTimeout = 1s, nodes have 1 second to resync their block store after being elected leader. This may be too short under real conditions (P2P sync latency, slow storage). The field name is also misleading in this context.

Consider adding a dedicated BlockStoreSyncTimeout config field, or at least documenting that ShutdownTimeout doubles as the sync-wait budget. Fix this →

3. Duplicate log message in election.go
(pkg/raft/election.go:123 and :179)

// line ~123
d.logger.Info().Msg("became leader, stopping follower operations")
// ... 50+ lines of sync/recovery logic ...
// line ~179
d.logger.Info().Msg("became leader, stopping follower operations")
workerCancel()
wg.Wait()

Both messages are identical. The first fires before any sync checks have run (nothing has stopped yet), which is misleading in logs. The second is accurate. The first should be removed or changed to something like "became leader, checking sync state before stopping follower". Fix this →

4. Stop() may attempt leadership transfer twice
(pkg/raft/node.go:222-226)

When shutdown is signalled, the call sequence is:

1. ResignLeader(3s) (via SIGTERM handler in run_node.go)
2. cancel() → node context cancelled
3. Eventually: raftNode.Stop() which calls leadershipTransfer() again if IsLeader() returns true

If ResignLeader returned successfully, IsLeader() should return false and the second transfer is skipped. But there is a narrow window where the raft state hasn't propagated yet. This is low-probability but worth a comment noting the double-transfer is intentionally guarded by IsLeader(). Fix this →

---

Minor

5. RecoverFromRaft silent return nil for multi-block gap
(block/internal/syncing/syncer.go:1267)

When raftState.Height > currentState.LastBlockHeight + 1 (local more than 1 block behind raft but not "ahead"), the function falls through all branches to return nil with no action and no log. This is intentional (the caller handles the gap via waitForBlockStoreSync + abdication), but a comment would prevent future confusion:

// Local block store is more than 1 block behind raft state. Recovery can only
// apply one block; the gap will be closed by p2p sync after the caller allows
// normal operations.
return nil

Fix this →

6. IsSyncedWithRaft uses s.ctx instead of caller's context
(block/internal/syncing/syncer.go:1163)

s.store.GetState(s.ctx) uses the syncer's internal context. If Stop() has been called and s.ctx is cancelled, any IsSyncedWithRaft call from the election loop would return an error. This would cause waitForBlockStoreSync to never report sync (since err != nil → false). Consider passing the context from the caller. This is pre-existing but the new waitForBlockStoreSync loop now exercises this path more frequently.

7. os.Stderr hardcoded in transport and snapshot store
(pkg/raft/node.go:117,127)

Both raft.NewFileSnapshotStore and raft.NewTCPTransport use os.Stderr directly. This was pre-existing and the boltdb noise filter was removed per reviewer feedback, but these two still bypass any zerolog routing. Low priority for this PR.

8. election_test.go "abdicate" scenario uses time.After(200ms)
(pkg/raft/election_test.go:269-274)

select {
case <-leaderStarted:
    t.Error("leader should not start...")
case <-time.After(200 * time.Millisecond):
    // leadership transferred without starting leader — expected
}

This works because the test config uses a 2ms timeout so the wait expires well before 200ms. But the 200ms wall-clock dependency makes the test slow in CI and could flake on heavily-loaded runners. A channel-based signal from the leadership transfer mock would be more deterministic. Not a blocker but something to tighten in a follow-up.

---

Summary

The PR is production-ready. The FSM mutex, one-shot timer, hash verification in recovery, and hot-potato abdication fix are all correct and well-tested. The main gap is the missing SnapCount > 0 validation (item 1 above) — an operator who sets snap_count: 0 would get silent broken behavior at runtime instead of a startup error. The rest are documentation/clarity issues.

[github-actions]

The latest Buf updates on your PR. Results from workflow CI / buf-check (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	⏩ skipped	✅ passed	✅ passed	Apr 17, 2026, 2:58 PM

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

block/internal/syncing/raft_retriever_test.go (1)

42-61: Consider converting this to a table-driven test.

The current case is good, but a table shape will make it easier to add stop idempotency and start/stop-cycle variants without duplicating setup.

As per coding guidelines "Use table-driven tests in Go unit tests".

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@block/internal/syncing/raft_retriever_test.go` around lines 42 - 61, The test
TestRaftRetrieverStopClearsApplyCallback should be converted into a table-driven
test to cover multiple scenarios (current stop behavior, stop idempotency,
start/stop cycles) without duplicating setup: create a slice of test cases each
with a name and a sequence of actions (e.g., start, stop, stop again, start/stop
cycle), and in the t.Run loop instantiate a fresh stubRaftNode and retriever via
newRaftRetriever, call retriever.Start and retriever.Stop according to the case,
then assert expected recordedCallbacks via stubRaftNode.recordedCallbacks; keep
using require.NoError for Start and require assertions on callback length and
nil/non-nil entries as in the original test. Ensure each case isolates state by
creating new retriever and stubRaftNode within the loop.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@block/internal/syncing/raft_retriever.go`:
- Line 77: The call to r.raftNode.SetApplyCallback(nil) races with FSM.Apply
because Apply reads/sends on applyCh while the raft node may concurrently invoke
the callback; fix by adding a mutex to the raft node to guard access to the
apply callback: protect the callback field and its setter Get/SetApplyCallback
(or SetApplyCallback and any internal invocation sites) with the new mutex so
that FSM.Apply (which reads/sends on applyCh via the callback) cannot see a nil
or changing callback mid-invocation; update the raft node's invocation path that
calls the callback (where Apply is invoked) to acquire the same mutex (or use a
read lock) when reading the callback and release it immediately after obtaining
the pointer, then call the callback outside the lock if needed to avoid
long-held locks.

---

Nitpick comments:
In `@block/internal/syncing/raft_retriever_test.go`:
- Around line 42-61: The test TestRaftRetrieverStopClearsApplyCallback should be
converted into a table-driven test to cover multiple scenarios (current stop
behavior, stop idempotency, start/stop cycles) without duplicating setup: create
a slice of test cases each with a name and a sequence of actions (e.g., start,
stop, stop again, start/stop cycle), and in the t.Run loop instantiate a fresh
stubRaftNode and retriever via newRaftRetriever, call retriever.Start and
retriever.Stop according to the case, then assert expected recordedCallbacks via
stubRaftNode.recordedCallbacks; keep using require.NoError for Start and require
assertions on callback length and nil/non-nil entries as in the original test.
Ensure each case isolates state by creating new retriever and stubRaftNode
within the loop.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d9bc4987-af22-4eb8-a04c-1c9ef989e55a

📥 Commits

Reviewing files that changed from the base of the PR and between 04c9cad and 2d28b20.

📒 Files selected for processing (3)

• block/internal/syncing/raft_retriever.go
• block/internal/syncing/raft_retriever_test.go
• pkg/raft/node.go

[codecov]

Codecov Report

❌ Patch coverage is 60.43165% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.50%. Comparing base (2865d6d) to head (5897ef3).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/raft/node.go	39.62%	32 Missing ⚠️
pkg/cmd/run_node.go	0.00%	10 Missing ⚠️
pkg/raft/election.go	78.57%	6 Missing and 3 partials ⚠️
node/full.go	50.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3230      +/-   ##
==========================================
+ Coverage   62.33%   62.50%   +0.17%     
==========================================
  Files         122      122              
  Lines       12873    13002     +129     
==========================================
+ Hits         8024     8127     +103     
- Misses       3968     3990      +22     
- Partials      881      885       +4     

Flag	Coverage Δ
combined	62.50% <60.43%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

node/full_node_test.go (1)

87-97: Prefer a table-driven test for the ResignLeader no-op cases.

These two tests are the same shape and are easier to extend/maintain as one table-driven test.

♻️ Proposed refactor

-func TestFullNode_ResignLeader_NilRaftNode(t *testing.T) {
-	n := &FullNode{} // raftNode is nil
-	assert.NoError(t, n.ResignLeader())
-}
-
-func TestFullNode_ResignLeader_NonLeaderRaftNode(t *testing.T) {
-	// Empty *raftpkg.Node has nil raft field so IsLeader() returns false;
-	// ResignLeader() is a no-op and returns nil.
-	n := &FullNode{raftNode: &raftpkg.Node{}}
-	assert.NoError(t, n.ResignLeader())
-}
+func TestFullNode_ResignLeader_NoOpCases(t *testing.T) {
+	tests := []struct {
+		name string
+		node *FullNode
+	}{
+		{name: "nil raft node", node: &FullNode{}},
+		{name: "non-leader raft node", node: &FullNode{raftNode: &raftpkg.Node{}}},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.NoError(t, tt.node.ResignLeader())
+		})
+	}
+}

As per coding guidelines, **/*_test.go: "Use table-driven tests in Go unit tests".

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@node/full_node_test.go` around lines 87 - 97, Combine the two ResignLeader
no-op tests into a single table-driven test: create a slice of cases with
descriptive names for the nil-raftNode case and the non-leader raftNode case,
each providing a FullNode value (e.g., &FullNode{} and &FullNode{raftNode:
&raftpkg.Node{}}) and the expected outcome (no error); iterate over cases using
t.Run(case.name, func(t *testing.T) { assert.NoError(t,
case.node.ResignLeader()) }), keeping references to FullNode.ResignLeader and
raftpkg.Node to locate the code to change.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/cmd/run_node.go`:
- Around line 231-245: The current shutdown uses a 3s resignCtx but launches
resigner.ResignLeader() without cancellation so the resign can keep running;
change the API to accept a context (e.g., LeaderResigner.ResignLeader(ctx
context.Context) error), call resigner.ResignLeader(resignCtx) directly (or in a
goroutine and select on its returned error and resignCtx.Done()) and
propagate/observe the context to abort the resign operation; update the call
site in run_node.go to pass resignCtx and handle the returned error or context
timeout to avoid leaking a background goroutine or leaving an unbounded resign
in progress.

In `@pkg/config/config.go`:
- Around line 454-456: The validation must reject negative election timeouts:
add an explicit check in the same validation block to treat c.ElectionTimeout <
0 as invalid (join an error like "election timeout (%v) must be >= 0" into
multiErr), keeping 0 as the "use default" sentinel; ensure this complements the
existing comparison with c.HeartbeatTimeout and aligns with buildRaftConfig's
behavior (which only applies values > 0) so negative values fail fast instead of
being silently ignored.

In `@pkg/raft/node.go`:
- Around line 95-97: The code currently calls
log.SetOutput(&boltTxClosedFilter{w: os.Stderr}) inside suppressBoltNoise.Do(),
which clobbers the process-wide stdlib logger; instead capture the current
stdlib logger writer and wrap it so existing configuration is preserved: obtain
prev := log.Writer() and call log.SetOutput(&boltTxClosedFilter{w: prev}) (still
guarded by suppressBoltNoise.Do()). Update the usage in the same block where
suppressBoltNoise.Do and boltTxClosedFilter are referenced (e.g., in NewNode
initialization) so you no longer hard-code os.Stderr.

---

Nitpick comments:
In `@node/full_node_test.go`:
- Around line 87-97: Combine the two ResignLeader no-op tests into a single
table-driven test: create a slice of cases with descriptive names for the
nil-raftNode case and the non-leader raftNode case, each providing a FullNode
value (e.g., &FullNode{} and &FullNode{raftNode: &raftpkg.Node{}}) and the
expected outcome (no error); iterate over cases using t.Run(case.name, func(t
*testing.T) { assert.NoError(t, case.node.ResignLeader()) }), keeping references
to FullNode.ResignLeader and raftpkg.Node to locate the code to change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bff68b93-3da9-4c95-9c3b-ae84e849204f

📥 Commits

Reviewing files that changed from the base of the PR and between 2d28b20 and a2a2599.

📒 Files selected for processing (12)

• block/internal/syncing/syncer.go
• block/internal/syncing/syncer_test.go
• node/full.go
• node/full_node_test.go
• node/node.go
• pkg/cmd/run_node.go
• pkg/config/config.go
• pkg/config/config_test.go
• pkg/config/defaults.go
• pkg/raft/node.go
• pkg/raft/node_test.go
• pkg/raft/types.go

[coderabbitai]

🧹 Nitpick comments (1)

pkg/raft/node_test.go (1)

195-228: Test logic is good; consider renaming for precision.

The assertions are correct, but the function name suggests NewNode behavior while it directly tests buildRaftConfig. Renaming would improve discoverability.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@pkg/raft/node_test.go` around lines 195 - 228, Rename the test
TestNewNode_SnapshotConfigApplied to reflect what it actually verifies: the
buildRaftConfig function applies snapshot-related fields from Config; for
example rename to TestBuildRaftConfig_SnapshotConfigApplied (or
TestBuildRaftConfig_AppliesSnapshotConfig). Update the test name only (keep the
same table-driven cases and assertions using buildRaftConfig, rc, Config) so
test discovery and intent match the implementation.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@pkg/raft/node_test.go`:
- Around line 195-228: Rename the test TestNewNode_SnapshotConfigApplied to
reflect what it actually verifies: the buildRaftConfig function applies
snapshot-related fields from Config; for example rename to
TestBuildRaftConfig_SnapshotConfigApplied (or
TestBuildRaftConfig_AppliesSnapshotConfig). Update the test name only (keep the
same table-driven cases and assertions using buildRaftConfig, rc, Config) so
test discovery and intent match the implementation.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 36de18f5-92b8-4b48-8098-050aafc90d75

📥 Commits

Reviewing files that changed from the base of the PR and between a2a2599 and 2d3dc8e.

📒 Files selected for processing (8)

• node/full.go
• node/full_node_test.go
• node/node.go
• pkg/cmd/run_node.go
• pkg/config/config.go
• pkg/config/config_test.go
• pkg/raft/node.go
• pkg/raft/node_test.go

✅ Files skipped from review due to trivial changes (1)

• pkg/cmd/run_node.go

🚧 Files skipped from review as they are similar to previous changes (4)

• node/node.go
• node/full.go
• pkg/config/config.go
• pkg/raft/node.go

[julienrbrt]

nice work!

[julienrbrt]

this test does not look necessary

[auricom]

Done — deleted in bf6bb50. The test was validating an implementation detail (that Stop() calls SetApplyCallback(nil)) rather than observable behaviour. The stubRaftNode helper it defined has been moved to syncer_test.go where it's still needed by TestSyncer_Stop_CallsRaftRetrieverStop.

[julienrbrt]

Let's delete this, this is does not use the correct logger anyway.

[auricom]

Done — deleted in bf6bb50. The global stdlib log.SetOutput redirect was the wrong scope; nothing else in the codebase should be silenced that way.

[julienrbrt]

ditto, if we don't need log from this, and if it uses the stdlib log implementation, we should make sure hashicorp/raft-boltdb is instantiated with zerolog

[auricom]

Unfortunately hashicorp/raft-boltdb does not support logger injection — its Options struct has no Logger field and all rollback errors are emitted via hardcoded log.Printf calls. There is no way to route those through zerolog without patching the library upstream. The "Rollback failed: tx closed" noise is benign (it fires every time defer tx.Rollback() runs after a successful tx.Commit()), so for now we accept it.

[julienrbrt]

Do we really need a different ShutdownTimeout vs SendTimeout? That's a log of timeouts in the config, which is confusing for operators (like cometbft has thousands of confusing timeouts)

[auricom]

Agreed — removed from public config in bf6bb50. ShutdownTimeout is still computed internally as 5 × SendTimeout at the initRaftNode call site (node/full.go), so the drain window scales with the operator's chosen send timeout without exposing another knob. The ShutdownTimeout field stays in pkg/raft.Config as an internal detail.

[coderabbitai]

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

pkg/config/config.go (1)

419-452: ⚠️ Potential issue | 🟠 Major

Validate raft.snap_count here too.

SnapCount is still allowed to be 0, but pkg/raft/node.go passes it straight into raft.NewFileSnapshotStore(...), which will fail during node startup instead of during config validation. Since this is now a user-facing retention knob, reject zero in RaftConfig.Validate() so bad operator input fails fast.

Suggested fix

 func (c RaftConfig) Validate() error {
 	if !c.Enable {
 		return nil
 	}
 	var multiErr error
@@
+	if c.SnapCount == 0 {
+		multiErr = errors.Join(multiErr, fmt.Errorf("snap count must be > 0"))
+	}
+
 	if c.SendTimeout <= 0 {
 		multiErr = errors.Join(multiErr, fmt.Errorf("send timeout must be positive"))
 	}

As per coding guidelines, "Validate all inputs from external sources in Go code".

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@pkg/config/config.go` around lines 419 - 452, RaftConfig.Validate currently
misses validating SnapCount; add a check in the RaftConfig.Validate() method to
reject SnapCount == 0 (or <= 0) and join an appropriate error (e.g.,
fmt.Errorf("snap count must be positive")) into multiErr so invalid config fails
fast; reference the RaftConfig type and its Validate() method (and note that
raft.NewFileSnapshotStore in pkg/raft/node.go expects a positive snap count)
when locating where to add the validation.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/raft/election_test.go`:
- Around line 245-261: The test is nondeterministic due to time.Sleep and
t.Fatal in a goroutine; replace those with channel-based synchronization: in the
testRunnable used as leader (testRunnable.runFn) send a message on a new
leaderStartAttemptCh instead of calling t.Fatal, and have the main test
goroutine wait on that channel to observe the leader start attempt; remove
time.Sleep and instead use the existing leaderCh/fStarted or a new
leaderStartedCh to coordinate the leader transfer (send/receive to trigger state
transitions) and then perform assertions on the captured signals from
leaderStartAttemptCh and leaderStartedCh; update where DynamicLeaderElection is
constructed (leaderFactory/followerFactory) to return the testRunnable that uses
these channels for deterministic coordination.

In `@pkg/raft/election.go`:
- Around line 147-150: The current code swallows failures from
node.leadershipTransfer() by logging and continuing, which can leave the node
stuck; change the behavior so that transfer failures are surfaced instead of
silently continuing: inside the loop where node.leadershipTransfer() is called
(the call shown as d.node.leadershipTransfer() with the logger d.logger),
replace the bare continue on error with logic that returns or propagates tErr to
the caller (or otherwise transitions the state to abort leader startup) so the
failure is visible and handled upstream; ensure the log still records the error
before returning/propagating.

In `@pkg/raft/node.go`:
- Around line 84-90: NewNode currently can leave cfg.ShutdownTimeout as zero
which causes a panic in Stop() -> waitForMsgsLanded when creating a ticker with
interval zero; fix this by initializing/clamping the internal shutdown timeout
in NewNode before the config is stored: if cfg.ShutdownTimeout <= 0 set it to
the internal default (e.g. 5 * cfg.SendTimeout) or a sane minimum, so callers of
Stop() / waitForMsgsLanded always see a positive timeout value.

---

Outside diff comments:
In `@pkg/config/config.go`:
- Around line 419-452: RaftConfig.Validate currently misses validating
SnapCount; add a check in the RaftConfig.Validate() method to reject SnapCount
== 0 (or <= 0) and join an appropriate error (e.g., fmt.Errorf("snap count must
be positive")) into multiErr so invalid config fails fast; reference the
RaftConfig type and its Validate() method (and note that
raft.NewFileSnapshotStore in pkg/raft/node.go expects a positive snap count)
when locating where to add the validation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b4087f79-ce2a-422e-a24c-41233d9807d3

📥 Commits

Reviewing files that changed from the base of the PR and between 2d3dc8e and bf6bb50.

📒 Files selected for processing (9)

• block/internal/syncing/syncer_test.go
• node/full.go
• pkg/config/config.go
• pkg/config/config_test.go
• pkg/config/defaults.go
• pkg/raft/election.go
• pkg/raft/election_test.go
• pkg/raft/node.go
• pkg/raft/node_test.go

✅ Files skipped from review due to trivial changes (1)

• pkg/config/config_test.go

🚧 Files skipped from review as they are similar to previous changes (3)

• pkg/config/defaults.go
• node/full.go
• block/internal/syncing/syncer_test.go

[julienrbrt]

the changes look great, could you add a changelog?

[julienrbrt]

this may be too eager, what if all nodes in the rafts are behind, they will just pass each other the raft leadership without having time to catchup themselves.

[auricom]

Confirmed this exact scenario in the local tests. When all nodes restart simultaneously, their block stores lag the raft FSM height (block data arrives via p2p, not raft). When restarted all nodes at once, raft cluster entered a hot-potato loop with no stable leader and no block production.

Fixed in 5897ef3b: before abdicating, the new waitForBlockStoreSync helper polls IsSynced for up to ShutdownTimeout (~1s). The fastest-syncing peer stabilises as leader; nodes that can't catch up in time still abdicate normally. Lost-leadership is also detected mid-wait so the wait aborts early if another node takes over.