Skip to content

perf: O(1) readset dedup in OCC multiversion store#3245

Draft
bdchatham wants to merge 1 commit intomainfrom
refactor/opt-02-occ-readset-dedup
Draft

perf: O(1) readset dedup in OCC multiversion store#3245
bdchatham wants to merge 1 commit intomainfrom
refactor/opt-02-occ-readset-dedup

Conversation

@bdchatham
Copy link
Copy Markdown
Contributor

Summary

  • UpdateReadSet in the OCC multiversion store (VersionIndexedStore) previously used a linear scan with bytes.Equal to deduplicate values per key
  • Added a parallel readsetDedup index (map[string]map[string]struct{}) for O(1) dedup lookups
  • The ReadSet type and all consumers (validation, SetReadset, iterators) are unchanged — only the internal dedup path is faster
  • Net: +16 / -12 lines, no behavior change

Motivation

UpdateReadSet is called on every storage read in every parallel transaction. For iterator-heavy workloads (range scans over storage, e.g., DEX order books, token holder lists), the same keys are visited multiple times within a single tx, triggering the dedup check each time. The previous O(n) scan with bytes.Equal compounded across many reads. OCC is enabled by default on all Sei validators (occ-enabled = true).

Design

Rather than changing the ReadSet type (which would ripple across the codebase), we keep the existing [][]byte slice for the public interface and add a private map[string]struct{} set per key as a shadow index. Both are updated together in UpdateReadSet. The dedup check hits the map (O(1)), while validation and SetReadset continue to read from the slice.

Test plan

  • All 30+ multiversion store tests pass (go test ./sei-cosmos/store/multiversion/)
  • gofmt -s -l clean
  • go build clean

🤖 Generated with Claude Code

Add a parallel dedup index (map[string]map[string]struct{}) alongside
the existing readset slice to replace O(n) linear scan + bytes.Equal
with O(1) map lookup when checking for duplicate values in
UpdateReadSet. The ReadSet type and all consumers (validation,
SetReadset, iterators) are unchanged — only the dedup check is faster.

This matters for iterator-heavy workloads where the same keys are
scanned multiple times within a single transaction, causing repeated
UpdateReadSet calls with identical values.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedApr 14, 2026, 6:39 PM

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 14, 2026

Codecov Report

❌ Patch coverage is 77.77778% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.26%. Comparing base (9795d18) to head (9a5802d).

Files with missing lines Patch % Lines
sei-cosmos/store/multiversion/mvkv.go 77.77% 1 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #3245   +/-   ##
=======================================
  Coverage   59.26%   59.26%           
=======================================
  Files        2070     2070           
  Lines      169788   169791    +3     
=======================================
+ Hits       100631   100634    +3     
  Misses      60358    60358           
  Partials     8799     8799           
Flag Coverage Δ
sei-chain-pr 91.33% <77.77%> (?)
sei-db 70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-cosmos/store/multiversion/mvkv.go 91.07% <77.77%> (+0.16%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant