Skip to content

uploaders/pulp: parallel chunks, direct artifacts#220

Open
andrewlukoshko wants to merge 1 commit intoAlmaLinux:masterfrom
andrewlukoshko:feat-optimize-pulp-upload
Open

uploaders/pulp: parallel chunks, direct artifacts#220
andrewlukoshko wants to merge 1 commit intoAlmaLinux:masterfrom
andrewlukoshko:feat-optimize-pulp-upload

Conversation

@andrewlukoshko
Copy link
Copy Markdown
Member

  • Replace sequential filesplit-based disk splitting with parallel ThreadPoolExecutor chunk uploads using byte-range reads via temp files
  • Add direct artifact creation via ArtifactsApi.create() for files smaller than chunk_size, bypassing the 3-step upload+commit flow
  • Eliminate redundant SHA256 hashing: file is now hashed once in upload_single_file and the hash is passed through _send_file and _commit_upload (was hashed 3 times per upload)
  • Add per-file upload timing logs at INFO level for all three upload paths (direct artifact, single chunk, multi-chunk)
  • Remove filesplit dependency from requirements.txt
  • Add 5 new tests: direct artifact creation, parallel chunk upload, chunk error propagation, and timing log verification for both paths

- Replace sequential filesplit-based disk splitting with parallel
  ThreadPoolExecutor chunk uploads using byte-range reads via temp files
- Add direct artifact creation via ArtifactsApi.create() for files
  smaller than chunk_size, bypassing the 3-step upload+commit flow
- Eliminate redundant SHA256 hashing: file is now hashed once in
  upload_single_file and the hash is passed through _send_file and
  _commit_upload (was hashed 3 times per upload)
- Add per-file upload timing logs at INFO level for all three upload
  paths (direct artifact, single chunk, multi-chunk)
- Remove filesplit dependency from requirements.txt
- Add 5 new tests: direct artifact creation, parallel chunk upload,
  chunk error propagation, and timing log verification for both paths
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant