chore(jobsdb): add consumer columns#7082
Draft
atzoum wants to merge 1 commit into
Draft
Conversation
1 task
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7082 +/- ##
==========================================
+ Coverage 79.49% 79.90% +0.40%
==========================================
Files 583 583
Lines 65202 65207 +5
==========================================
+ Hits 51835 52103 +268
+ Misses 10295 10014 -281
- Partials 3072 3090 +18 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1 task
atzoum
added a commit
that referenced
this pull request
Jun 12, 2026
## Description Without a minimum age guard, newly created datasets, whether written by normal ingestion or created as a compaction destination, can be picked up for compaction almost immediately, before workers have had a chance to process all their jobs. This causes premature I/O churn: jobs get moved again with little benefit. This PR introduces `compactionMinDSAge` (default: `2m`): partially-processed datasets are skipped for compaction until they reach this minimum age, giving workers a processing window before a dataset becomes eligible. The age is read from a creation timestamp stored as a PostgreSQL table comment on each jobs table. **Fully-processed datasets** and datasets without a recorded creation time **are unaffected**. **Config:** `JobsDB.compactionMinDSAge` (reloadable) ## Linear Ticket resolves PIPE-3114 ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used. <!-- GitButler Footer Boundary Top --> --- This is **part 1 of 3 in a stack** made with GitButler: - <kbd> 3 </kbd> #7083 - <kbd> 2 </kbd> #7082 - <kbd> 1 </kbd> #7077 👈 <!-- GitButler Footer Boundary Bottom -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two new columns to all jobsdb tables (jobs and job_status) as the foundational schema step for multi-consumer jobsdb support:
consumerson jobs tables — tracks which consumers a job is intended forconsumeron job_status tables — identifies which consumer a status row belongs toBoth columns are added universally to all handles (single- and multi-consumer alike), with safe defaults that preserve existing single-consumer semantics. All existing read, write, and compaction paths are updated to carry the new columns through.
Linear Ticket
Resolves PIPE-3115
Security
This is part 2 of 3 in a stack made with GitButler: