feat(jobsdb): multiconsumer datasets and flip#7083
Draft
atzoum wants to merge 4 commits into
Draft
Conversation
This was referenced Jun 12, 2026
30fd3cf to
4e7cb64
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## feat.mcJobsDBDDL #7083 +/- ##
===================================================
Coverage ? 79.50%
===================================================
Files ? 584
Lines ? 65444
Branches ? 0
===================================================
Hits ? 52033
Misses ? 10315
Partials ? 3096 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
atzoum
added a commit
that referenced
this pull request
Jun 12, 2026
## Description Without a minimum age guard, newly created datasets, whether written by normal ingestion or created as a compaction destination, can be picked up for compaction almost immediately, before workers have had a chance to process all their jobs. This causes premature I/O churn: jobs get moved again with little benefit. This PR introduces `compactionMinDSAge` (default: `2m`): partially-processed datasets are skipped for compaction until they reach this minimum age, giving workers a processing window before a dataset becomes eligible. The age is read from a creation timestamp stored as a PostgreSQL table comment on each jobs table. **Fully-processed datasets** and datasets without a recorded creation time **are unaffected**. **Config:** `JobsDB.compactionMinDSAge` (reloadable) ## Linear Ticket resolves PIPE-3114 ## Security - [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used. <!-- GitButler Footer Boundary Top --> --- This is **part 1 of 3 in a stack** made with GitButler: - <kbd> 3 </kbd> #7083 - <kbd> 2 </kbd> #7082 - <kbd> 1 </kbd> #7077 👈 <!-- GitButler Footer Boundary Bottom -->
91463e1 to
fccf41a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds the foundational schema and infrastructure for multi-consumer support in jobsdb. Today each jobsdb handle has a single implicit consumer: a job's latest status determines its state. This is a first step towards allowing jobs to be consumed independently by multiple consumers, each with its own status timeline per job.
The feature is opt-in per handle via
WithMultiConsumer(). Single-consumer handles are completely unaffected — no schema changes, no query overhead.An existing single-consumer jobsdb can be converted to multi-consumer at runtime. On first startup with
WithMultiConsumer(), jobsdb upgrades any existing datasets to the multi-consumer schema. The upgrade is idempotent and crash-safe.Linear Ticket
resolves PIPE-3058
Security
This is part 3 of 3 in a stack made with GitButler: