Skip to content

feat(jobsdb): multiconsumer datasets and flip#7083

Draft
atzoum wants to merge 4 commits into
feat.mcJobsDBDDLfrom
feat.mcJobsDBDDL2
Draft

feat(jobsdb): multiconsumer datasets and flip#7083
atzoum wants to merge 4 commits into
feat.mcJobsDBDDLfrom
feat.mcJobsDBDDL2

Conversation

@atzoum

@atzoum atzoum commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Description

Adds the foundational schema and infrastructure for multi-consumer support in jobsdb. Today each jobsdb handle has a single implicit consumer: a job's latest status determines its state. This is a first step towards allowing jobs to be consumed independently by multiple consumers, each with its own status timeline per job.

The feature is opt-in per handle via WithMultiConsumer(). Single-consumer handles are completely unaffected — no schema changes, no query overhead.

An existing single-consumer jobsdb can be converted to multi-consumer at runtime. On first startup with WithMultiConsumer(), jobsdb upgrades any existing datasets to the multi-consumer schema. The upgrade is idempotent and crash-safe.

Linear Ticket

resolves PIPE-3058

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

This is part 3 of 3 in a stack made with GitButler:

@atzoum atzoum marked this pull request as draft June 12, 2026 08:34
@atzoum atzoum force-pushed the feat.mcJobsDBDDL2 branch from 30fd3cf to 4e7cb64 Compare June 12, 2026 09:01
@codecov

codecov Bot commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.71473% with 36 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (feat.mcJobsDBDDL@f0c14c3). Learn more about missing BASE report.

Files with missing lines Patch % Lines
jobsdb/jobsdb_dataset_ddl.go 20.83% 11 Missing and 8 partials ⚠️
jobsdb/jobsdb_multiconsumer.go 81.81% 6 Missing and 6 partials ⚠️
jobsdb/jobsdb_compaction.go 98.71% 1 Missing and 1 partial ⚠️
jobsdb/jobsdb_store.go 83.33% 1 Missing and 1 partial ⚠️
jobsdb/jobsdb_get.go 95.23% 1 Missing ⚠️
Additional details and impacted files
@@                 Coverage Diff                 @@
##             feat.mcJobsDBDDL    #7083   +/-   ##
===================================================
  Coverage                    ?   79.50%           
===================================================
  Files                       ?      584           
  Lines                       ?    65444           
  Branches                    ?        0           
===================================================
  Hits                        ?    52033           
  Misses                      ?    10315           
  Partials                    ?     3096           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@atzoum atzoum changed the title feat(josbdb): multiconsumer datasets and flip feat(jobsdb): multiconsumer datasets and flip Jun 12, 2026
atzoum added a commit that referenced this pull request Jun 12, 2026
## Description

Without a minimum age guard, newly created datasets, whether written by
normal ingestion or created as a compaction destination, can be picked
up for compaction almost immediately, before workers have had a chance
to process all their jobs. This causes premature I/O churn: jobs get
moved again with little benefit.

This PR introduces `compactionMinDSAge` (default: `2m`):
partially-processed datasets are skipped for compaction until they reach
this minimum age, giving workers a processing window before a dataset
becomes eligible. The age is read from a creation timestamp stored as a
PostgreSQL table comment on each jobs table.

**Fully-processed datasets** and datasets without a recorded creation
time **are unaffected**.

**Config:** `JobsDB.compactionMinDSAge` (reloadable)

## Linear Ticket

resolves PIPE-3114

## Security

- [x] The code changed/added as part of this pull request won't create
any security issues with how the software is being used.

<!-- GitButler Footer Boundary Top -->
---
This is **part 1 of 3 in a stack** made with GitButler:
- <kbd>&nbsp;3&nbsp;</kbd> #7083 
- <kbd>&nbsp;2&nbsp;</kbd> #7082 
- <kbd>&nbsp;1&nbsp;</kbd> #7077 👈 
<!-- GitButler Footer Boundary Bottom -->
@atzoum atzoum force-pushed the feat.mcJobsDBDDL branch from a54abf0 to f0c14c3 Compare June 12, 2026 13:59
@atzoum atzoum force-pushed the feat.mcJobsDBDDL2 branch from 91463e1 to fccf41a Compare June 12, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant