Skip to content

chore(jobsdb): add consumer columns#7082

Draft
atzoum wants to merge 1 commit into
masterfrom
feat.mcJobsDBDDL
Draft

chore(jobsdb): add consumer columns#7082
atzoum wants to merge 1 commit into
masterfrom
feat.mcJobsDBDDL

Conversation

@atzoum

@atzoum atzoum commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds two new columns to all jobsdb tables (jobs and job_status) as the foundational schema step for multi-consumer jobsdb support:

  • consumers on jobs tables — tracks which consumers a job is intended for
  • consumer on job_status tables — identifies which consumer a status row belongs to

Both columns are added universally to all handles (single- and multi-consumer alike), with safe defaults that preserve existing single-consumer semantics. All existing read, write, and compaction paths are updated to carry the new columns through.

Linear Ticket

Resolves PIPE-3115

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

This is part 2 of 3 in a stack made with GitButler:

@codecov

codecov Bot commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.90%. Comparing base (dfef1f1) to head (f0c14c3).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7082      +/-   ##
==========================================
+ Coverage   79.49%   79.90%   +0.40%     
==========================================
  Files         583      583              
  Lines       65202    65207       +5     
==========================================
+ Hits        51835    52103     +268     
+ Misses      10295    10014     -281     
- Partials     3072     3090      +18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@atzoum atzoum marked this pull request as draft June 12, 2026 07:54
atzoum added a commit that referenced this pull request Jun 12, 2026
## Description

Without a minimum age guard, newly created datasets, whether written by
normal ingestion or created as a compaction destination, can be picked
up for compaction almost immediately, before workers have had a chance
to process all their jobs. This causes premature I/O churn: jobs get
moved again with little benefit.

This PR introduces `compactionMinDSAge` (default: `2m`):
partially-processed datasets are skipped for compaction until they reach
this minimum age, giving workers a processing window before a dataset
becomes eligible. The age is read from a creation timestamp stored as a
PostgreSQL table comment on each jobs table.

**Fully-processed datasets** and datasets without a recorded creation
time **are unaffected**.

**Config:** `JobsDB.compactionMinDSAge` (reloadable)

## Linear Ticket

resolves PIPE-3114

## Security

- [x] The code changed/added as part of this pull request won't create
any security issues with how the software is being used.

<!-- GitButler Footer Boundary Top -->
---
This is **part 1 of 3 in a stack** made with GitButler:
- <kbd>&nbsp;3&nbsp;</kbd> #7083 
- <kbd>&nbsp;2&nbsp;</kbd> #7082 
- <kbd>&nbsp;1&nbsp;</kbd> #7077 👈 
<!-- GitButler Footer Boundary Bottom -->
Base automatically changed from feat.compactionMinDSAge to master June 12, 2026 13:58
@atzoum atzoum force-pushed the feat.mcJobsDBDDL branch from a54abf0 to f0c14c3 Compare June 12, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant