fix: self-heal compaction segment positions and add L0 force-select bypass#48907
Merged
Conversation
mergify
Bot
added
dco-passed
Contributor
|
[ci-v2-notice] To rerun ci-v2 checks, comment with:
If you have any questions or requests, please contact @zhikunyao. |
XuanYang-cn
added a commit
to XuanYang-cn/milvus
that referenced
this pull request
Apr 10, 2026
…lect bypass Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR milvus-io#47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: milvus-io#46435 pr: milvus-io#48907 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
XuanYang-cn
added a commit
to XuanYang-cn/milvus
that referenced
this pull request
Apr 10, 2026
…lect bypass Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR milvus-io#47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: milvus-io#46435 pr: milvus-io#48907 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
XuanYang-cn
added a commit
to XuanYang-cn/milvus
that referenced
this pull request
Apr 10, 2026
…lect bypass Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR milvus-io#47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: milvus-io#46435 pr: milvus-io#48907 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
This was referenced Apr 10, 2026
Contributor
Codecov Report✅ All modified and coverable lines are covered by tests. Please upload reports for the commit ff981c0 to get more accurate results. Additional details and impacted files@@ Coverage Diff @@
## master #48907 +/- ##
=======================================
Coverage 78.18% 78.18%
=======================================
Files 2174 2174
Lines 361086 361086
=======================================
Hits 282324 282324
Misses 70089 70089
Partials 8673 8673
🚀 New features to boost your workflow:
|
Contributor
sre-ci-robot
added
low-code-coverage
Contributor
Author
|
/ci-rerun-e2e-default |
Contributor
Author
|
/ci-rerun-gosdk |
XuanYang-cn
force-pushed
the
silent-delete-loss
branch
from
6f2d754 to
933b321
Compare
sre-ci-robot
added
low-code-coverage
Contributor
Author
|
/ci-rerun-ut-go |
1 similar comment
Contributor
Author
|
/ci-rerun-ut-go |
…ypass Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR milvus-io#47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: milvus-io#46435 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
XuanYang-cn
force-pushed
the
silent-delete-loss
branch
from
85a2d21 to
6d2a5a9
Compare
sre-ci-robot
added
the
low-code-coverage
sre-ci-robot
removed
the
low-code-coverage
Contributor
Author
|
/ci-rerun-e2e-defaul |
Contributor
Author
|
/ci-rerun-e2e-default |
XuanYang-cn
added a commit
to XuanYang-cn/milvus
that referenced
this pull request
Apr 21, 2026
…lect bypass Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR milvus-io#47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: milvus-io#46435 pr: milvus-io#48907 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
XuanYang-cn
added a commit
to XuanYang-cn/milvus
that referenced
this pull request
Apr 21, 2026
…lect bypass Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR milvus-io#47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: milvus-io#46435 pr: milvus-io#48907 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Contributor
|
This PR introduced a force mode on selecting L0 in compactions. While it fixes the leftover L0 issue, regression of performance is expected. Per offline discussion, the leftover L0 should be no use and we should remove them in the future PRs. This PR should be considered a temporary workaround. |
Contributor
|
/lgtm |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tedxu, XuanYang-cn The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
sre-ci-robot
pushed a commit
that referenced
this pull request
Apr 23, 2026
…lect bypass (#48910) Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR #47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: #46435 pr: #48907 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
sre-ci-robot
pushed a commit
that referenced
this pull request
Apr 23, 2026
…lect bypass (#48909) Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR #47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: #46435 pr: #48907 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>
This was referenced Apr 27, 2026
XuanYang-cn
added a commit
to XuanYang-cn/milvus
that referenced
this pull request
Apr 27, 2026
…lect bypass (milvus-io#48910) Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR milvus-io#47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: milvus-io#46435 pr: milvus-io#48907 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: yangxuan <xuan.yang@zilliz.com> fix: Fast finish compaction when L0Comp hit zero L1/L2 (milvus-io#47154) (milvus-io#47187) See also: milvus-io#46435, milvus-io#48907 pr: milvus-io#47154 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
XuanYang-cn
added a commit
to XuanYang-cn/milvus
that referenced
this pull request
Apr 28, 2026
…ect bypass (milvus-io#48910) fix: Fast finish compaction when L0Comp hit zero L1/L2 (milvus-io#47154) (milvus-io#47187) Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR milvus-io#47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: milvus-io#46435, milvus-io#48907 pr: milvus-io#47154 pr: milvus-io#48907 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
XuanYang-cn
added a commit
to XuanYang-cn/milvus
that referenced
this pull request
Apr 28, 2026
…ect bypass (milvus-io#48910) - fix: Fast finish compaction when L0Comp hit zero L1/L2 (milvus-io#47154) (milvus-io#47187) Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR milvus-io#47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: milvus-io#46435, milvus-io#48907 pr: milvus-io#47154 pr: milvus-io#48907 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
sre-ci-robot
pushed a commit
that referenced
this pull request
Apr 28, 2026
sre-ci-robot
pushed a commit
that referenced
this pull request
Apr 28, 2026
XuanYang-cn
added a commit
to XuanYang-cn/milvus
that referenced
this pull request
Apr 29, 2026
…ect bypass (milvus-io#48910) fix: Fast finish compaction when L0Comp hit zero L1/L2 (milvus-io#47154) (milvus-io#47187) Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR milvus-io#47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss. There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max. See also: milvus-io#46435, milvus-io#48907 pr: milvus-io#47154 pr: milvus-io#48907 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compaction inherits StartPosition/DmlPosition from source segments via getMinPosition without recalculating from actual data. The import position bug (PR #47276) wrote wrong timestamps on imported segments, and these wrong positions persist and propagate through compaction. L0 compaction then misses L1/L2 segments due to StartPosition mismatches, causing zombie L0 segments and silent delete loss.
There is also a latent bug: DmlPosition in mix/clustering compaction uses getMinPosition, but DmlPosition represents the latest entity timestamp and should use max.
See also: #46435