fix: persist member join status with conflict retry#10357
Conversation
|
Auto Cherry-pick Instructions CLA Recheck Instructions |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #10357 +/- ##
==========================================
+ Coverage 53.15% 53.25% +0.09%
==========================================
Files 533 533
Lines 63457 63470 +13
==========================================
+ Hits 33733 33798 +65
+ Misses 26277 26227 -50
+ Partials 3447 3445 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Runtime validation update for the release-1.0 backport path: Scope:
Result:
Evidence verified:
Boundary:
|
| joinErrors = append(joinErrors, fmt.Errorf("pod %s: %w", pod.Name, err)) | ||
| } else { | ||
| key := types.NamespacedName{Namespace: r.protoITS.Namespace, Name: r.protoITS.Name} | ||
| if err := component.UpdateReplicaStatusWithRetry(r.transCtx.Context, r.cli, key, pod.Name, func(status *component.ReplicaStatus) error { |
There was a problem hiding this comment.
[P1] This writes the live InstanceSet from inside the transformer after memberJoin succeeds, which is the wrong layer for this failure. A conflict while persisting memberJoined is a normal reconcile failure: the controller should retry the whole reconcile and rely on the memberJoin action idempotency, not introduce a second write path outside the DAG/plan execution model. Keep replicas-status persistence in the normal DAG/update flow instead of special-casing this annotation write.
|
[P1] The production failure is caused by accepting scale-in while the previous scale-out member lifecycle is still in progress, not by the resourceVersion conflict itself. If any replica still has pending DataLoaded or MemberJoined state, scale-in can delete it and skip or race the engine memberLeave path regardless of this retry. Add an operation/state gate so scale-in waits for scale-out member lifecycle completion, or define an explicit safe cancellation path, before deleting those replicas. |
3 participants
Problem
Oracle 3-to-2 horizontal scaling hit a controller race in two reproduced runs. The component controller successfully executed
memberJoinfor the new pod, then the same reconcile lost the following InstanceSet annotation update to a resourceVersion conflict. A later scale-in readmemberJoined=falsefrom the stale replicas-status annotation and skippedmemberLeave, leaving a stale Oracle DG broker member for the deleted pod.Observed evidence from Oracle validation:
succeed to join member for pod ...-oracle-2Operation cannot be fulfilled on instancesets... object has been modifiedjoined replicas: []whilememberLeavewas definedmemberLeavekbagent action was executedFix
UpdateReplicaStatusWithRetryfor InstanceSet replicas-status annotation writes.memberJoinsucceeds, persistmemberJoined=truethrough a fresh get/update retry loop instead of depending only on the later graph update.memberJoined=trueis still persisted.Tests
make test-go-generateKUBEBUILDER_ASSETS="/Users/wei/Library/Application Support/io.kubebuilder.envtest/k8s/1.26.1-darwin-arm64" go test ./pkg/controller/component -count=1KUBEBUILDER_ASSETS="/Users/wei/Library/Application Support/io.kubebuilder.envtest/k8s/1.26.1-darwin-arm64" go test ./controllers/apps/component -count=1Validation boundary
This closes the controller persistence race for new
memberJoinexecutions. It does not changememberLeavesemantics for replicas that are explicitly still marked unjoined, because forcing leave for known-unjoined replicas would require a broader addon action idempotency contract. Oracle should validate this exact patch image in the same vcluster scenario before this PR is marked ready.Fixes #10359