Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update endpointslice controller maximum sync backoff delay to match expected sequence of delays #112353

Merged
merged 1 commit into from Sep 14, 2022
Merged

Update endpointslice controller maximum sync backoff delay to match expected sequence of delays #112353

merged 1 commit into from Sep 14, 2022

Conversation

dgrisonnet
Copy link
Member

What type of PR is this?

/kind bug

What this PR does / why we need it:

Update the maximum sync backoff value to 1000s to match the sequence of delays expected by the endpointslice controller when syncing Services:

Before this change the sequence was:

1s, 2s, 4s, 8s, 16s, 32s, 64s, 100s

Now it is:

1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s, 1000s

Special notes for your reviewer:

The expectation for the sequence of delays was detailed in the following comment but in practice, the queue was configured for a shorter sequence.

kubernetes/pkg/controller/endpointslice/endpointslice_controller.go

Line 59 in 2b2be7f

// 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s, 1000s (max)

Does this PR introduce a user-facing change?

Increase the maximum backoff delay of the endpointslice controller to match the expected sequence of delays when syncing Services.

/cc @aojea

@dgrisonnet
endpointslice: update max sync backoff
0f0526c 
Update the maximum sync backoff value to 1000s to match the sequence of
delays expected by the endpointslice controller when syncing Services:

Before this change the sequence was:

> 1s, 2s, 4s, 8s, 16s, 32s, 64s, 100s

Now it is:

> 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s, 1000s

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 9, 2022
@k8s-ci-robot
Copy link
Contributor

@dgrisonnet: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/network Categorizes an issue or PR as relevant to SIG Network. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 9, 2022
@aojea
Copy link
Member

aojea commented Sep 9, 2022

/assign @wojtek-t @robscott
they have more context, I think they came up with the number based on real feedback

@robscott
Copy link
Member

This change looks reasonable to me, but I'll have to defer to @wojtek-t or @mborsz here as they have a better handle on the scalability side of this.

/cc @mborsz

@mborsz
Copy link
Member

mborsz commented Sep 13, 2022

I don't have enough context to help with review of this change.

@dgrisonnet
Copy link
Member Author

I don't have any context either, but I personally feel that a human-made comment detailing the exact delay sequence of the controller is more trustworthy than a potential typo.

wojtek-t
wojtek-t reviewed Sep 14, 2022
View reviewed changes
pkg/controller/endpointslice/endpointslice_controller.go
@@ -69,7 +69,7 @@ const (
// defaultSyncBackOff is the default backoff period for syncService calls.
defaultSyncBackOff = 1 * time.Second
// maxSyncBackOff is the max backoff period for syncService calls.
maxSyncBackOff = 100 * time.Second
maxSyncBackOff = 1000 * time.Second
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was introduced here:
https://github.com/kubernetes/kubernetes/pull/89438/files

This backoff is used when we fail the operation:
https://github.com/kubernetes/kubernetes/pull/89438/files#diff-8765679aa8ca8cc500a6dc038b8707cbc29ddc2d2f94f1da911e2cd5e4df5816R104

If we can't update ES within 100s backoff, we're really in big troubles or this is some kind of permanent error (like too big object or sth). Extending this to 1000s in those cases makes sense to me.

Although I'm not sure if the original numbers where backed by some real world scenarios...

@wojtek-t
Copy link
Member

/lgtm
/approve

[Cluster failed to start]
/retest

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 14, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgrisonnet, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 14, 2022
@k8s-ci-robot k8s-ci-robot merged commit c7d47e4 into kubernetes:master Sep 14, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Sep 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wojtek-t wojtek-t wojtek-t left review comments

@aojea aojea Awaiting requested review from aojea

@mborsz mborsz Awaiting requested review from mborsz

Assignees

@robscott robscott

@wojtek-t wojtek-t

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/network Categorizes an issue or PR as relevant to SIG Network. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.26
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@dgrisonnet @k8s-ci-robot @aojea @robscott @mborsz @wojtek-t