Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[sched] more error types when retry patching a pod's status #111809

Merged
merged 1 commit into from Aug 24, 2022
Merged

[sched] more error types when retry patching a pod's status #111809

merged 1 commit into from Aug 24, 2022

Conversation

Huang-Wei
Copy link
Member

What type of PR is this?

/kind feature
/sig scheduling

What this PR does / why we need it:

Our prod clusters' log shows that some intermittent errors can happen during PATCHing a pod's unschedulable status:

  1. ServiceUnavailable - this occurred when API Server cannot handle the request temporarily. 
    E0805 20:54:21.624945  123623 scheduler.go:356] Error updating pod foo: the server is currently unable to handle the request (patch pods foo)
  2. InternalError - this usually occurred due to glitch of interim webhooks. 
    E0811 23:32:30.886582  213747 scheduler.go:357] Error updating pod foo: Internal error occurred: failed calling webhook "xyz": Post "xyz": context deadline exceeded

This PR accormordates these 2 errors in addition of existing net.ConnectionRefused error.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Scheduler now retries updating a pod's status on ServiceUnavailable and InternalError errors, in addition to net ConnectionRefused error.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. labels Aug 12, 2022
@k8s-ci-robot
Copy link
Contributor

Please note that we're already in Test Freeze for the release-1.25 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.25.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Fri Aug 12 01:32:18 UTC 2022.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 12, 2022
@k8s-ci-robot
Copy link
Contributor

@Huang-Wei: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Aug 12, 2022
@Huang-Wei
Copy link
Member Author

/cc @sanposhiho @alculquicondor

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 12, 2022
@alculquicondor
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 12, 2022
@alculquicondor
Copy link
Member

/retest
Ref #111578

@k8s-ci-robot k8s-ci-robot merged commit 626ff5c into kubernetes:master Aug 24, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Aug 24, 2022
@Huang-Wei Huang-Wei deleted the retriable-sched-err branch August 24, 2022 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alculquicondor alculquicondor Awaiting requested review from alculquicondor

@sanposhiho sanposhiho Awaiting requested review from sanposhiho

@denkensk denkensk Awaiting requested review from denkensk

Assignees

@alculquicondor alculquicondor

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.26
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@Huang-Wei @k8s-ci-robot @alculquicondor