Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retry Unix domain sockets on Windows nodes for the plugin registration mechanism #110075

Merged
merged 1 commit into from Jun 28, 2022
Merged

Retry Unix domain sockets on Windows nodes for the plugin registration mechanism #110075

merged 1 commit into from Jun 28, 2022

Conversation

luckerby
Copy link
Contributor

@luckerby luckerby commented May 16, 2022 •
edited by k8s-ci-robot

What type of PR is this?

/kind bug

What this PR does / why we need it:

The plugin registration mechanism for Kubelet makes use of Unix domain sockets. On Linux this is working as expected, but on Windows - using the current Kubelet code - a socket is sometimes not validated given its backing file exists but the socket itself is not yet ready for communication at the time the plugin watcher code runs. One such occurrence can be seen here #104584 (comment).

Given the Kubelet ignores a plugin if its socket cannot be dialed on Windows, various workarounds need to be used - such as liveness probes - to force plugin re-registrations. However, this mechanism can require from just a couple to several hundred restarts for the containers responsible for creating the respective sockets, as described in the aforementioned issue. In turn this means lengthy amounts of time are sometimes required for a node to be up and running (e.g. CSI secrets store driver is essentially broken for the duration of the restarts, preventing any pod relying on it to inject secrets to start). Further, this impacts features such as cluster autoscaler on Windows.

This PR changes the check performed by Kubelet against Unix domain sockets on Windows nodes from a one-time attempt to a retry approach.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Please see the Kubelet logs here when running the original code in AKS 1.22, showing sockets being ignored multiple times. With the code in this PR the logs change as seen here with at most one retry until the respective target socket is successfully dialed.

Does this PR introduce a user-facing change?

kubelet: add retry of checking Unix domain sockets on Windows nodes for the plugin registration mechanism

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 16, 2022
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented May 16, 2022 •
edited

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: luckerby / name: Mihai Albert (ea5ec6cfb7fe630f0884a5be01cf20bf8dd3dfec, f1f9b889581e86546e58d1afed5dc728914e5067, e5d1e2292a807d4e38296e8af53faefb5f85952f, 119dd639c51f2d2d38a947cf64fc8d399c5e3dbe, 3df9131977922690218e3308251d1079621c2592)

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 16, 2022
@k8s-ci-robot
Copy link
Contributor

Welcome @luckerby!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 16, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @luckerby. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot requested a review from pacoxu May 16, 2022 23:37
@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 16, 2022
@luckerby luckerby force-pushed the 104584-retry-dial-on-socket-windows-base branch from 3bbdb1f to 3df9131 Compare May 17, 2022 00:14
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 17, 2022
@jsturtevant
Copy link
Contributor

/ok-to-test

xref: #104584

/sig windows
/triage accepted
/priority important-soon
/milestone v1.25

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. sig/windows Categorizes an issue or PR as relevant to SIG Windows. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 17, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.25 milestone May 17, 2022
@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 17, 2022
@jsturtevant
Copy link
Contributor

/cc @mauriciopoppe @aramase @andyzhangx @marosset

@luckerby luckerby force-pushed the 104584-retry-dial-on-socket-windows-base branch from fe2359a to c666656 Compare June 9, 2022 22:00
@jsturtevant
Copy link
Contributor

/lgtm
/assign @derekwaynecarr @marosset @SergeyKanzhelev

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 10, 2022
@marosset
Copy link
Contributor

/lgtm
Thanks for adding additional test coverage for the scenarios where sockets are initialized on Windows yet.

@marosset marosset moved this from In Review (v1.25) to Reviewed - Needs Approval From Other SIGs in SIG-Windows Jun 16, 2022
@pacoxu pacoxu moved this from Triage to Needs Approver in SIG Node PR Triage Jun 20, 2022
@pacoxu
Copy link
Member

pacoxu commented Jun 20, 2022

/assign @mrunalp

@mrunalp mrunalp moved this from Needs Approver to Done in SIG Node PR Triage Jun 28, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: luckerby, mrunalp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 28, 2022
@k8s-ci-robot k8s-ci-robot merged commit b19d50d into kubernetes:master Jun 28, 2022
SIG-Windows automation moved this from Reviewed - Needs Approval From Other SIGs to Done (v1.25) Jun 28, 2022
@aramase
Copy link
Member

aramase commented Jun 28, 2022

@luckerby Is there a plan to cherry pick this to older releases (1.24 and 1.23)?

@aramase aramase mentioned this pull request Jun 28, 2022
Closed
@luckerby
Copy link
Contributor Author

@aramase Yes, I plan to do this for older releases as well in the coming days

@claudiubelu claudiubelu mentioned this pull request Jul 1, 2022
Merged
This was referenced Jul 3, 2022
Merged
Merged
@jsturtevant jsturtevant mentioned this pull request Jul 26, 2022
Closed
@mauriciopoppe mauriciopoppe mentioned this pull request Aug 4, 2022
Closed
k8s-ci-robot added a commit that referenced this pull request Aug 10, 2022
@k8s-ci-robot
Merge pull request #110934 from luckerby/automated-cherry-pick-of-#11…
0892347 
…0075-upstream-release-1.23

Automated cherry pick of #110075: Add retry logic for Unix Domain sockets on Windows
k8s-ci-robot added a commit that referenced this pull request Aug 11, 2022
@k8s-ci-robot
Merge pull request #110935 from luckerby/automated-cherry-pick-of-#11…
0b08f3c 
…0075-upstream-release-1.24

Automated cherry pick of #110075: Add retry logic for Unix Domain sockets on Windows
@mauriciopoppe mauriciopoppe mentioned this pull request Nov 17, 2022
Closed
@mpatlasov mpatlasov mentioned this pull request Oct 4, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsturtevant jsturtevant jsturtevant left review comments

@mauriciopoppe mauriciopoppe mauriciopoppe left review comments

@andyzhangx andyzhangx andyzhangx left review comments

@aramase aramase aramase left review comments

@mrunalp mrunalp mrunalp approved these changes

@pacoxu pacoxu Awaiting requested review from pacoxu

@Random-Liu Random-Liu Awaiting requested review from Random-Liu

@marosset marosset Awaiting requested review from marosset

Assignees

@mrunalp mrunalp

@jsturtevant jsturtevant

@derekwaynecarr derekwaynecarr

@SergeyKanzhelev SergeyKanzhelev

@marosset marosset

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/windows Categorizes an issue or PR as relevant to SIG Windows. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node PR Triage
Done
SIG-Windows
  
Done (v1.25)
Milestone
v1.25
Development

Successfully merging this pull request may close these issues.

None yet