Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resolve winkernel proxier treating stale VIPs as valid #113521

Merged
merged 2 commits into from Nov 2, 2022
Merged

resolve winkernel proxier treating stale VIPs as valid #113521

merged 2 commits into from Nov 2, 2022

Conversation

daschott
Copy link
Contributor

@daschott daschott commented Nov 2, 2022 •
edited

What type of PR is this?

/kind bug

What this PR does / why we need it:

There is an issue on Windows Server 2019 that can cause the kube-proxy call into HNS to delete the load balancer proxy rules to fail. This case is not handled very well and we should add resiliency for such failures. If the backends of a particular services are all deleted, but the call to HNS to cleanup the load balancer proxy rules did not succeed, if the same service get's re-created again in the future and the number of new endpoints matches the number of endpoints that used to exist, kube-proxy will think that the old service policy proxy rules that never got deleted is still valid. This will lead to in-cluster network failures for traffic passing through Windows Nodes.

To improve on this issue this PR:

  1. adds an identifier to the load balancer proxy rules that is a basic hash of the set of backends (endpoint IDs) that are unique for each endpoint, to avoid the case of a service VIP that leaked being re-used again.
  2. adds a retry to the proxy rules deletion, to workaround the issue of HNS failing to delete the proxy rules sometimes.
  3. improves error logging in cases of proxy rules failing to be cleaned up due to a HNS failure.

Which issue(s) this PR fixes:

Fixes #113524

Special notes for your reviewer:

At this time, I have not confirmed if this issue also occurs on Windows Server 2022.

Does this PR introduce a user-facing change?

Resolves an issue that causes winkernel proxier to treat stale VIPs as valid

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 2, 2022
@k8s-ci-robot
Copy link
Contributor

@daschott: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 2, 2022
@daschott
Copy link
Contributor Author

daschott commented Nov 2, 2022

/sig windows

@k8s-ci-robot k8s-ci-robot added sig/windows Categorizes an issue or PR as relevant to SIG Windows. approved Indicates a PR has been approved by an approver from all required OWNERS files. sig/network Categorizes an issue or PR as relevant to SIG Network. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 2, 2022
@daschott daschott mentioned this pull request Nov 2, 2022
Merged
@sbangari
Copy link
Contributor

sbangari commented Nov 2, 2022

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 2, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: daschott, sbangari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit b60b0c7 into kubernetes:master Nov 2, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Nov 2, 2022
@daschott daschott mentioned this pull request Nov 3, 2022
Merged
k8s-ci-robot added a commit that referenced this pull request Dec 2, 2022
@k8s-ci-robot
Merge pull request #113567 from daschott/automated-cherry-pick-of-#11…
0b9a983 
…3521-upstream-release-1.24

Automated cherry pick of #113521: added backend hashing to winkernel proxier
k8s-ci-robot added a commit that referenced this pull request Dec 2, 2022
@k8s-ci-robot
Merge pull request #113558 from daschott/automated-cherry-pick-of-#11…
a0de9e1 
…3521-upstream-release-1.25

Automated cherry pick of #113521: added backend hashing to winkernel proxier
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@feiskyer feiskyer Awaiting requested review from feiskyer

@sbangari sbangari Awaiting requested review from sbangari

Assignees

@sbangari sbangari

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/windows Categorizes an issue or PR as relevant to SIG Windows. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
[sig-windows] PR tracking
Status: Done
Milestone
v1.26
Development

Successfully merging this pull request may close these issues.

resolve winkernel proxier treating stale VIPs as valid
3 participants
@daschott @k8s-ci-robot @sbangari