Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubelet: fix nil pointer in startReflector for standalone mode #113501

Merged
merged 3 commits into from Nov 9, 2022
Merged

kubelet: fix nil pointer in startReflector for standalone mode #113501

merged 3 commits into from Nov 9, 2022

Conversation

pacoxu
Copy link
Member

@pacoxu pacoxu commented Nov 1, 2022 •
edited

What type of PR is this?

/kind bug

What this PR does / why we need it:

  1. The startReflector was introduced in reduce configmap and secret watch of kubelet #99393 @chenyw1990 since v1.21
  2. Fix leaking goroutines in kubelet integration test #110138 fixed a goroutine leak in it @wojtek-t
  3. This PR focus on NPE fix.

Which issue(s) this PR fixes:

Fixes #113492

Special notes for your reviewer:

2022-10-31T20:00:48+0000 gke-b27bb59e05314f15b662-a7f3-3599-vm kubelet[3019]: k8s.io/kubernetes/pkg/kubelet/util/manager.(*objectCacheItem).startReflector(0xc000e03ce0)\r\n
2022-10-31T20:00:48+0000 gke-b27bb59e05314f15b662-a7f3-3599-vm kubelet[3019]: pkg/kubelet/util/manager/watch_based_manager.go:123 +0x91\r\n

https://github.com/kubernetes/kubernetes/blob/1f60f12e0ca7d26f9464e036b78f0f3974735ed4/pkg/kubelet/status/status_manager.go#L665-L672

if running in standalone mode, we should not update the pod status.

Does this PR introduce a user-facing change?

kubelet: fix nil pointer in reflector start for standalone mode

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 1, 2022
@k8s-ci-robot k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 1, 2022
@pacoxu
Copy link
Member Author

pacoxu commented Nov 1, 2022

BTW, should this be cherry-picked to v1.23-v1.25?

@pacoxu
Copy link
Member Author

pacoxu commented Nov 1, 2022

/test pull-kubernetes-unit

@pacoxu pacoxu added this to Needs Reviewer in SIG Node PR Triage Nov 1, 2022
@aojea
Copy link
Member

aojea commented Nov 1, 2022

an e2e should be easy to add, is just creating a static pod with a reference to a configmap

#113492

aojea
aojea reviewed Nov 1, 2022
View reviewed changes
pkg/kubelet/kubelet.go Outdated
@@ -1635,7 +1635,7 @@ func (kl *Kubelet) syncPod(ctx context.Context, updateType kubetypes.SyncPodType
}

// ensure the kubelet knows about referenced secrets or configmaps used by the pod
if !kl.podWorkers.IsPodTerminationRequested(pod.UID) {
if !kl.podWorkers.IsPodTerminationRequested(pod.UID) && kl.kubeClient != nil {
if kl.secretManager != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question, why kl.secretManager and kl.configMapManager should be not nil in a standalone kubelet?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to be better to not set secret and configmap manager if kubeClient is nil.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, no one to talk to :)

@aojea
Copy link
Member

aojea commented Nov 1, 2022

/assign @bobbypage @SergeyKanzhelev

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 2, 2022
@pacoxu pacoxu marked this pull request as draft November 2, 2022 03:24
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 2, 2022
@pacoxu pacoxu marked this pull request as ready for review November 2, 2022 03:43
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 2, 2022
@aojea
Copy link
Member

aojea commented Nov 4, 2022

LGTM

@bobbypage @SergeyKanzhelev can you PTAL?

@aojea
Copy link
Member

aojea commented Nov 4, 2022

oh ,is still wip ?

@pacoxu
Copy link
Member Author

pacoxu commented Nov 4, 2022

Let me check again later.

@pacoxu pacoxu changed the title [wip]kubelet: fix nil pointer in startReflector for standalone mode kubelet: fix nil pointer in startReflector for standalone mode Nov 4, 2022
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 4, 2022
@pacoxu
Copy link
Member Author

pacoxu commented Nov 4, 2022

oh ,is still wip ?

I test on Mac directly at first and the previous CI failure is for linus checking /dev/kmsg permission. Is it OK to add a fake check in linux oom code（a little odd）.

@pacoxu
Copy link
Member Author

pacoxu commented Nov 4, 2022

/test pull-kubernetes-unit
to double check

@bart0sh
Copy link
Contributor

bart0sh commented Nov 7, 2022

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 7, 2022
@pacoxu
Copy link
Member Author

pacoxu commented Nov 7, 2022

/assign @mrunalp

@pacoxu
Copy link
Member Author

pacoxu commented Nov 8, 2022

ping @klueska
Do you have time to take a look before code freeze?

bobbypage
bobbypage reviewed Nov 8, 2022
View reviewed changes
pkg/kubelet/volume_host.go Outdated Show resolved Hide resolved
pkg/kubelet/volume_host.go Outdated Show resolved Hide resolved
pkg/kubelet/kubelet.go

klet.secretManager = secretManager
klet.configMapManager = configMapManager
if klet.kubeClient != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: klet.kubeClient is equal to kubeDeps.KubeClient (set above on line 502)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes.

#113501 (comment)

  • After klet is set, I changed to use klet.kubeClient here.
  • Before klet is set, we use kubeDeps.KubeClient

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is per @bart0sh 's comment.

  • I think this is clear and will help if we will have some refactoring on this later.

@bobbypage
Copy link
Member

LGTM, just a few doc comments, but change and test makes sense.

/lgtm

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 8, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mrunalp, pacoxu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 8, 2022
@bobbypage
Copy link
Member

Thanks for updates!

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 9, 2022
@pacoxu
Copy link
Member Author

pacoxu commented Nov 9, 2022 •
edited

This was approved before code freeze. Can anyone help to add the milestone label?

@kubernetes/milestone-maintainers

@aojea
Copy link
Member

aojea commented Nov 9, 2022

/milestone v1.26

@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Nov 9, 2022
@k8s-ci-robot k8s-ci-robot merged commit 70263d5 into kubernetes:master Nov 9, 2022
SIG Node PR Triage automation moved this from Needs Reviewer to Done Nov 9, 2022
@liggitt liggitt mentioned this pull request Mar 6, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dims dims dims left review comments

@bobbypage bobbypage bobbypage left review comments

@aojea aojea aojea left review comments

@bart0sh bart0sh bart0sh left review comments

@mrunalp mrunalp mrunalp approved these changes

@tallclair tallclair Awaiting requested review from tallclair

@mtaufen mtaufen Awaiting requested review from mtaufen

@SergeyKanzhelev SergeyKanzhelev Awaiting requested review from SergeyKanzhelev

Assignees

@mrunalp mrunalp

@bobbypage bobbypage

@SergeyKanzhelev SergeyKanzhelev

@bart0sh bart0sh

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node PR Triage
Done
Milestone
v1.26
Development

Successfully merging this pull request may close these issues.

kubelet crashes when attempting to bring up a static pod that refers to a configmap
8 participants
@pacoxu @aojea @dims @bart0sh @bobbypage @k8s-ci-robot @mrunalp @SergeyKanzhelev