Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert "promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta" #112076

Merged
merged 1 commit into from Aug 27, 2022
Merged

Revert "promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta" #112076

merged 1 commit into from Aug 27, 2022

Conversation

rphillips
Copy link
Member

@rphillips rphillips commented Aug 26, 2022 •
edited by liggitt

Reverts #107329

OpenShift found issues rendering updated ConfigMaps to pods. When ConfigMaps get updated within the API, they do not get rendered to the resulting pod's filesystem by the Kubelet with the following error:

Aug 26 20:22:30 test1-lkz4t-master-1 kubenswrapper[1474]: E0826 20:22:30.751336    1474 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/d381fba5-463d-420a-8ef2-ba4d9a94846d-trusted-ca-bundle podName:d381fba5-463d-420a-8ef2-ba4d9a94846d nodeName:}" failed. No retries permitted until 2022-08-26 20:24:32.751309823 +0000 UTC m=+2312.350897767 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp
 failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/d381fba5-463d-420a-8ef2-ba4d9a94846d-trusted-ca-bundle") pod "insights-operator-5655ffbd97-x2gqw" (UID: "d381fba5-463d-420a-8ef2-ba4d9a94846d") : requesting quota on existing directory /var/lib/kubelet/pods/d381fba5-463d-420a-8ef2-ba4d9a94846d/volumes/kubernetes.io~configmap/trusted-ca-bundle but different pod be50da94-ebf1-4cb0-9e0b-2949fd2bed7b f9
8fa631-4940-4217-9b82-4e5ad6720238

We will need to backport to 1.25.

Fix a 1.25 regression updating mounted configmaps when the LocalStorageCapacityIsolationFSQuotaMonitoring feature is enabled. LocalStorageCapacityIsolationFSQuotaMonitoring feature gate is demoted back to Alpha.

/sig node
/kind regression
/priority important-soon

cc @derekwaynecarr @pacoxu @dchen1107 @liggitt @deads2k

@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. sig/node Categorizes an issue or PR as relevant to SIG Node. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. kind/regression Categorizes issue or PR as related to a regression from a prior release. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 26, 2022
@k8s-ci-robot
Copy link
Contributor

@rphillips: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Aug 26, 2022
@k8s-ci-robot k8s-ci-robot added sig/testing Categorizes an issue or PR as relevant to SIG Testing. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 26, 2022
@rphillips rphillips mentioned this pull request Aug 26, 2022
Merged
@kikisdeliveryservice
Copy link
Member

Related enhancement: kubernetes/enhancements#1029

@k8s-ci-robot
Copy link
Contributor

@rphillips: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kubernetes-e2e-gce-alpha-features 32a90f5 link false /test pull-kubernetes-e2e-gce-alpha-features

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@pacoxu
Copy link
Member

pacoxu commented Aug 26, 2022

/lgtm
If there is critical issue with the feature.

I will take a look next week.

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 26, 2022
@pacoxu pacoxu mentioned this pull request Aug 27, 2022
Open
21 tasks
@pacoxu pacoxu added this to Needs Approver in SIG Node PR Triage Aug 27, 2022
@liggitt
Copy link
Member

liggitt commented Aug 27, 2022

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 27, 2022
@pacoxu pacoxu mentioned this pull request Aug 27, 2022
Closed
@k8s-ci-robot k8s-ci-robot merged commit 569b14c into kubernetes:master Aug 27, 2022
SIG Node PR Triage automation moved this from Needs Approver to Done Aug 27, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Aug 27, 2022
@openshift-ci-robot openshift-ci-robot mentioned this pull request Aug 27, 2022
Closed
dghubble added a commit to poseidon/typhoon that referenced this pull request Aug 27, 2022
@dghubble
Disable LocalStorageCapacityIsolationFSQuotaMonitoring feature
34b8833 
* Kubernetes v1.25.0 moved the LocalStorageCapacityIsolationFSQuotaMonitoring
feature from alpha to beta, but it breaks Kubelet updating ConfigMaps in
Pods, as shown by conformance tests
* Kubernetes is rolling LocalStorageCapacityIsolationFSQuotaMonitoring back
to alpha so its not enabled by default, but that will require a release
* Disable the feature gate directly as a workaround for now to make
Kubernetes v1.25.0 usable

```
FailedMount: MountVolume.SetUp failed for volume "configmap-volume" : requesting quota on existing directory /var/lib/kubelet/pods/f09fae17-ff16-4a05-aab3-7b897cb5b732/volumes/kubernetes.io~configmap/configmap-volume but different pod 673ad247-abf0-434e-99eb-1c3f57d7fdaa a4568e94-2b2d-438f-a4bd-c9edc814e478
```

Rel:

* kubernetes/kubernetes#112076
* kubernetes/kubernetes#107329
dghubble added a commit to poseidon/typhoon that referenced this pull request Aug 27, 2022
@dghubble
Disable LocalStorageCapacityIsolationFSQuotaMonitoring feature
275fc0f 
* Kubernetes v1.25.0 moved the LocalStorageCapacityIsolationFSQuotaMonitoring
feature from alpha to beta, but it breaks Kubelet updating ConfigMaps in
Pods, as shown by conformance tests
* Kubernetes is rolling LocalStorageCapacityIsolationFSQuotaMonitoring back
to alpha so its not enabled by default, but that will require a release
* Disable the feature gate directly as a workaround for now to make
Kubernetes v1.25.0 usable

```
FailedMount: MountVolume.SetUp failed for volume "configmap-volume" : requesting quota on existing directory /var/lib/kubelet/pods/f09fae17-ff16-4a05-aab3-7b897cb5b732/volumes/kubernetes.io~configmap/configmap-volume but different pod 673ad247-abf0-434e-99eb-1c3f57d7fdaa a4568e94-2b2d-438f-a4bd-c9edc814e478
```

Rel:

* kubernetes/kubernetes#112076
* kubernetes/kubernetes#107329
@dghubble dghubble mentioned this pull request Aug 27, 2022
Merged
dghubble-robot pushed a commit to poseidon/terraform-azure-kubernetes that referenced this pull request Aug 27, 2022
@dghubble
Disable LocalStorageCapacityIsolationFSQuotaMonitoring feature
d71810b 
* Kubernetes v1.25.0 moved the LocalStorageCapacityIsolationFSQuotaMonitoring
feature from alpha to beta, but it breaks Kubelet updating ConfigMaps in
Pods, as shown by conformance tests
* Kubernetes is rolling LocalStorageCapacityIsolationFSQuotaMonitoring back
to alpha so its not enabled by default, but that will require a release
* Disable the feature gate directly as a workaround for now to make
Kubernetes v1.25.0 usable

```
FailedMount: MountVolume.SetUp failed for volume "configmap-volume" : requesting quota on existing directory /var/lib/kubelet/pods/f09fae17-ff16-4a05-aab3-7b897cb5b732/volumes/kubernetes.io~configmap/configmap-volume but different pod 673ad247-abf0-434e-99eb-1c3f57d7fdaa a4568e94-2b2d-438f-a4bd-c9edc814e478
```

Rel:

* kubernetes/kubernetes#112076
* kubernetes/kubernetes#107329
dghubble-robot pushed a commit to poseidon/terraform-digitalocean-kubernetes that referenced this pull request Aug 27, 2022
@dghubble
Disable LocalStorageCapacityIsolationFSQuotaMonitoring feature
33833d1 
* Kubernetes v1.25.0 moved the LocalStorageCapacityIsolationFSQuotaMonitoring
feature from alpha to beta, but it breaks Kubelet updating ConfigMaps in
Pods, as shown by conformance tests
* Kubernetes is rolling LocalStorageCapacityIsolationFSQuotaMonitoring back
to alpha so its not enabled by default, but that will require a release
* Disable the feature gate directly as a workaround for now to make
Kubernetes v1.25.0 usable

```
FailedMount: MountVolume.SetUp failed for volume "configmap-volume" : requesting quota on existing directory /var/lib/kubelet/pods/f09fae17-ff16-4a05-aab3-7b897cb5b732/volumes/kubernetes.io~configmap/configmap-volume but different pod 673ad247-abf0-434e-99eb-1c3f57d7fdaa a4568e94-2b2d-438f-a4bd-c9edc814e478
```

Rel:

* kubernetes/kubernetes#112076
* kubernetes/kubernetes#107329
dghubble-robot pushed a commit to poseidon/terraform-onprem-kubernetes that referenced this pull request Aug 27, 2022
@dghubble
Disable LocalStorageCapacityIsolationFSQuotaMonitoring feature
e9b6c84 
* Kubernetes v1.25.0 moved the LocalStorageCapacityIsolationFSQuotaMonitoring
feature from alpha to beta, but it breaks Kubelet updating ConfigMaps in
Pods, as shown by conformance tests
* Kubernetes is rolling LocalStorageCapacityIsolationFSQuotaMonitoring back
to alpha so its not enabled by default, but that will require a release
* Disable the feature gate directly as a workaround for now to make
Kubernetes v1.25.0 usable

```
FailedMount: MountVolume.SetUp failed for volume "configmap-volume" : requesting quota on existing directory /var/lib/kubelet/pods/f09fae17-ff16-4a05-aab3-7b897cb5b732/volumes/kubernetes.io~configmap/configmap-volume but different pod 673ad247-abf0-434e-99eb-1c3f57d7fdaa a4568e94-2b2d-438f-a4bd-c9edc814e478
```

Rel:

* kubernetes/kubernetes#112076
* kubernetes/kubernetes#107329
dghubble-robot pushed a commit to poseidon/terraform-aws-kubernetes that referenced this pull request Aug 27, 2022
@dghubble
Disable LocalStorageCapacityIsolationFSQuotaMonitoring feature
96ca485 
* Kubernetes v1.25.0 moved the LocalStorageCapacityIsolationFSQuotaMonitoring
feature from alpha to beta, but it breaks Kubelet updating ConfigMaps in
Pods, as shown by conformance tests
* Kubernetes is rolling LocalStorageCapacityIsolationFSQuotaMonitoring back
to alpha so its not enabled by default, but that will require a release
* Disable the feature gate directly as a workaround for now to make
Kubernetes v1.25.0 usable

```
FailedMount: MountVolume.SetUp failed for volume "configmap-volume" : requesting quota on existing directory /var/lib/kubelet/pods/f09fae17-ff16-4a05-aab3-7b897cb5b732/volumes/kubernetes.io~configmap/configmap-volume but different pod 673ad247-abf0-434e-99eb-1c3f57d7fdaa a4568e94-2b2d-438f-a4bd-c9edc814e478
```

Rel:

* kubernetes/kubernetes#112076
* kubernetes/kubernetes#107329
dghubble-robot pushed a commit to poseidon/terraform-google-kubernetes that referenced this pull request Aug 27, 2022
@dghubble
Disable LocalStorageCapacityIsolationFSQuotaMonitoring feature
46606f7 
* Kubernetes v1.25.0 moved the LocalStorageCapacityIsolationFSQuotaMonitoring
feature from alpha to beta, but it breaks Kubelet updating ConfigMaps in
Pods, as shown by conformance tests
* Kubernetes is rolling LocalStorageCapacityIsolationFSQuotaMonitoring back
to alpha so its not enabled by default, but that will require a release
* Disable the feature gate directly as a workaround for now to make
Kubernetes v1.25.0 usable

```
FailedMount: MountVolume.SetUp failed for volume "configmap-volume" : requesting quota on existing directory /var/lib/kubelet/pods/f09fae17-ff16-4a05-aab3-7b897cb5b732/volumes/kubernetes.io~configmap/configmap-volume but different pod 673ad247-abf0-434e-99eb-1c3f57d7fdaa a4568e94-2b2d-438f-a4bd-c9edc814e478
```

Rel:

* kubernetes/kubernetes#112076
* kubernetes/kubernetes#107329
k8s-ci-robot added a commit that referenced this pull request Aug 29, 2022
@k8s-ci-robot
Merge pull request #112078 from rphillips/automated-cherry-pick-of-#1…
2b09991 
…12076-upstream-release-1.25

Automated cherry pick of #112076: Revert "promote
@aojea
Copy link
Member

aojea commented Aug 29, 2022

/cc @wojtek-t @deads2k

wg-reliability

@kikisdeliveryservice kikisdeliveryservice mentioned this pull request Aug 29, 2022
Merged
kikisdeliveryservice added a commit to kikisdeliveryservice/k8s-website that referenced this pull request Aug 29, 2022
@kikisdeliveryservice
Revert: promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta
96a8fde 
This feature has been reverted to alpha in master and 1.25.
Related-to: kubernetes/kubernetes#112076
Related-to: kubernetes/kubernetes#112078
Related-to: kubernetes/enhancements#1029
@kikisdeliveryservice kikisdeliveryservice mentioned this pull request Aug 29, 2022
Merged
kikisdeliveryservice added a commit to kikisdeliveryservice/k8s-website that referenced this pull request Aug 30, 2022
@kikisdeliveryservice
Revert: promote LocalStorageCapacityIsolationFSQuotaMonitoring to beta
04f9794 
This feature has been reverted to alpha in master and 1.25.
Related-to: kubernetes/kubernetes#112076
Related-to: kubernetes/kubernetes#112078
Related-to: kubernetes/enhancements#1029
@openshift-ci-robot openshift-ci-robot mentioned this pull request Sep 1, 2022
Merged
@openshift-ci-robot openshift-ci-robot mentioned this pull request Sep 16, 2022
Closed
@pacoxu pacoxu mentioned this pull request Sep 21, 2022
Draft
11 tasks
@swghosh swghosh mentioned this pull request Oct 26, 2022
Merged
Snaipe pushed a commit to aristanetworks/monsoon that referenced this pull request Apr 13, 2023
@dghubble
Disable LocalStorageCapacityIsolationFSQuotaMonitoring feature
a7015b8 
* Kubernetes v1.25.0 moved the LocalStorageCapacityIsolationFSQuotaMonitoring
feature from alpha to beta, but it breaks Kubelet updating ConfigMaps in
Pods, as shown by conformance tests
* Kubernetes is rolling LocalStorageCapacityIsolationFSQuotaMonitoring back
to alpha so its not enabled by default, but that will require a release
* Disable the feature gate directly as a workaround for now to make
Kubernetes v1.25.0 usable

```
FailedMount: MountVolume.SetUp failed for volume "configmap-volume" : requesting quota on existing directory /var/lib/kubelet/pods/f09fae17-ff16-4a05-aab3-7b897cb5b732/volumes/kubernetes.io~configmap/configmap-volume but different pod 673ad247-abf0-434e-99eb-1c3f57d7fdaa a4568e94-2b2d-438f-a4bd-c9edc814e478
```

Rel:

* kubernetes/kubernetes#112076
* kubernetes/kubernetes#107329
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@odinuge odinuge Awaiting requested review from odinuge

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@deads2k deads2k Awaiting requested review from deads2k

@wojtek-t wojtek-t Awaiting requested review from wojtek-t

Assignees

@pacoxu pacoxu

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/regression Categorizes issue or PR as related to a regression from a prior release. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
SIG Node PR Triage
Done
Milestone
v1.26
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@rphillips @k8s-ci-robot @kikisdeliveryservice @pacoxu @liggitt @aojea