Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip mount point checks when possible during mount cleanup. #109676

Merged
merged 8 commits into from Jul 14, 2022

Conversation

cartermckinnon
Copy link
Contributor

@cartermckinnon cartermckinnon commented Apr 26, 2022

What type of PR is this?

/kind bug
/kind api-change

What this PR does / why we need it:

This is a continuation of #109117. Please see that PR for more background.

Calls to mounter.Unmount are preceded and followed by expensive mount point checks. These checks are not necessary on *nix-s with a umount implementation that performs a similar check itself. This PR adds a mechanism to detect the "safe" behavior, and avoid mount point checks when possible.

This change represents a significant optimization of CleanupMountPoint; enabling use-cases where pods have many mounts, and there is high pod churn. We (EKS) have observed several cases of instability and poor node health in such scenarios, which were resolved with this change.

Which issue(s) this PR fixes:

No issue available.

Special notes for your reviewer:

I chose to add a function to the Mounter interface in order to keep the CleanupMountPoint implementation generic for Unix and Windows. If the "safe" umount behavior is not detected, the existing code paths are unchanged.

Does this PR introduce a user-facing change?

On compatible systems, a mounter's Unmount implementation is changed to not return an error when the specified target can be detected as not a mount point. On Linux, the behavior of detecting a mount point depends on `umount` command is validated when the mounter is created. Additionally, mount point checks will be skipped in CleanupMountPoint/CleanupMountWithForce if the mounter's Unmount having the changed behavior of not returning error when target is not a mount point.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot
Copy link
Contributor

Please note that we're already in Test Freeze for the release-1.24 branch. This means every merged PR has to be cherry-picked into the release branch to be part of the upcoming v1.24.0 release.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. labels Apr 26, 2022
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Apr 26, 2022

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Apr 26, 2022
@k8s-ci-robot
Copy link
Contributor

@cartermckinnon: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Apr 26, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @cartermckinnon. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Apr 26, 2022
@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Apr 26, 2022
@@ -244,6 +244,11 @@ func (mounter *Mounter) IsLikelyNotMountPoint(file string) (bool, error) {
return true, nil
}

// CanSafelySkipMountPointCheck always returns false on Windows
func (mounter *Mounter) CanSafelySkipMountPointCheck() bool {
return false
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used false here to avoid changing any behavior on Windows; but looking at the Unmount implementation, I think this might be fine to return true?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ya I see the same, it won't make a difference since windows mounter.List does nothing and has a todo

// List returns a list of all mounted filesystems. todo
. i would leave it false, it is up to whoever eventually implements that List todo to figure out when/if CanSafelySkipMountPointCheck should ever return true

@k8s-triage-robot
Copy link

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

@wongma7
Copy link
Contributor

wongma7 commented Apr 26, 2022

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 26, 2022
@@ -303,6 +339,9 @@ func (mounter *Mounter) Unmount(target string) error {
// Rewrite err with the actual exit error of the process.
err = &exec.ExitError{ProcessState: command.ProcessState}
}
if mounter.withSafeNotMountedBehavior && strings.Contains(string(output), errNotMounted) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this in the right place? if umount exits 1 I think err will be non-nil so this belongs in the above if statement. Will need to test.

  1. create Pod with a volume
  2. ssh onto Node and unmount one of its volumes
  3. delete Pod
  4. does this condition actually get hit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is within the if err != nil {} block; and I think it should come after the ErrNoChildProcess check (as is). I'll do that test, to confirm.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put a log here, and update Unmount comment.

Since we are changing the library here, we also need to make sure this behavior change will not affect any other logic.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

many in-tree drivers uses this function, could you double check this behavior change will not affect any of them?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also if mount is bad/ corrupted, is it possible that "unmount" returns message "is not a mount point"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have all the context on what qualifies as corrupted; but all such not mounted errors will be silenced after this PR (if the mounter supports it). I can't find any in-tree drivers that rely on this error message (or any specific error message), so I'm not concerned about silencing it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a quick search, I think corrupted mounts often result in failed: Operation not permitted from umount. That behavior won't change with this PR, the error will be returned.

@cartermckinnon
Copy link
Contributor Author

/retest

@cartermckinnon
Copy link
Contributor Author

cartermckinnon commented Apr 27, 2022

I'm not able to reproduce this test failure using:

go test -run ^Test_Run_OneVolumeDetachFailNodeWithReadWriteOnce$ k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler

The line immediately preceding the failed assertion is a sleep statement: https://github.com/cartermckinnon/kubernetes/blob/master/pkg/controller/volume/attachdetach/reconciler/reconciler_test.go#L655

I'm going to try it again, but this seems like flakiness that isn't related to the changes in this PR.

@cartermckinnon
Copy link
Contributor Author

/retest

@k8s-ci-robot k8s-ci-robot removed the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Apr 27, 2022
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 14, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cartermckinnon, jingxu97, manugupt1, wongma7

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 14, 2022
@jingxu97
Copy link
Contributor

Thank you very much for the great effort on this PR! Sorry for the delayed review.

@k8s-triage-robot
Copy link

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

  • The PR does have any do-not-merge/* labels
  • The PR does not have the needs-ok-to-test label
  • The PR is mergeable (does not have a needs-rebase label)
  • The PR is approved (has cncf-cla: yes, lgtm, approved labels)
  • The PR is failing tests required for merge

You can:

/retest

@k8s-ci-robot k8s-ci-robot merged commit 7ac06e1 into kubernetes:master Jul 14, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.25 milestone Jul 14, 2022
nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Jul 18, 2022
Recently the k8s.io/mount-utils package added more runtime dectection.
When creating a new Mounter, the detect is run every time. This is
unfortunate, as it logs a message like the following:

```
mount_linux.go:283] Detected umount with safe 'not mounted' behavior
```

This message might be useful, so it probably good to keep it.

In Ceph-CSI there are various locations where Mounter instances are
created. Moving that to the DefaultNodeServer type reduces it to a
single place. Some utility functions need to accept the additional
parameter too, so that has been modified as well.

See-also: kubernetes/kubernetes#109676
Signed-off-by: Niels de Vos <ndevos@redhat.com>
@cartermckinnon cartermckinnon deleted the optimize-unmounts branch July 18, 2022 22:44
nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Jul 19, 2022
Recently the k8s.io/mount-utils package added more runtime dectection.
When creating a new Mounter, the detect is run every time. This is
unfortunate, as it logs a message like the following:

```
mount_linux.go:283] Detected umount with safe 'not mounted' behavior
```

This message might be useful, so it probably good to keep it.

In Ceph-CSI there are various locations where Mounter instances are
created. Moving that to the DefaultNodeServer type reduces it to a
single place. Some utility functions need to accept the additional
parameter too, so that has been modified as well.

See-also: kubernetes/kubernetes#109676
Signed-off-by: Niels de Vos <ndevos@redhat.com>
nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Jul 20, 2022
Recently the k8s.io/mount-utils package added more runtime dectection.
When creating a new Mounter, the detect is run every time. This is
unfortunate, as it logs a message like the following:

```
mount_linux.go:283] Detected umount with safe 'not mounted' behavior
```

This message might be useful, so it probably good to keep it.

In Ceph-CSI there are various locations where Mounter instances are
created. Moving that to the DefaultNodeServer type reduces it to a
single place. Some utility functions need to accept the additional
parameter too, so that has been modified as well.

See-also: kubernetes/kubernetes#109676
Signed-off-by: Niels de Vos <ndevos@redhat.com>
nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Jul 20, 2022
Recently the k8s.io/mount-utils package added more runtime dectection.
When creating a new Mounter, the detect is run every time. This is
unfortunate, as it logs a message like the following:

```
mount_linux.go:283] Detected umount with safe 'not mounted' behavior
```

This message might be useful, so it probably good to keep it.

In Ceph-CSI there are various locations where Mounter instances are
created. Moving that to the DefaultNodeServer type reduces it to a
single place. Some utility functions need to accept the additional
parameter too, so that has been modified as well.

See-also: kubernetes/kubernetes#109676
Signed-off-by: Niels de Vos <ndevos@redhat.com>
mergify bot pushed a commit to ceph/ceph-csi that referenced this pull request Jul 21, 2022
Recently the k8s.io/mount-utils package added more runtime dectection.
When creating a new Mounter, the detect is run every time. This is
unfortunate, as it logs a message like the following:

```
mount_linux.go:283] Detected umount with safe 'not mounted' behavior
```

This message might be useful, so it probably good to keep it.

In Ceph-CSI there are various locations where Mounter instances are
created. Moving that to the DefaultNodeServer type reduces it to a
single place. Some utility functions need to accept the additional
parameter too, so that has been modified as well.

See-also: kubernetes/kubernetes#109676
Signed-off-by: Niels de Vos <ndevos@redhat.com>
dlipovetsky pushed a commit to dlipovetsky/kubernetes that referenced this pull request Apr 6, 2023
…ounts

Skip mount point checks when possible during mount cleanup.
k8s-ci-robot added a commit that referenced this pull request May 12, 2023
…#109676-upstream-release-1.24

Automated cherry pick of #109676: Skip mount point checks when possible during mount
wongma7 pushed a commit to wongma7/kubernetes that referenced this pull request Jul 24, 2023
…nup.

Description:
* "Calls to mounter.Unmount are preceded and followed by expensive mount point checks. These checks are not necessary
on *nix-s with a umount implementation that performs a similar check itself. This PR adds a mechanism to detect the
"safe" behavior, and avoid mount point checks when possible. This change represents a significant optimization of
CleanupMountPoint; enabling use-cases where pods have many mounts, and there is high pod churn. We (EKS) have observed
several cases of instability and poor node health in such scenarios, which were resolved with this change." (PR
description)

Upstream PR, Issue, KEP, etc. links:
* Cherry pick of upstream Kubernetes PR kubernetes#109676 (kubernetes#109676)

If this patch is based on an upstream commit, how (if at all) do this patch and the upstream source differ?
* No differences

If this patch's changes have not been added by upstream, why not?
* N/A

Other patches related to this patch:
* None

Changes made to this patch after its initial creation and reasons for these changes:
* None

Kubernetes version this patch can be dropped:
* v1.25 -- upstream includes these changes starting in this version

Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Swizzmaster pushed a commit to Swizzmaster/kubernetes that referenced this pull request Feb 29, 2024
Swizzmaster pushed a commit to Swizzmaster/kubernetes that referenced this pull request Feb 29, 2024
Swizzmaster pushed a commit to Swizzmaster/kubernetes that referenced this pull request Feb 29, 2024
cherrypick kubernetes#109676 - 1.20

See merge request eks-dataplane/eks-kubernetes-patches!22
Swizzmaster pushed a commit to Swizzmaster/kubernetes that referenced this pull request Feb 29, 2024
cherrypick kubernetes#109676 - 1.22

See merge request eks-dataplane/eks-kubernetes-patches!24
Swizzmaster pushed a commit to Swizzmaster/kubernetes that referenced this pull request Feb 29, 2024
cherrypick kubernetes#109676 - 1.23

See merge request eks-dataplane/eks-kubernetes-patches!21
Swizzmaster pushed a commit to Swizzmaster/kubernetes that referenced this pull request Feb 29, 2024
Swizzmaster pushed a commit to Swizzmaster/kubernetes that referenced this pull request Feb 29, 2024
cherrypick kubernetes#109676 - 1.21

See merge request eks-dataplane/eks-kubernetes-patches!28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants