Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes getNestedMountpoints grouping #112571

Merged
merged 1 commit into from Oct 19, 2022

Conversation

claudiubelu
Copy link
Contributor

@claudiubelu claudiubelu commented Sep 19, 2022

What type of PR is this?

/kind bug
/sig windows

/priority important-soon

What this PR does / why we need it:

Currently, getNestedMountpoints sorts using sort.Strings, which would sort the following strings in this exact order:

/dir/nested, /dir/nested-vol, /dir/nested.vol, /dir/nested/double, /dir/nested2

Because of this, nested/double is returned as well, even though it shouldn't have been. This issue is worse on Windows, where the path separator is typically the backslash.

This commit addresses this issue by checking if a nested mount point has been previously seen or not.

Which issue(s) this PR fixes:

Fixes #112570
Related: #51540

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Nested mountpoints are now group correctly on all cases.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. sig/windows Categorizes an issue or PR as relevant to SIG Windows. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/storage Categorizes an issue or PR as relevant to SIG Storage. labels Sep 19, 2022
@claudiubelu
Copy link
Contributor Author

/test pull-kubernetes-e2e-kind

@claudiubelu
Copy link
Contributor Author

/assign @saad-ali

@marosset
Copy link
Contributor

/milestone v1.26

@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Sep 22, 2022
@marosset
Copy link
Contributor

@claudiubelu - can you add a release note? This can change behaviors for users.

@marosset
Copy link
Contributor

/triage accepted
/release-note needed

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 22, 2022
@marosset
Copy link
Contributor

/release-note-needed

@marosset
Copy link
Contributor

/remove-label release-note-none
/label release-note-needed

@k8s-ci-robot
Copy link
Contributor

@marosset: The label(s) /label release-note-needed, /remove-label release-note-none cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, official-cve-feed

In response to this:

/remove-label release-note-none
/label release-note-needed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@marosset
Copy link
Contributor

/shrug

@k8s-ci-robot k8s-ci-robot added the ¯\_(ツ)_/¯ ¯\\\_(ツ)_/¯ label Sep 22, 2022
@HeshamAboElMagd
Copy link

Hello @saad-ali , @claudiubelu . Release bug triage team here, just wanted to touch base with you for updates on this ticket.
Thanks!

@marosset
Copy link
Contributor

marosset commented Oct 6, 2022

/assign @xing-yang @msau42

@k8s-ci-robot k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Oct 11, 2022
@k8s-ci-robot k8s-ci-robot removed the release-note-none Denotes a PR that doesn't merit a release note. label Oct 11, 2022
// grouped. For example, the following strings are sorted in this exact order:
// /dir/nested, /dir/nested-vol, /dir/nested.vol, /dir/nested/double, /dir/nested2
// The issue is a bit worse for Windows paths, since the \'s value is higher than /'s:
// \dir\nested, \dir\nested-vol, \dir\nested.vol \dir\nested\double, \dir\nested2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add unit tests for he windows paths?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It won't matter, since the mount paths are cleaned on L39 (cleaned := filepath.Clean(vol.MountPath)), which means that all the /s will be replaced by \.

// The previously found nested mountpoint (or "" if none found yet)
prevNestedMP := ""
// The previously found nested mountpoints.
// NOTE: We can't simply rely on sort.Strings to have all the mountpoints sorted and
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is we still need to rely on sorting to ensure that nested directories do come at some point after their parent directories right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're still sorted (L45), but now we're checking at L71-77 if that directory was seen already.

For example, let's say we have \dir\nested\foo, \dir\nested2, \dir\nested. This will be sorted to \dir\nested, \dir\nested2, \dir\nested\foo. Being the first one, \dir\nested is added to retval. The second one, doesn't match any previous mountpoint (\dir\nested2 doesn't have the \dir\nested\ prefix), it is then added to retval. FInally, \dir\nested\foo has the prefix \dir\nested\, so it's skipped.

The sort is still useful, it still makes sure that \dir\nested\foo is sorted after \dir\nested, which will mean that \dir\nested\foo will be treated as an already seen mountpoint.

// NOTE: We can't simply rely on sort.Strings to have all the mountpoints sorted and
// grouped. For example, the following strings are sorted in this exact order:
// /dir/nested, /dir/nested-vol, /dir/nested.vol, /dir/nested/double, /dir/nested2
// The issue is a bit worse for Windows paths, since the \'s value is higher than /'s:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an example where Windows paths have a different ordering than Linux? So far this example looks the same.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, the ordering I've written here for Windows is wrong, it was supposed to be: \dir\nested, \dir\nested-vol, \dir\nested.vol, \dir\nested2, \dir\nested\double (2 has a smaller ascii value than \)

// The issue is a bit worse for Windows paths, since the \'s value is higher than /'s:
// \dir\nested, \dir\nested-vol, \dir\nested.vol \dir\nested\double, \dir\nested2
// Because of this, we should use a list of previously mounted mountpoints, rather than only one.
prevNestedMPs := []string{}
// examine each mount point to see if it's nested beneath this volume
// (but skip any that are double-nested beneath this volume)
// For example, if this volume is mounted as /dir and other volumes are mounted
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joelsmith @jsafrane @sjenning do you remember why we have this special case for double nested where we don't use the full path?

I thought it could have been related to supporting mounting a configmap/secret as file, but it doesn't seem like that's supported: #61545

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dug into this, we really need only 1 level of nested mounts here. It's not obvious from the code, I had to create a pod with three Secrets mounted into each other:

apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: Secret
  data:
    secret: bXkgc2VjcmV0Cg==
  metadata:
    name: first
  type: Opaque
- apiVersion: v1
  kind: Secret
  data:
    secret: c2VjcmV0Cg==
  metadata:
    name: second
  type: Opaque
- apiVersion: v1
  kind: Secret
  data:
    secret: c2VjcmV0Cg==
  metadata:
    name: third
  type: Opaque
- apiVersion: v1
  kind: Pod
  metadata:
    name: busybox-pod
  spec:
    containers:
    - image: gcr.io/google_containers/busybox:latest
      name: busybox
      command: ["/bin/sh"]
      args: ["-c", "while sleep 3600; do true; done"]
      volumeMounts:
      - mountPath: /first
        name: first 
      - mountPath: /first/second
        name: second
      - mountPath: /first/second/third
        name: third
    restartPolicy: OnFailure
    volumes:
    - name: first
      secret:
        secretName: first
    - name: second
      secret:
        secretName: second
    - name: third
      secret:
        secretName: third

On the host, /var/lib/kubelet/pods/<uid>/ looks like this:

$ ls -l /var/lib/kubelet/pods/1fc8e54a-3997-4abe-b7dd-3319b433c0c4/volumes/kubernetes.io~secret/
drwxrwxrwt. 4 root root 120 Oct 18 08:35 first
drwxrwxrwt. 4 root root 120 Oct 18 08:35 second
drwxrwxrwt. 3 root root 100 Oct 18 08:35 third

$ ls -l /var/lib/kubelet/pods/1fc8e54a-3997-4abe-b7dd-3319b433c0c4/volumes/kubernetes.io~secret/first
drwxr-xr-x. 2 root root 40 Oct 18 08:35 second
lrwxrwxrwx. 1 root root 13 Oct 18 08:35 secret -> ..data/secret

$ ls -l /var/lib/kubelet/pods/1fc8e54a-3997-4abe-b7dd-3319b433c0c4/volumes/kubernetes.io~secret/second
lrwxrwxrwx. 1 root root 13 Oct 18 08:35 secret -> ..data/secret
drwxr-xr-x. 2 root root 40 Oct 18 08:35 third
  • first/ contains an empty directory second, where the container runtime will bind-mount /var/lib/kubelet/pods/1fc8e54a-3997-4abe-b7dd-3319b433c0c4/volumes/kubernetes.io~secret/second
  • second/ contains an empty directory third, where the container runtime will bind-mount /var/lib/kubelet/pods/1fc8e54a-3997-4abe-b7dd-3319b433c0c4/volumes/kubernetes.io~secret/third

Therefore, only one level of nesting is needed. Second (third, ...) level is brought from their respective pod-local volumes via bind mounts.

@@ -89,7 +89,7 @@ func TestGetNestedMountpoints(t *testing.T) {
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't related to your change, but can you add a test case for triple nested volumes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That case is already treated in the Big Pod case below.

Currently, getNestedMountpoints sorts using sort.Strings, which would
sort the following strings in this exact order:

/dir/nested, /dir/nested-vol, /dir/nested.vol, /dir/nested/double, /dir/nested2

Because of this, "nested/double" is returned as well, even though it shouldn't
have been. This issue is worse on Windows, where the path separator is typically
the backslash.

This commit addresses this issue by checking if a nested mount point has been
previously seen or not.
@msau42
Copy link
Member

msau42 commented Oct 18, 2022

/retest
/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 18, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: claudiubelu, msau42

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 18, 2022
@k8s-ci-robot k8s-ci-robot merged commit b6d89e7 into kubernetes:master Oct 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/windows Categorizes an issue or PR as relevant to SIG Windows. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on. ¯\_(ツ)_/¯ ¯\\\_(ツ)_/¯
Projects
Archived in project
Status: Done
Development

Successfully merging this pull request may close these issues.

getNestedMountpoints may not group mountpoints correctly
8 participants