New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixes getNestedMountpoints grouping #112571
Fixes getNestedMountpoints grouping #112571
Conversation
/test pull-kubernetes-e2e-kind |
/assign @saad-ali |
/milestone v1.26 |
@claudiubelu - can you add a release note? This can change behaviors for users. |
/triage accepted |
/release-note-needed |
/remove-label release-note-none |
@marosset: The label(s) In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/shrug |
Hello @saad-ali , @claudiubelu . Release bug triage team here, just wanted to touch base with you for updates on this ticket. |
/assign @xing-yang @msau42 |
pkg/volume/util/nested_volumes.go
Outdated
// grouped. For example, the following strings are sorted in this exact order: | ||
// /dir/nested, /dir/nested-vol, /dir/nested.vol, /dir/nested/double, /dir/nested2 | ||
// The issue is a bit worse for Windows paths, since the \'s value is higher than /'s: | ||
// \dir\nested, \dir\nested-vol, \dir\nested.vol \dir\nested\double, \dir\nested2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add unit tests for he windows paths?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It won't matter, since the mount paths are cleaned on L39 (cleaned := filepath.Clean(vol.MountPath)
), which means that all the /
s will be replaced by \
.
// The previously found nested mountpoint (or "" if none found yet) | ||
prevNestedMP := "" | ||
// The previously found nested mountpoints. | ||
// NOTE: We can't simply rely on sort.Strings to have all the mountpoints sorted and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My understanding is we still need to rely on sorting to ensure that nested directories do come at some point after their parent directories right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They're still sorted (L45), but now we're checking at L71-77 if that directory was seen already.
For example, let's say we have \dir\nested\foo
, \dir\nested2
, \dir\nested
. This will be sorted to \dir\nested
, \dir\nested2
, \dir\nested\foo
. Being the first one, \dir\nested
is added to retval
. The second one, doesn't match any previous mountpoint (\dir\nested2
doesn't have the \dir\nested\
prefix), it is then added to retval
. FInally, \dir\nested\foo
has the prefix \dir\nested\
, so it's skipped.
The sort is still useful, it still makes sure that \dir\nested\foo
is sorted after \dir\nested
, which will mean that \dir\nested\foo
will be treated as an already seen mountpoint.
// NOTE: We can't simply rely on sort.Strings to have all the mountpoints sorted and | ||
// grouped. For example, the following strings are sorted in this exact order: | ||
// /dir/nested, /dir/nested-vol, /dir/nested.vol, /dir/nested/double, /dir/nested2 | ||
// The issue is a bit worse for Windows paths, since the \'s value is higher than /'s: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there an example where Windows paths have a different ordering than Linux? So far this example looks the same.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed, the ordering I've written here for Windows is wrong, it was supposed to be: \dir\nested, \dir\nested-vol, \dir\nested.vol, \dir\nested2, \dir\nested\double
(2
has a smaller ascii value than \
)
// The issue is a bit worse for Windows paths, since the \'s value is higher than /'s: | ||
// \dir\nested, \dir\nested-vol, \dir\nested.vol \dir\nested\double, \dir\nested2 | ||
// Because of this, we should use a list of previously mounted mountpoints, rather than only one. | ||
prevNestedMPs := []string{} | ||
// examine each mount point to see if it's nested beneath this volume | ||
// (but skip any that are double-nested beneath this volume) | ||
// For example, if this volume is mounted as /dir and other volumes are mounted |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@joelsmith @jsafrane @sjenning do you remember why we have this special case for double nested where we don't use the full path?
I thought it could have been related to supporting mounting a configmap/secret as file, but it doesn't seem like that's supported: #61545
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dug into this, we really need only 1 level of nested mounts here. It's not obvious from the code, I had to create a pod with three Secrets mounted into each other:
apiVersion: v1
kind: List
items:
- apiVersion: v1
kind: Secret
data:
secret: bXkgc2VjcmV0Cg==
metadata:
name: first
type: Opaque
- apiVersion: v1
kind: Secret
data:
secret: c2VjcmV0Cg==
metadata:
name: second
type: Opaque
- apiVersion: v1
kind: Secret
data:
secret: c2VjcmV0Cg==
metadata:
name: third
type: Opaque
- apiVersion: v1
kind: Pod
metadata:
name: busybox-pod
spec:
containers:
- image: gcr.io/google_containers/busybox:latest
name: busybox
command: ["/bin/sh"]
args: ["-c", "while sleep 3600; do true; done"]
volumeMounts:
- mountPath: /first
name: first
- mountPath: /first/second
name: second
- mountPath: /first/second/third
name: third
restartPolicy: OnFailure
volumes:
- name: first
secret:
secretName: first
- name: second
secret:
secretName: second
- name: third
secret:
secretName: third
On the host, /var/lib/kubelet/pods/<uid>/
looks like this:
$ ls -l /var/lib/kubelet/pods/1fc8e54a-3997-4abe-b7dd-3319b433c0c4/volumes/kubernetes.io~secret/
drwxrwxrwt. 4 root root 120 Oct 18 08:35 first
drwxrwxrwt. 4 root root 120 Oct 18 08:35 second
drwxrwxrwt. 3 root root 100 Oct 18 08:35 third
$ ls -l /var/lib/kubelet/pods/1fc8e54a-3997-4abe-b7dd-3319b433c0c4/volumes/kubernetes.io~secret/first
drwxr-xr-x. 2 root root 40 Oct 18 08:35 second
lrwxrwxrwx. 1 root root 13 Oct 18 08:35 secret -> ..data/secret
$ ls -l /var/lib/kubelet/pods/1fc8e54a-3997-4abe-b7dd-3319b433c0c4/volumes/kubernetes.io~secret/second
lrwxrwxrwx. 1 root root 13 Oct 18 08:35 secret -> ..data/secret
drwxr-xr-x. 2 root root 40 Oct 18 08:35 third
first/
contains an empty directorysecond
, where the container runtime will bind-mount/var/lib/kubelet/pods/1fc8e54a-3997-4abe-b7dd-3319b433c0c4/volumes/kubernetes.io~secret/second
second/
contains an empty directorythird
, where the container runtime will bind-mount/var/lib/kubelet/pods/1fc8e54a-3997-4abe-b7dd-3319b433c0c4/volumes/kubernetes.io~secret/third
Therefore, only one level of nesting is needed. Second (third, ...) level is brought from their respective pod-local volumes via bind mounts.
@@ -89,7 +89,7 @@ func TestGetNestedMountpoints(t *testing.T) { | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isn't related to your change, but can you add a test case for triple nested volumes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That case is already treated in the Big Pod
case below.
a79e719
to
6e9f428
Compare
Currently, getNestedMountpoints sorts using sort.Strings, which would sort the following strings in this exact order: /dir/nested, /dir/nested-vol, /dir/nested.vol, /dir/nested/double, /dir/nested2 Because of this, "nested/double" is returned as well, even though it shouldn't have been. This issue is worse on Windows, where the path separator is typically the backslash. This commit addresses this issue by checking if a nested mount point has been previously seen or not.
/retest |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: claudiubelu, msau42 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
What type of PR is this?
/kind bug
/sig windows
/priority important-soon
What this PR does / why we need it:
Currently,
getNestedMountpoints
sorts usingsort.Strings
, which would sort the following strings in this exact order:Because of this,
nested/double
is returned as well, even though it shouldn't have been. This issue is worse on Windows, where the path separator is typically the backslash.This commit addresses this issue by checking if a nested mount point has been previously seen or not.
Which issue(s) this PR fixes:
Fixes #112570
Related: #51540
Special notes for your reviewer:
Does this PR introduce a user-facing change?
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: