Minimal checkpointing support #104907

adrianreber · 2021-09-10T13:55:51Z

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This PR introduces the minimal checkpoint support as discussed in kubernetes/enhancements#1990

With the changes in the PR it is only possible to checkpoint a container. One of the main use cases, as discussed in the KEP of this PR is the forensic use case: Checkpoint a container and leave it running, take the checkpoint image and restart the container somewhere else (sandboxed environment) for analysis.

Possible steps could be:

Checkpoint container out of Kubernetes: curl -skv -X POST "https://localhost:10250/checkpoint/default/counters/wildfly"
Take the checkpoint archive from /var/lib/kubelet/checkpoints and transfer it to another system
Use crictl restore --import=<archive> to restore a copy of the container somewhere else.

In the KEP we mentioned that checkpoints will be stored in OCI images and can be pushed to a registry. This is still planned but currently not implemented on the container engine level. It will be implemented in the future.

Which issue(s) this PR fixes:

This does not really completely fix any open issues, but it is a step forward to close issues like:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Added a new feature gate `CheckpointRestore` to enable support to checkpoint containers. If enabled it is possible to checkpoint a container using the newly kubelet API (/checkpoint/{podNamespace}/{podName}/{containerName}).

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2021-09-10T13:56:00Z

Hi @adrianreber. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-triage-robot · 2021-09-10T14:41:23Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

ehashman · 2021-09-14T21:24:40Z

This needs to be implemented behind a feature flag.

/ok-to-test

adrianreber · 2021-09-17T12:07:10Z

/test pull-kubernetes-unit

adrianreber · 2021-09-20T15:55:45Z

Added unit tests and a feature flag (disabled by default).

derekwaynecarr

@adrianreber this is super close.

can you address the following:

ensure checkpoint directory is only created if the feature gate is enabled
ensure i can checkpoint an init container and/or ephemeral container

Thanks!

derekwaynecarr · 2022-07-13T18:47:51Z

pkg/kubelet/cri/remote/remote_runtime.go

+
+	ctx, cancel := func() (context.Context, context.CancelFunc) {
+		defaultTimeout := int64(r.timeout / time.Second)
+		if options.Timeout > defaultTimeout {


what could cause this timeout to need to vary?

Checkpoint duration directly depends on the number of memory pages the processes in the container require. For very large containers checkpointing can take longer than the default CRI timeout of 2 seconds.

derekwaynecarr · 2022-07-13T18:48:41Z

pkg/kubelet/kubelet.go

@@ -1251,6 +1254,9 @@ func (kl *Kubelet) setupDataDirs() error {
 	if err := os.MkdirAll(kl.getPodResourcesDir(), 0750); err != nil {
 		return fmt.Errorf("error creating podresources directory: %v", err)
 	}
+	if err := os.MkdirAll(kl.getCheckpointsDir(), 0750); err != nil {


we should only create the checkpoint directory if the feature gate is enabled.

Good point. Changed.

derekwaynecarr · 2022-07-13T18:53:33Z

pkg/kubelet/server/server.go

+// checkpoint handles the checkpoint API request. It checks if the requested
+// podNamespace, pod and container actually exist and only then calls out
+// to the runtime to actually checkpoint the container.
+func (s *Server) checkpoint(request *restful.Request, response *restful.Response) {


initially we want to avoid exposing checkpoint operations from the k8s control plane itself.

we need to better understand the pros/cons from a forensic use case.

derekwaynecarr · 2022-07-13T18:55:28Z

pkg/kubelet/server/server.go

+	containerName := request.PathParameter("containerName")
+
+	found := false
+	for _, container := range pod.Spec.Containers {


this means we are unable to checkpoint init containers or ephemeral containers.

can you add a check for pod.Spec.InitContainers and a check for pod.Spec.EphemeralContainers if the feature gate for ephemeral containers is enabled?

Thanks for catching this. I changed it and was able to checkpoint init and ephemeral containers.

This adds the minimal CRI API changes to support checkpoint/restore and to enable implementation of these checkpoint/restore interfaces on container engine level. Signed-off-by: Adrian Reber <areber@redhat.com>

Signed-off-by: Adrian Reber <areber@redhat.com>

This adds the first infrastructure code parts to the kubelet to support checkpoint/restore. Signed-off-by: Adrian Reber <areber@redhat.com>

This is the first step to implement checkpointing and restoring of container and containers starting from the lowest layer in the kubelet. Signed-off-by: Adrian Reber <areber@redhat.com>

Signed-off-by: Adrian Reber <areber@redhat.com>

This adds the last pieces to wire through the container checkpoint support in the kubelet. Signed-off-by: Adrian Reber <areber@redhat.com>

Signed-off-by: Adrian Reber <areber@redhat.com>

adrianreber · 2022-07-14T10:32:52Z

@adrianreber this is super close.

🎆

can you address the following:

1. ensure checkpoint directory is only created if the feature gate is enabled

Done. Good point.

2. ensure i can checkpoint an init container and/or ephemeral container

Also done. Thanks for your very specific review. That made it really easy to implement the requested changes.

derekwaynecarr · 2022-07-14T13:03:46Z

Thanks for the updates as requested.

/approve
/lgtm

k8s-ci-robot · 2022-07-14T13:04:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adrianreber, derekwaynecarr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/features/OWNERS~~ [derekwaynecarr]
~~pkg/kubelet/OWNERS~~ [derekwaynecarr]
~~staging/src/k8s.io/cri-api/pkg/OWNERS~~ [derekwaynecarr]
~~test/e2e_node/OWNERS~~ [derekwaynecarr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

adrianreber · 2022-07-14T13:27:07Z

/test pull-kubernetes-e2e-kind-ipv6
/test pull-kubernetes-e2e-kind

adrianreber · 2022-07-14T14:35:56Z

/test pull-kubernetes-e2e-kind

adrianreber · 2022-07-14T14:49:41Z

/test pull-kubernetes-e2e-gce-ubuntu-containerd

adrianreber · 2022-07-14T15:12:30Z

/test pull-kubernetes-e2e-gce-ubuntu-containerd

k8s-ci-robot · 2022-07-14T16:41:43Z

@adrianreber: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-node-e2e-containerd-features	dcd59805bf4134f6de7ef08dcf1bd75f1deeb3fd	link	false	`/test pull-kubernetes-node-e2e-containerd-features`
pull-kubernetes-node-e2e-containerd-alpha-features	0b1cc8ae5d242ecfe9f0b03c843a9bc07d755ef9	link	false	`/test pull-kubernetes-node-e2e-containerd-alpha-features`
pull-kubernetes-e2e-gce-ubuntu-containerd	`92ea6e3`	link	true	`/test pull-kubernetes-e2e-gce-ubuntu-containerd`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

adrianreber · 2022-07-14T16:44:08Z

/test pull-kubernetes-e2e-gce-ubuntu-containerd

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 10, 2021

k8s-ci-robot requested review from mrunalp and mtaufen September 10, 2021 13:58

adrianreber changed the title ~~Minimal checkpoint~~ Minimal checkpointing support Sep 10, 2021

ehashman added this to Triage in SIG Node PR Triage Sep 10, 2021

mrunalp mentioned this pull request Sep 14, 2021

Add Forensic Container Checkpointing KEP kubernetes/enhancements#1990

Merged

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 14, 2021

adrianreber force-pushed the 2021-09-10-checkpoint branch 2 times, most recently from c4eb187 to 71f1bdb Compare September 17, 2021 10:10

adrianreber force-pushed the 2021-09-10-checkpoint branch 4 times, most recently from 42e0bba to 948556e Compare September 20, 2021 15:55

derekwaynecarr requested changes Jul 13, 2022

View reviewed changes

adrianreber added 8 commits July 14, 2022 10:27

Add CRI API definitions for checkpoint/restore

df4ca2d

This adds the minimal CRI API changes to support checkpoint/restore and to enable implementation of these checkpoint/restore interfaces on container engine level. Signed-off-by: Adrian Reber <areber@redhat.com>

make update after CRI API changes (checkpoint/restore)

eeb5923

Signed-off-by: Adrian Reber <areber@redhat.com>

kubelet: add checkpoint/restore infrastructure

564f0e9

This adds the first infrastructure code parts to the kubelet to support checkpoint/restore. Signed-off-by: Adrian Reber <areber@redhat.com>

kubelet: add CheckpointContainer in remote runtime

1ac7d78

This is the first step to implement checkpointing and restoring of container and containers starting from the lowest layer in the kubelet. Signed-off-by: Adrian Reber <areber@redhat.com>

kubelet: add CheckpointContainer() on the service level

3e6f506

Signed-off-by: Adrian Reber <areber@redhat.com>

kubelet: add CheckpointContainer() to the runtime

8c24857

Signed-off-by: Adrian Reber <areber@redhat.com>

kubelet: wire checkpoint container support through

fc37a7a

This adds the last pieces to wire through the container checkpoint support in the kubelet. Signed-off-by: Adrian Reber <areber@redhat.com>

Add e2e tests for checkpointing

92ea6e3

Signed-off-by: Adrian Reber <areber@redhat.com>

adrianreber force-pushed the 2021-09-10-checkpoint branch from 343c6ac to 92ea6e3 Compare July 14, 2022 10:28

adrianreber requested a review from derekwaynecarr July 14, 2022 10:31

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 14, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 14, 2022

k8s-ci-robot merged commit a655368 into kubernetes:master Jul 14, 2022

SIG Node PR Triage automation moved this from Needs Reviewer to Done Jul 14, 2022

pacoxu mentioned this pull request Aug 26, 2022

Add kubernetes v1.25 supported version as v1.24 containerd/containerd#7326

Merged

sapphire006 mentioned this pull request Sep 4, 2022

Doubt - Relation with Kubernetes checkpoint-restore/checkpointctl#25

Closed

This was referenced Sep 9, 2022

Feature gate ContainerCheckpoint misnamed in docs kubernetes/website#36705

Closed

correct CheckpointContainer to ContainerCheckpoint kubernetes/website#36699

Merged

deads2k mentioned this pull request Feb 6, 2024

KEP-2008: Graduate "Forensic Container Checkpointing" to Beta kubernetes/enhancements#4288

Merged

2 tasks

yeazelm mentioned this pull request Feb 29, 2024

Checkpoint/Restart or Live Motion bottlerocket-os/bottlerocket#3803

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minimal checkpointing support #104907

Minimal checkpointing support #104907

adrianreber commented Sep 10, 2021 •

edited

k8s-ci-robot commented Sep 10, 2021

k8s-triage-robot commented Sep 10, 2021

ehashman commented Sep 14, 2021

adrianreber commented Sep 17, 2021

adrianreber commented Sep 20, 2021

derekwaynecarr left a comment

derekwaynecarr Jul 13, 2022

adrianreber Jul 14, 2022

derekwaynecarr Jul 13, 2022

adrianreber Jul 14, 2022

derekwaynecarr Jul 13, 2022

derekwaynecarr Jul 13, 2022

adrianreber Jul 14, 2022

adrianreber commented Jul 14, 2022

derekwaynecarr commented Jul 14, 2022

k8s-ci-robot commented Jul 14, 2022

adrianreber commented Jul 14, 2022

adrianreber commented Jul 14, 2022

adrianreber commented Jul 14, 2022

adrianreber commented Jul 14, 2022

k8s-ci-robot commented Jul 14, 2022

adrianreber commented Jul 14, 2022

Minimal checkpointing support #104907

Minimal checkpointing support #104907

Conversation

adrianreber commented Sep 10, 2021 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Sep 10, 2021

k8s-triage-robot commented Sep 10, 2021

ehashman commented Sep 14, 2021

adrianreber commented Sep 17, 2021

adrianreber commented Sep 20, 2021

derekwaynecarr left a comment

Choose a reason for hiding this comment

derekwaynecarr Jul 13, 2022

Choose a reason for hiding this comment

adrianreber Jul 14, 2022

Choose a reason for hiding this comment

derekwaynecarr Jul 13, 2022

Choose a reason for hiding this comment

adrianreber Jul 14, 2022

Choose a reason for hiding this comment

derekwaynecarr Jul 13, 2022

Choose a reason for hiding this comment

derekwaynecarr Jul 13, 2022

Choose a reason for hiding this comment

adrianreber Jul 14, 2022

Choose a reason for hiding this comment

adrianreber commented Jul 14, 2022

derekwaynecarr commented Jul 14, 2022

k8s-ci-robot commented Jul 14, 2022

adrianreber commented Jul 14, 2022

adrianreber commented Jul 14, 2022

adrianreber commented Jul 14, 2022

adrianreber commented Jul 14, 2022

k8s-ci-robot commented Jul 14, 2022

adrianreber commented Jul 14, 2022

adrianreber commented Sep 10, 2021 •

edited