dynamic resource allocation #111023

pohly · 2022-07-08T12:51:17Z

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This implements the API and control plane part (kube-scheduler, kube-controller-manager, kube-apiserver, controller side of example driver) for Dynamic Resource Allocation.

Which issue(s) this PR fixes:

Related-to: kubernetes/enhancements#3063

Special notes for your reviewer:

Let's try to gather who has reviewed what:

API, architecture: dynamic resource allocation #111023 (comment)
kubelet: dynamic resource allocation #111023 (comment)
pkg/util/dynamic-resource-allocation (aka future k8s.io/dynamic-resources): dynamic resource allocation #111023 (comment)
scheduler: dynamic resource allocation #111023 (review)
controller: dynamic resource allocation #111023 (comment), dynamic resource allocation #111023 (review)
metrics: dynamic resource allocation #111023 (review)
test/e2e/dra: dynamic resource allocation #111023 (comment)
~~dependencies: @dims, @liggitt~~ - no longer needed, dependencies were removed

This is using the revised API from kubernetes/enhancements#3502.

To try out the feature, build Kubernetes, then in one console run:

RUNTIME_CONFIG=resource.k8s.io/v1alpha1 FEATURE_GATES=DynamicResourceAllocation=true KUBELET_RESOLV_CONF="/etc/resolv-9999.conf" DNS_ADDON="coredns" CGROUP_DRIVER=systemd CONTAINER_RUNTIME_ENDPOINT=unix:///var/run/crio/crio.sock LOG_LEVEL=6 ENABLE_CSI_SNAPSHOTTER=false API_SECURE_PORT=6444 ALLOW_PRIVILEGED=1 PATH=/nvme/gopath/src/k8s.io/kubernetes/third_party/etcd:$PATH ./hack/local-up-cluster.sh -O

In another:

go run ./test/e2e/dra/test-driver --feature-gates ContextualLogging=true -v=5 controller

In yet another:

sudo mkdir -p /var/run/cdi && sudo chmod a+rwx /var/run/cdi /var/lib/kubelet/plugins_registry /var/lib/kubelet/plugins/
go run ./test/e2e/dra/test-driver --feature-gates ContextualLogging=true -v=6 kubelet-plugin

And finally:

$ kubectl create -f test/e2e/dra/test-driver/deploy/example/resourceclass.yaml 
resourceclass/example created
$ kubectl create -f test/e2e/dra/test-driver/deploy/example/pod-inline.yaml 
configmap/pause-claim-parameters created
pod/pause created
$ kubectl get resourceclaims
NAME             CLASSNAME   ALLOCATIONMODE         STATE                AGE
pause-resource   example     WaitForFirstConsumer   allocated,reserved   19s
pohly@pohly-desktop:/nvme/gopath/src/k8s.io/kubernetes$ kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
pause   1/1     Running   0          23s

Does this PR introduce a user-facing change?

Add a `ResourceClaim` API (in the resource.k8s.io/v1alpha1 API group and
behind the `DynamicResourceAllocation` feature gate).
The new API is more flexible than the existing Device Plugins feature of Kubernetes because it
allows Pods to request (claim) special kinds of resources, which can be available at node level, cluster
level, or following any other model you implement.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/issues/3063
- [Usage]: TODO

pkg/features/kube_features.go

k8s-ci-robot

@pohly: 4 invalid OWNERS files

In response to this:

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This implements the API and control plane part (kube-scheduler, kube-controller-manager, kube-apiserver, controller side of example driver) for Dynamic Resource Allocation.

Which issue(s) this PR fixes:

Related-to: kubernetes/enhancements#3063

Special notes for your reviewer:

This is using the revised API from pohly/enhancements#13. Before we move forward with the implementation, I would like to get confirmation from @alculquicondor and @thockin that this approach is okay and clarify a few remaining questions around the API. I'll add some of my own questions as PR comments.

We also need to decide how we want to merge this: everything in one big PR or split up along reasonable boundaries? Right now, the kubelet changes would be a good fit for a separate PR (different developer and reviewers, current functionality from this PR can be used without kubelet changes as the extended Pod spec will simply get ignored by kubelet).

Either way, the PR is not complete yet. The functionality for the control plane is all there, but tests and API validation are incomplete.

Does this PR introduce a user-facing change?
When the new `DynamicResourceAllocation` feature is enabled, additional resources can be requested for pods through the new ResourceClaim API and third-party resource drivers.
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
- [KEP]: https://github.com/kubernetes/enhancements/issues/3063
- [Usage]: TODO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

staging/src/k8s.io/component-helpers/cdi/OWNERS

test/integration/cdi/example-driver/OWNERS

pkg/controller/resourceclaim/OWNERS

pkg/scheduler/framework/plugins/dynamicresources/OWNERS

pkg/apis/core/types.go

pkg/apis/core/register.go

pkg/apis/core/types.go

k8s-ci-robot

@pohly: 2 invalid OWNERS files

In response to this:

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This implements the API and control plane part (kube-scheduler, kube-controller-manager, kube-apiserver, controller side of example driver) for Dynamic Resource Allocation.

Which issue(s) this PR fixes:

Related-to: kubernetes/enhancements#3063

Special notes for your reviewer:

This is using the revised API from pohly/enhancements#13. Before we move forward with the implementation, I would like to get confirmation from @alculquicondor and @thockin that this approach is okay and clarify a few remaining questions around the API. I'll add some of my own questions as PR comments.

We also need to decide how we want to merge this: everything in one big PR or split up along reasonable boundaries? Right now, the kubelet changes would be a good fit for a separate PR (different developer and reviewers, current functionality from this PR can be used without kubelet changes as the extended Pod spec will simply get ignored by kubelet).

Either way, the PR is not complete yet. The functionality for the control plane is all there, but tests and API validation are incomplete.

Does this PR introduce a user-facing change?
When the new `DynamicResourceAllocation` feature is enabled, additional resources can be requested for pods through the new ResourceClaim API and third-party resource drivers.
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
- [KEP]: https://github.com/kubernetes/enhancements/issues/3063
- [Usage]: TODO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/controller/resourceclaim/OWNERS

pkg/scheduler/framework/plugins/dynamicresources/OWNERS

k8s-ci-robot · 2022-11-11T23:22:17Z

@pohly: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-node-crio-dra	`7f7749b`	link	false	`/test pull-kubernetes-node-crio-dra`
pull-publishing-bot-validate	`7b6e911`	link	false	`/test pull-publishing-bot-validate`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

BenTheElder · 2022-11-11T23:42:05Z

The publishing bot tests want the package to exist in the merged branch before adding rules to publish it.

dims · 2022-11-11T23:55:04Z

/skip pull-publishing-bot-validate

NOTE: this is by design, this CI job will fail in this PR as this PR is adding the new staging repo.

BenTheElder · 2022-11-12T00:02:39Z

/skip
(doesn't take arguments / doesn't work with arguments, skips all failing non-required-but-reporting-status jobs)
https://prow.k8s.io/command-help

No particular benefit and no relevant changes, it's just to stay up-to-date and to avoid having to pull that in when merging kubernetes#111023 which indirectly depends on the newer release.

liggitt · 2023-02-16T19:05:36Z

staging/src/k8s.io/api/core/v1/types.go

+	// +listType=set
+	// +featureGate=DynamicResourceAllocation
+	// +optional
+	Claims []ResourceClaim `json:"claims,omitempty" protobuf:"bytes,3,opt,name=claims"`


We just ran across this while reviewing #107776

The ResourceRequirements type is also used within the PersistentVolumeSpec type (which is also used indirectly in pod specs via ephemeral volumes), and the "clear if feature-gate is disabled" logic is only wired to podspec-handling. That means in 1.26 PVC objects or PodSpecs with ephemeral volumes defined can persist data in this field even if the feature is disabled.

Can you open a PR to clear this field in PVC and ephemeral volume create/update flows, and file a follow-up issue to consider splitting the type used in PVCs so it doesn't continue inheriting work done that is targeted at pods?

81981266 · 2023-12-22T04:34:01Z

Have we decided whether to implement Topology support for DRA or not? @pohly

pohly · 2024-01-03T15:57:56Z

Have we decided whether to implement Topology support for DRA or not?

With topology you mean sub-node topologuy like NUMA? This is out-of-scope for this KEP. It operates at the same level as current scheduler, which is nodes.

However, there are some aspects of DRA that can be used to implement a controller which makes holistic decisions about all pending claims needed by a pod. See @klueska's presentations at KubeCon and the Slack discussions.