[KEP-3521] Part 1: New Pod API .spec.schedulingGates #113274

Huang-Wei · 2022-10-22T01:47:45Z

What type of PR is this?

/kind feature
/sig scheduling

What this PR does / why we need it:

This is the first part of implementing KEP 3521, including:

Pod API definition, feature gate, codegen, validation, and drop disabled fields
disallow binding a Pod carrying non-nil schedulingGates
disallow creating a Pod with non-nil nodeName and non-nil schedulingGates
if needed, adds a {type:PodScheduled, reason:SchedulingGated} condition upon pod creation
new literal SchedulingGated in the STATUS column of k get pod

Which issue(s) this PR fixes:

Part of #113269

Special notes for your reviewer:

Get best review UX commit by commit.

Does this PR introduce a user-facing change?

New Pod API field `.spec.schedulingGates` is introduced to enable users to control when to mark a Pod as scheduling ready.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/3521-pod-scheduling-readiness

k8s-triage-robot · 2022-10-22T01:59:28Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

Huang-Wei · 2022-10-22T15:34:08Z

cc @ahg-g

There are 2 PRs upcoming: scheduler implementation and integration/e2e tests.

pkg/apis/core/types.go

ahg-g · 2022-10-27T01:03:46Z

pkg/apis/core/types.go

@@ -3017,6 +3030,13 @@ type PodOS struct {
 	Name OSName
 }

+// PodSchedulingGate is associated to a Pod to guard its scheduling.
+type PodSchedulingGate struct {


do we need a struct?

Yeah, I asked for it for future proofing.

pkg/apis/core/validation/validation.go

pkg/registry/core/pod/strategy.go

ahg-g · 2022-10-27T01:24:51Z

pkg/printers/internalversion/printers.go

@@ -81,6 +81,9 @@ const (

 	// nodeLabelRole specifies the role of a node
 	nodeLabelRole = "kubernetes.io/role"
+
+	// schedulingPaused indicates a Pod is gated by one or more scheduling readiness gates.


I would use blocked instead of pause.

Agree. Or SchedulingGated (it should be rare enough that calling it gated will help people find the field.

Updated to schedulingGated which is more close to the new field name.

ahg-g · 2022-10-27T01:25:24Z

pkg/printers/internalversion/printers.go

+	// If the Pod carries {type:PodScheduled, reason:WaitingForGates}, set reason to SchedulingPaused.
+	for _, condition := range pod.Status.Conditions {
+		if condition.Type == api.PodScheduled && condition.Reason == api.PodReasonWaitingForGates {
+			reason = schedulingPaused


why not use the same reason?

It's b/c this "reason" is used in the output of k get pod, and thusPodReasonWaitingForGates which holds the literal "WaitingForGates" loses the context - a word describing that it's scheduling gated would be more self-descriptive.

ahg-g · 2022-10-27T01:25:56Z

/assign @smarterclayton

ahg-g · 2022-10-27T02:16:32Z

/label api-review

smarterclayton · 2022-10-27T02:18:17Z

pkg/registry/core/pod/storage/storage.go

@@ -221,6 +223,10 @@ func (r *BindingREST) setPodHostAndAnnotations(ctx context.Context, podUID types
 		if pod.Spec.NodeName != "" {
 			return nil, fmt.Errorf("pod %v is already assigned to node %q", pod.Name, pod.Spec.NodeName)
 		}
+		// Reject binding to a scheduling un-ready Pod.
+		if utilfeature.DefaultFeatureGate.Enabled(features.PodSchedulingReadiness) && len(pod.Spec.SchedulingGates) != 0 {


This should really be in validation - a regular user can patch a pod and set spec.nodeName without binding.

ah, update's validation has already been covered. Here it's an additional validation for pod creation - prevent user creating a pod directly with nodeName and non-empty scheduling gates.

But here pod.Spec.NodeName won't be empty, we returned in the above. I think we should move this ahead.

not quite. L223~L225 is checking whether it's updating nodeName X to Y.

prevent user creating a pod directly with nodeName and non-empty scheduling gates

I see, misunderstood the words here. Great catch.

pkg/apis/core/types.go

Huang-Wei · 2022-11-03T20:54:37Z

If you can change the validation string i'll approve - everything else can be improved before beta.

Thanks. Tracked in #113608

smarterclayton · 2022-11-03T21:18:19Z

/approve

k8s-ci-robot · 2022-11-03T21:18:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~api/OWNERS~~ [smarterclayton]
~~pkg/api/OWNERS~~ [smarterclayton]
~~pkg/apis/OWNERS~~ [smarterclayton]
~~pkg/features/OWNERS~~ [smarterclayton]
~~pkg/generated/openapi/OWNERS~~ [smarterclayton]
~~pkg/printers/OWNERS~~ [smarterclayton]
~~pkg/registry/OWNERS~~ [smarterclayton]
~~staging/src/k8s.io/api/OWNERS~~ [smarterclayton]
~~staging/src/k8s.io/client-go/OWNERS~~ [smarterclayton]
~~test/e2e/framework/OWNERS~~ [smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

- New API field .spec.schedulingGates - Validation and drop disabled fields - Disallow binding a Pod carrying non-nil schedulingGates - Disallow creating a Pod with non-nil nodeName and non-nil schedulingGates - Adds a {type:PodScheduled, reason:WaitingForGates} condition if necessary - New literal SchedulingGated in the STATUS column of `k get pod`

ahg-g · 2022-11-03T21:54:19Z

/lgtm

Huang-Wei · 2022-11-03T21:55:15Z

/hold
for CI.

Huang-Wei · 2022-11-04T03:18:20Z

/unhold

ahg-g · 2022-11-04T04:07:53Z

/hold cancel

soltysh · 2022-11-04T12:36:20Z

@soltysh would you (or delegate to someone) please review the CLI changes? In particular, the status volume of k get pod will output "SchedulingGated" if the pod carries non-empty .spec.schedulingGates.

A bit late to the game, but this looks great - thx!

fedebongio · 2022-11-08T21:09:30Z

/remove-sig api-machinery

fedebongio · 2022-11-08T21:09:37Z

/triage accepted

brandond · 2022-11-11T00:42:48Z

@Huang-Wei @smarterclayton there is a typo the PR comment that made it into the CHANGELOG.

From the user-facing change section (that is scraped by the changelog generator), .spec.scheudlingGates should be .spec.schedulingGates

Huang-Wei · 2022-11-11T06:34:31Z

@brandond thanks for the catch.. Rasied a PR fixing the typos: #113843

k8s-ci-robot requested review from aojea and caesarxuchao October 22, 2022 01:48

Huang-Wei force-pushed the kep-3521-A branch 2 times, most recently from f3b90e4 to d14305f Compare October 22, 2022 05:22

k8s-ci-robot added area/e2e-test-framework Issues or PRs related to refactoring the kubernetes e2e test framework area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Oct 22, 2022

Huang-Wei mentioned this pull request Oct 22, 2022

[KEP-3521] Part 2: Core scheduling implementation #113275

Merged

5 tasks

ahg-g reviewed Oct 27, 2022

View reviewed changes

k8s-ci-robot assigned smarterclayton Oct 27, 2022

k8s-ci-robot added the api-review Categorizes an issue or PR as actively needing an API review. label Oct 27, 2022

smarterclayton reviewed Oct 27, 2022

View reviewed changes

pkg/apis/core/types.go Outdated Show resolved Hide resolved

smarterclayton reviewed Oct 27, 2022

View reviewed changes

pkg/apis/core/types.go Outdated Show resolved Hide resolved

smarterclayton reviewed Oct 27, 2022

View reviewed changes

pkg/apis/core/types.go Outdated Show resolved Hide resolved

k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Nov 3, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 3, 2022

Huang-Wei added 2 commits November 3, 2022 14:32

Automated codegen

fc831d7

Huang-Wei force-pushed the kep-3521-A branch from 6d1cba1 to fc831d7 Compare November 3, 2022 21:32

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 3, 2022

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 3, 2022

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 4, 2022

k8s-ci-robot merged commit 8c77820 into kubernetes:master Nov 4, 2022

Huang-Wei deleted the kep-3521-A branch November 4, 2022 05:06

k8s-ci-robot removed the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Nov 8, 2022

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 8, 2022

howardjohn mentioned this pull request Dec 9, 2022

CNI race condition alternative design istio/istio#40303

Open

matglas mentioned this pull request Dec 15, 2022

Mark nodes as "NotReady" until critical pods from daemonsets are Ready #75890

Closed

Huang-Wei mentioned this pull request Feb 15, 2023

Graduate PodSchedulingReadiness to beta #115815

Merged

lianghao208 mentioned this pull request Feb 16, 2023

Validating the scheduling gate name qualified #115821

Merged

liggitt moved this from Assigned to API review completed, 1.26 in API Reviews Aug 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KEP-3521] Part 1: New Pod API .spec.schedulingGates #113274

[KEP-3521] Part 1: New Pod API .spec.schedulingGates #113274

Huang-Wei commented Oct 22, 2022 •

edited

k8s-triage-robot commented Oct 22, 2022

Huang-Wei commented Oct 22, 2022

ahg-g Oct 27, 2022

smarterclayton Oct 27, 2022

ahg-g Oct 27, 2022

smarterclayton Oct 27, 2022

Huang-Wei Oct 27, 2022

ahg-g Oct 27, 2022

Huang-Wei Oct 27, 2022

ahg-g commented Oct 27, 2022

ahg-g commented Oct 27, 2022

smarterclayton Oct 27, 2022

Huang-Wei Oct 27, 2022

kerthcet Oct 31, 2022 •

edited

Huang-Wei Oct 31, 2022

kerthcet Nov 1, 2022

Huang-Wei commented Nov 3, 2022

smarterclayton commented Nov 3, 2022

k8s-ci-robot commented Nov 3, 2022

ahg-g commented Nov 3, 2022

Huang-Wei commented Nov 3, 2022

Huang-Wei commented Nov 4, 2022

ahg-g commented Nov 4, 2022

soltysh commented Nov 4, 2022

fedebongio commented Nov 8, 2022

fedebongio commented Nov 8, 2022

brandond commented Nov 11, 2022

Huang-Wei commented Nov 11, 2022

[KEP-3521] Part 1: New Pod API .spec.schedulingGates #113274

[KEP-3521] Part 1: New Pod API .spec.schedulingGates #113274

Conversation

Huang-Wei commented Oct 22, 2022 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-triage-robot commented Oct 22, 2022

Huang-Wei commented Oct 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahg-g commented Oct 27, 2022

ahg-g commented Oct 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kerthcet Oct 31, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Huang-Wei commented Nov 3, 2022

smarterclayton commented Nov 3, 2022

k8s-ci-robot commented Nov 3, 2022

ahg-g commented Nov 3, 2022

Huang-Wei commented Nov 3, 2022

Huang-Wei commented Nov 4, 2022

ahg-g commented Nov 4, 2022

soltysh commented Nov 4, 2022

fedebongio commented Nov 8, 2022

fedebongio commented Nov 8, 2022

brandond commented Nov 11, 2022

Huang-Wei commented Nov 11, 2022

Huang-Wei commented Oct 22, 2022 •

edited

kerthcet Oct 31, 2022 •

edited