Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add metric for terminated pods with tracking finalizer #113176

Merged
merged 1 commit into from Oct 21, 2022
Merged

Add metric for terminated pods with tracking finalizer #113176

merged 1 commit into from Oct 21, 2022

Conversation

alculquicondor
Copy link
Member

@alculquicondor alculquicondor commented Oct 19, 2022 •
edited

What type of PR is this?

/kind feature

/sig apps
/sig instrumentation

What this PR does / why we need it:

Add a counter metric to keep track of terminated Pods as they get and loose the tracking finalizer.

Which issue(s) this PR fixes:

Part of kubernetes/enhancements#2307

Special notes for your reviewer:

The KEP originally proposed a gauge, but it proved to be difficult to test whether the controller saw finalizers at all.
A counter with an event label (with values add and remove) can be used to construct a gauge.

Does this PR introduce a user-facing change?

New metric job_controller_terminated_pods_tracking_finalizer can be used to monitor whether the job controller is removing Pod finalizers from terminated Pods after accounting them in Job status.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: git.k8s.io/enhancements/sig-apps/2307-job-tracking-without-lingering-pods
- [Usage]: https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-tracking-with-finalizers

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/feature Categorizes issue or PR as related to a new feature. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. labels Oct 19, 2022
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 19, 2022
@k8s-ci-robot k8s-ci-robot added area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. approved Indicates a PR has been approved by an approver from all required OWNERS files. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 19, 2022
@alculquicondor
Copy link
Member Author

/assign @soltysh
for sig-apps

and can someone from @kubernetes/sig-instrumentation-approvers take a look?

logicalhan
logicalhan reviewed Oct 19, 2022
View reviewed changes
Copy link
Member

@logicalhan logicalhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve
(from sig instrumentation)

pkg/controller/job/metrics/metrics.go Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 19, 2022
@alculquicondor
Copy link
Member Author

/hold for sig apps review

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 19, 2022
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 19, 2022
@alculquicondor
Copy link
Member Author

/retest

Opened #113207 for the flakies

mimowo
mimowo reviewed Oct 20, 2022
View reviewed changes
pkg/controller/job/metrics/metrics.go Outdated
TerminatedPodsTrackingFinalizerTotal = metrics.NewCounterVec(
&metrics.CounterOpts{
Subsystem: JobControllerSubsystem,
Name: "terminated_pods_tracking_finalizer",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest to add the _total suffix for consistency with other counter metrics. Seems to be the best practice by Prometheus: https://prometheus.io/docs/practices/naming/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, that was the intention, but I forgot to rename it :)

pkg/controller/job/tracking_utils_test.go Outdated Show resolved Hide resolved
@alculquicondor
Add metric for terminated pods with tracking finalizer
12d308f 
Change-Id: I26f3169588c30ed82250cb7baff8e277f8d13bb7
@alculquicondor
Copy link
Member Author

Rebased and squashed

@mimowo
Copy link
Contributor

mimowo commented Oct 20, 2022

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 20, 2022
@dgrisonnet
Copy link
Member

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 20, 2022
soltysh
soltysh approved these changes Oct 20, 2022
View reviewed changes
Copy link
Contributor

@soltysh soltysh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@soltysh
Copy link
Contributor

soltysh commented Oct 20, 2022

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 20, 2022
@soltysh
Copy link
Contributor

soltysh commented Oct 20, 2022

networking glitch when getting outh token on google API
/test pull-kubernetes-e2e-gce-ubuntu-containerd

@soltysh
Copy link
Contributor

soltysh commented Oct 20, 2022

/priority important-longterm

@k8s-ci-robot k8s-ci-robot added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 20, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, logicalhan, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@alculquicondor
Copy link
Member Author

/retest

@pacoxu
Copy link
Member

pacoxu commented Oct 21, 2022

/test pull-kubernetes-e2e-gce-ubuntu-containerd

@k8s-ci-robot k8s-ci-robot merged commit 2313e2b into kubernetes:master Oct 21, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Oct 21, 2022
This was referenced Oct 31, 2022
Closed
Merged
@mimowo mimowo mentioned this pull request Nov 10, 2022
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@logicalhan logicalhan logicalhan left review comments

@mimowo mimowo mimowo left review comments

@soltysh soltysh soltysh approved these changes

@kow3ns kow3ns Awaiting requested review from kow3ns

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

Assignees

@soltysh soltysh

@logicalhan logicalhan

@mimowo mimowo

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.26
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@alculquicondor @mimowo @dgrisonnet @soltysh @k8s-ci-robot @pacoxu @logicalhan