Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add a metric for retroactive sc errors #113323

Merged
merged 1 commit into from Oct 26, 2022
Merged

add a metric for retroactive sc errors #113323

merged 1 commit into from Oct 26, 2022

Conversation

RomanBednar
Copy link
Contributor

@RomanBednar RomanBednar commented Oct 25, 2022 •
edited

What type of PR is this?

/kind feature

What this PR does / why we need it:

Adds a metric that is required for graduating RetroactiveDefaultStorageClass feature to beta.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

This is how I tested that the metric works:

1. Test that metric is registered

Metrics are registered with alpha stability

curl -k --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key https://localhost:10257/metrics
.
.
.
# HELP retroactive_storageclass_errors_total [ALPHA] Total number of failed retroactive StorageClass assignments to persistent volume claim
# TYPE retroactive_storageclass_errors_total counter
retroactive_storageclass_errors_total 0
# HELP retroactive_storageclass_total [ALPHA] Total number of retroactive StorageClass assignments to persistent volume claim
# TYPE retroactive_storageclass_total counter
retroactive_storageclass_total 0
.
.
.
2. Test metric for retroactive SC attempt count

Disable default SC

$ kc patch sc/standard -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Create PVC without SC

$ cat /tmp/pvc.yaml 
apiVersion: v1  
kind: PersistentVolumeClaim  
metadata:  
  name: task-pv-claim  
spec:  
  accessModes:  
     - ReadWriteOnce  
  resources:  
    requests:  
      storage: 1Gi
 
$ kc create -f /tmp/pvc.yaml

Check the metric is recording attempts

$ curl -ks --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key https://localhost:10257/metrics | grep -v ^# | grep retroactive_storageclass

retroactive_storageclass_errors_total 0
retroactive_storageclass_total 2

Make SC default again

$ kc patch sc/standard -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Verify feature is enabled and working

$ kc get pvc/task-pv-claim -o yaml | grep storageClassName
  storageClassName: standard

Check there is no error metric

$ curl -ks --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key https://localhost:10257/metrics | grep -v ^# | grep retroactive_storageclass

retroactive_storageclass_errors_total 0
retroactive_storageclass_total 16
3. Test metric for retroactive SC attempt failure count

Disable default SC

$ kc patch sc/standard -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Create PVC without SC

$ cat /tmp/pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

$ kc create -f /tmp/pvc.yaml

Flag SC as default and halt KCM process

$ kc patch sc/standard -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' && kill -SIGSTOP 87764
storageclass.storage.k8s.io/standard patched

Resume KCM process and check for errors

$ kill -SIGCONT 87764

$ curl -ks --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key https://localhost:10257/metrics | grep -v ^# | grep retroactive_storageclass
retroactive_storageclass_errors_total 4
retroactive_storageclass_total 63

Does this PR introduce a user-facing change?

Metrics for RetroactiveDefaultStorageClass feature are now available. To see an attempt count for updating PVC retroactively with a default StorageClass see `retroactive_storageclass_total` metric and for total numer of errors see `retroactive_storageclass_errors_total`.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/issues/3333

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 25, 2022
@k8s-ci-robot
Copy link
Contributor

@RomanBednar: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Oct 25, 2022
@k8s-ci-robot k8s-ci-robot added sig/apps Categorizes an issue or PR as relevant to SIG Apps. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 25, 2022
@RomanBednar RomanBednar changed the title add a metric for retroactive sc errors WIP: add a metric for retroactive sc errors Oct 25, 2022
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 25, 2022
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 25, 2022
@RomanBednar RomanBednar changed the title WIP: add a metric for retroactive sc errors add a metric for retroactive sc errors Oct 25, 2022
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 25, 2022
@RomanBednar RomanBednar mentioned this pull request Oct 25, 2022
Merged
@jsafrane
Copy link
Member

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 26, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsafrane, RomanBednar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 26, 2022
@jsafrane
Copy link
Member

/retest-required

@k8s-ci-robot k8s-ci-robot merged commit ca03736 into kubernetes:master Oct 26, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Oct 26, 2022
@RomanBednar RomanBednar mentioned this pull request Jun 2, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsafrane jsafrane jsafrane left review comments

@saad-ali saad-ali Awaiting requested review from saad-ali

@thockin thockin Awaiting requested review from thockin

Assignees

@jsafrane jsafrane

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/apps Categorizes an issue or PR as relevant to SIG Apps. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
v1.26
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@RomanBednar @k8s-ci-robot @jsafrane