Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wire up feature enablement metric in feature_gate.go #112690

Merged
merged 2 commits into from Sep 26, 2022
Merged

wire up feature enablement metric in feature_gate.go #112690

merged 2 commits into from Sep 26, 2022

Conversation

logicalhan
Copy link
Member

@logicalhan logicalhan commented Sep 23, 2022 •
edited

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

This change allows us to track feature enablement generically.

Add `kubernetes_feature_enabled` metric series to track whether each active feature gate is enabled.

This is the output of the metric for apiserver using local-up-cluster.sh:

# HELP kubernetes_feature_enabled [ALPHA] This metric records the data about the stage and enablement of a k8s feature.
# TYPE kubernetes_feature_enabled gauge
kubernetes_feature_enabled{name="APIListChunking",stage="BETA"} 1
kubernetes_feature_enabled{name="APIPriorityAndFairness",stage="BETA"} 1
kubernetes_feature_enabled{name="APIResponseCompression",stage="BETA"} 1
kubernetes_feature_enabled{name="APISelfSubjectReview",stage="ALPHA"} 0
kubernetes_feature_enabled{name="APIServerIdentity",stage="ALPHA"} 0
kubernetes_feature_enabled{name="APIServerTracing",stage="ALPHA"} 0
kubernetes_feature_enabled{name="AdvancedAuditing",stage=""} 1
kubernetes_feature_enabled{name="AggregatedDiscoveryEndpoint",stage="ALPHA"} 0
kubernetes_feature_enabled{name="AllAlpha",stage="ALPHA"} 0
kubernetes_feature_enabled{name="AllBeta",stage="BETA"} 0
kubernetes_feature_enabled{name="AnyVolumeDataSource",stage="BETA"} 1
kubernetes_feature_enabled{name="AppArmor",stage="BETA"} 1
kubernetes_feature_enabled{name="CPUManager",stage="BETA"} 1
kubernetes_feature_enabled{name="CPUManagerPolicyAlphaOptions",stage="ALPHA"} 0
kubernetes_feature_enabled{name="CPUManagerPolicyBetaOptions",stage="BETA"} 1
kubernetes_feature_enabled{name="CPUManagerPolicyOptions",stage="BETA"} 1
kubernetes_feature_enabled{name="CSIInlineVolume",stage=""} 1
kubernetes_feature_enabled{name="CSIMigration",stage=""} 1
kubernetes_feature_enabled{name="CSIMigrationAWS",stage=""} 1
kubernetes_feature_enabled{name="CSIMigrationAzureDisk",stage=""} 1
kubernetes_feature_enabled{name="CSIMigrationAzureFile",stage="BETA"} 1
kubernetes_feature_enabled{name="CSIMigrationGCE",stage=""} 1
kubernetes_feature_enabled{name="CSIMigrationPortworx",stage="BETA"} 0
kubernetes_feature_enabled{name="CSIMigrationRBD",stage="ALPHA"} 0
kubernetes_feature_enabled{name="CSIMigrationvSphere",stage="BETA"} 1
kubernetes_feature_enabled{name="CSINodeExpandSecret",stage="ALPHA"} 0
kubernetes_feature_enabled{name="CSIStorageCapacity",stage=""} 1
kubernetes_feature_enabled{name="CSIVolumeHealth",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ContainerCheckpoint",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ContextualLogging",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ControllerManagerLeaderMigration",stage=""} 1
kubernetes_feature_enabled{name="CronJobTimeZone",stage="BETA"} 1
kubernetes_feature_enabled{name="CustomCPUCFSQuotaPeriod",stage="ALPHA"} 0
kubernetes_feature_enabled{name="CustomResourceValidationExpressions",stage="BETA"} 1
kubernetes_feature_enabled{name="DaemonSetUpdateSurge",stage=""} 1
kubernetes_feature_enabled{name="DelegateFSGroupToCSIDriver",stage="BETA"} 1
kubernetes_feature_enabled{name="DevicePlugins",stage="BETA"} 1
kubernetes_feature_enabled{name="DisableAcceleratorUsageMetrics",stage=""} 1
kubernetes_feature_enabled{name="DisableCloudProviders",stage="ALPHA"} 0
kubernetes_feature_enabled{name="DisableKubeletCloudCredentialProviders",stage="ALPHA"} 0
kubernetes_feature_enabled{name="DownwardAPIHugePages",stage="BETA"} 1
kubernetes_feature_enabled{name="DryRun",stage=""} 1
kubernetes_feature_enabled{name="DynamicKubeletConfig",stage="DEPRECATED"} 0
kubernetes_feature_enabled{name="EfficientWatchResumption",stage=""} 1
kubernetes_feature_enabled{name="EndpointSliceTerminatingCondition",stage="BETA"} 1
kubernetes_feature_enabled{name="EphemeralContainers",stage=""} 1
kubernetes_feature_enabled{name="ExecProbeTimeout",stage=""} 1
kubernetes_feature_enabled{name="ExpandCSIVolumes",stage=""} 1
kubernetes_feature_enabled{name="ExpandInUsePersistentVolumes",stage=""} 1
kubernetes_feature_enabled{name="ExpandPersistentVolumes",stage=""} 1
kubernetes_feature_enabled{name="ExpandedDNSConfig",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ExperimentalHostUserNamespaceDefaulting",stage="BETA"} 0
kubernetes_feature_enabled{name="GRPCContainerProbe",stage="BETA"} 1
kubernetes_feature_enabled{name="GracefulNodeShutdown",stage="BETA"} 1
kubernetes_feature_enabled{name="GracefulNodeShutdownBasedOnPodPriority",stage="BETA"} 1
kubernetes_feature_enabled{name="HPAContainerMetrics",stage="ALPHA"} 0
kubernetes_feature_enabled{name="HPAScaleToZero",stage="ALPHA"} 0
kubernetes_feature_enabled{name="HonorPVReclaimPolicy",stage="ALPHA"} 0
kubernetes_feature_enabled{name="IPTablesOwnershipCleanup",stage="ALPHA"} 0
kubernetes_feature_enabled{name="IdentifyPodOS",stage=""} 1
kubernetes_feature_enabled{name="InTreePluginAWSUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginAzureDiskUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginAzureFileUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginGCEUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginPortworxUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginRBDUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginvSphereUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="JobMutableNodeSchedulingDirectives",stage="BETA"} 1
kubernetes_feature_enabled{name="JobPodFailurePolicy",stage="ALPHA"} 0
kubernetes_feature_enabled{name="JobReadyPods",stage="BETA"} 1
kubernetes_feature_enabled{name="JobTrackingWithFinalizers",stage="BETA"} 1
kubernetes_feature_enabled{name="KMSv2",stage="ALPHA"} 0
kubernetes_feature_enabled{name="KubeletCredentialProviders",stage="BETA"} 1
kubernetes_feature_enabled{name="KubeletInUserNamespace",stage="ALPHA"} 0
kubernetes_feature_enabled{name="KubeletPodResources",stage="BETA"} 1
kubernetes_feature_enabled{name="KubeletPodResourcesGetAllocatable",stage="BETA"} 1
kubernetes_feature_enabled{name="KubeletTracing",stage="ALPHA"} 0
kubernetes_feature_enabled{name="LegacyServiceAccountTokenNoAutoGeneration",stage="BETA"} 1
kubernetes_feature_enabled{name="LocalStorageCapacityIsolation",stage=""} 1
kubernetes_feature_enabled{name="LocalStorageCapacityIsolationFSQuotaMonitoring",stage="ALPHA"} 0
kubernetes_feature_enabled{name="LogarithmicScaleDown",stage="BETA"} 1
kubernetes_feature_enabled{name="LoggingAlphaOptions",stage="ALPHA"} 0
kubernetes_feature_enabled{name="LoggingBetaOptions",stage="BETA"} 1
kubernetes_feature_enabled{name="MatchLabelKeysInPodTopologySpread",stage="ALPHA"} 0
kubernetes_feature_enabled{name="MaxUnavailableStatefulSet",stage="ALPHA"} 0
kubernetes_feature_enabled{name="MemoryManager",stage="BETA"} 1
kubernetes_feature_enabled{name="MemoryQoS",stage="ALPHA"} 0
kubernetes_feature_enabled{name="MinDomainsInPodTopologySpread",stage="BETA"} 0
kubernetes_feature_enabled{name="MixedProtocolLBService",stage="BETA"} 1
kubernetes_feature_enabled{name="MultiCIDRRangeAllocator",stage="ALPHA"} 0
kubernetes_feature_enabled{name="NetworkPolicyEndPort",stage=""} 1
kubernetes_feature_enabled{name="NetworkPolicyStatus",stage="ALPHA"} 0
kubernetes_feature_enabled{name="NodeInclusionPolicyInPodTopologySpread",stage="ALPHA"} 0
kubernetes_feature_enabled{name="NodeOutOfServiceVolumeDetach",stage="ALPHA"} 0
kubernetes_feature_enabled{name="NodeSwap",stage="ALPHA"} 0
kubernetes_feature_enabled{name="OpenAPIEnums",stage="BETA"} 1
kubernetes_feature_enabled{name="OpenAPIV3",stage="BETA"} 1
kubernetes_feature_enabled{name="PodAndContainerStatsFromCRI",stage="ALPHA"} 0
kubernetes_feature_enabled{name="PodDeletionCost",stage="BETA"} 1
kubernetes_feature_enabled{name="PodDisruptionConditions",stage="ALPHA"} 0
kubernetes_feature_enabled{name="PodHasNetworkCondition",stage="ALPHA"} 0
kubernetes_feature_enabled{name="PodSecurity",stage=""} 1
kubernetes_feature_enabled{name="ProbeTerminationGracePeriod",stage="BETA"} 1
kubernetes_feature_enabled{name="ProcMountType",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ProxyTerminatingEndpoints",stage="ALPHA"} 0
kubernetes_feature_enabled{name="QOSReserved",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ReadWriteOncePod",stage="ALPHA"} 0
kubernetes_feature_enabled{name="RecoverVolumeExpansionFailure",stage="ALPHA"} 0
kubernetes_feature_enabled{name="RemainingItemCount",stage="BETA"} 1
kubernetes_feature_enabled{name="RemoveSelfLink",stage=""} 1
kubernetes_feature_enabled{name="RetroactiveDefaultStorageClass",stage="ALPHA"} 0
kubernetes_feature_enabled{name="RotateKubeletServerCertificate",stage="BETA"} 1
kubernetes_feature_enabled{name="SELinuxMountReadWriteOncePod",stage="ALPHA"} 0
kubernetes_feature_enabled{name="SeccompDefault",stage="BETA"} 1
kubernetes_feature_enabled{name="ServerSideApply",stage=""} 1
kubernetes_feature_enabled{name="ServerSideFieldValidation",stage="BETA"} 1
kubernetes_feature_enabled{name="ServiceIPStaticSubrange",stage="BETA"} 1
kubernetes_feature_enabled{name="ServiceInternalTrafficPolicy",stage="BETA"} 1
kubernetes_feature_enabled{name="SizeMemoryBackedVolumes",stage="BETA"} 1
kubernetes_feature_enabled{name="StatefulSetAutoDeletePVC",stage="ALPHA"} 0
kubernetes_feature_enabled{name="StatefulSetMinReadySeconds",stage=""} 1
kubernetes_feature_enabled{name="StorageVersionAPI",stage="ALPHA"} 0
kubernetes_feature_enabled{name="StorageVersionHash",stage="BETA"} 1
kubernetes_feature_enabled{name="TopologyAwareHints",stage="BETA"} 1
kubernetes_feature_enabled{name="TopologyManager",stage="BETA"} 1
kubernetes_feature_enabled{name="UserNamespacesStatelessPodsSupport",stage="ALPHA"} 0
kubernetes_feature_enabled{name="VolumeCapacityPriority",stage="ALPHA"} 0
kubernetes_feature_enabled{name="WatchBookmark",stage=""} 1
kubernetes_feature_enabled{name="WinDSR",stage="ALPHA"} 0
kubernetes_feature_enabled{name="WinOverlay",stage="BETA"} 1
kubernetes_feature_enabled{name="WindowsHostProcessContainers",stage="BETA"} 1

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Sep 23, 2022
@k8s-ci-robot k8s-ci-robot added kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 23, 2022
@logicalhan logicalhan force-pushed the feature-gate branch 3 times, most recently from 9400ea7 to fb4828f Compare September 23, 2022 03:06
@k8s-ci-robot k8s-ci-robot added the area/dependency Issues or PRs related to dependency changes label Sep 23, 2022
@k8s-triage-robot
Copy link

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

@logicalhan
Copy link
Member Author

/retest

@k8s-ci-robot k8s-ci-robot added sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. labels Sep 23, 2022
@k8s-ci-robot k8s-ci-robot added area/kubelet sig/network Categorizes an issue or PR as relevant to SIG Network. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Sep 23, 2022
@logicalhan
add unit test coverage
5f58977 
Change-Id: Idfbc9e72f1e0051a9423b23d1f0397364bb3c251
@liggitt
Copy link
Member

liggitt commented Sep 26, 2022

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 26, 2022
@logicalhan
Copy link
Member Author

/retest

2 similar comments
@logicalhan
Copy link
Member Author

/retest

@logicalhan
Copy link
Member Author

/retest

@k8s-ci-robot k8s-ci-robot merged commit d2ae6fb into kubernetes:master Sep 26, 2022
SIG Node PR Triage automation moved this from Triage to Done Sep 26, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Sep 26, 2022
@logicalhan logicalhan mentioned this pull request Sep 27, 2022
Closed
@fedebongio
Copy link
Contributor

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 27, 2022
@jberkus
Copy link

jberkus commented Sep 27, 2022

Isn't this a feature? Why is it not a feature?

@logicalhan
Copy link
Member Author

Isn't this a feature? Why is it not a feature?

We do not mandate KEPs to add metrics, i.e. this is not a feature. This is a metric that tells you if a feature is enabled. Furthermore, this is an already mandated question of production readiness review.

@sftim
Copy link
Contributor

sftim commented Sep 27, 2022 •
edited

Can I suggest this tweak to the release note?

Add `kubernetes_feature_enabled` metric series to track whether each active feature gate is enabled.

@logicalhan
Copy link
Member Author

Can I suggest this tweak to the release note?

Add `kubernetes_feature_enabled` metric series to track whether each active feature gate is enabled.

Sure thing.

@pacoxu pacoxu mentioned this pull request Sep 28, 2022
Merged
@aojea
Copy link
Member

aojea commented Sep 28, 2022 •
edited

@logicalhan @liggitt GA stage is represented as empty?

kubernetes_feature_enabled{name="AdvancedAuditing",stage=""} 1

kubernetes_feature_enabled{name="PodSecurity",stage=""} 1

EDIT

This is really cool by the way, nice to have this exposed via metrics

@aojea aojea mentioned this pull request Sep 28, 2022
Open
@logicalhan
Copy link
Member Author

This is really cool by the way, nice to have this exposed via metrics

Thanks! Yeah I dig it too :)

@logicalhan @liggitt GA stage is represented as empty?

kubernetes_feature_enabled{name="AdvancedAuditing",stage=""} 1

kubernetes_feature_enabled{name="PodSecurity",stage=""} 1

Yeah, that's a result of this: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/component-base/featuregate/feature_gate.go#L80-L82

@jberkus
Copy link

jberkus commented Sep 29, 2022

Isn't this a feature? Why is it not a feature?

We do not mandate KEPs to add metrics, i.e. this is not a feature. This is a metric that tells you if a feature is enabled. Furthermore, this is an already mandated question of production readiness review.

I was thinking of it more for the release announcement, etc. This seems really useful to users.

@logicalhan
Copy link
Member Author

I was thinking of it more for the release announcement, etc. This seems really useful to users.

Ah, okay. Yeah would be happy to announce it as a feature. I also agree this will be extremely useful to our users, it feels very much an obvious thing to do in retrospect.

@jberkus
Copy link

jberkus commented Oct 5, 2022

@fsmunoz @ramrodo FYI. This is a feature worth specially listing, even without a KEP.

@ramrodo
Copy link
Member

ramrodo commented Oct 6, 2022

@fsmunoz @ramrodo FYI. This is a feature worth specially listing, even without a KEP.

Sure. We'll ensure this release note is included.

@logicalhan
Copy link
Member Author

@fsmunoz @ramrodo FYI. This is a feature worth specially listing, even without a KEP.

Do you know anyone who can help me with this PR?

@fsmunoz
Copy link
Contributor

fsmunoz commented Nov 28, 2022

This will be included in the Major Themes for the v1.26 release blog.

This was referenced Mar 13, 2023
Closed
Closed
@JaredTan95 JaredTan95 mentioned this pull request Apr 25, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liggitt liggitt liggitt left review comments

@saad-ali saad-ali Awaiting requested review from saad-ali

@yujuhong yujuhong Awaiting requested review from yujuhong

Assignees

@liggitt liggitt

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver area/dependency Issues or PRs related to dependency changes area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node PR Triage
Done
Milestone
v1.26
Development

Successfully merging this pull request may close these issues.

None yet
10 participants
@logicalhan @k8s-triage-robot @liggitt @k8s-ci-robot @fedebongio @jberkus @sftim @aojea @ramrodo @fsmunoz