/assign @deads2k @soltysh @tkashem

I think that this is a better solution for the etcd healthzchecks, I'm not familiar with the code so I plumbed it here, but it can go directly into the etcd healthcheck

Looking for feedback first, I will add tests if this is an acceptable solution

/hold

/sig api-machinery

🤔 👀

/test pull-kubernetes-integration

the etcd check specifically was attempted to be made non-blocking in #104437, but that didn't get completed

/test pull-kubernetes-integration

the etcd check specifically was attempted to be made non-blocking in #104437, but that didn't get completed

let's target the etcd check specifically, I've also found a bug in current logic

/assign @liggitt @lavalamp

as original reviewers of #104437

I think this is ready to go as long as we don't make N separate checkers...

/triage accepted

I think this is ready to go as long as we don't make N separate checkers...

kindly ping @lavalamp #112046 (comment)

Sorry I missed your last update.

/lgtm
/approve

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, lavalamp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

If you can figure out an easy way to count how many health checks this does in e.g. our e2e tests before/after this change, and the difference is very large, a backport might be reasonable.

(Do we scrape etcd metrics at the test end? That would be the first place I'd look. This might not be super easy to measure.)

(Do we scrape etcd metrics at the test end? That would be the first place I'd look. This might not be super easy to measure.)

Mike Spreitzer added tooling to dump the metrics of etcd on the integration tests (all test share the same etcd instance)
You just download the tarball with the mtrics

wget https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/112252/pull-kubernetes-integration/1568270000328806400/artifacts/etcd-scrapes.tgz

and run the script he created pointing to the file

$HOME/go/src/k8s.io/kubernetes/hack/run-prometheus-on-etcd-scrapes.sh etcd-scrapes.tgz

I didn't find any metrics that shows the number of Get received on the healthz endpoint, or something that can give us an idea ... I've compared a dumped version from before and other from after this patch and I couldn't find any significant difference-

I don't think that our e2e abuse the /healthz/etcd endpoint , it seems Openshift does much more stress on the endpoint and that's why we were carrying over with David's patch

OK, thanks for checking!

I missed that originally - but this looks great - I'm really happy that you made that happen.

Re @lavalamp questions - i don't think we would find anything in our tests (which don't overload any of the endpoints FWICT), but it certainly can help protecting some real-world scenarios.
But I don't think I can come up with a justifying argument for backport.