New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubelet: fix pod log line corruption when using timestamps and long lines #113481
Conversation
@rphillips: This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
5d44829
to
ddae396
Compare
Verified it locally that the fix indeed fixes the issue. /lgtm |
/assign thanks for the clear test case and images. /approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: derekwaynecarr, rphillips The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/priority important-soon |
will this fix be cherry-picked into older k8s versions? if so, which versions? EDIT: ah saw the PRs up to 1.23 from @rphillips . Thanks 👍🏻 |
…13481-upstream-release-1.25 Automated cherry pick of #113481: kubelet: fix pod log line corruption when using timestamps
resume kube job log using kube log timestamp require fix for kubernetes/kubernetes#77603 in kubernetes/kubernetes#113481 fix is backported to the following kubernetes version - 1.23.14 - 1.24.8 - 1.25.4 required fix for ansible/ansible-runner#1161 fix is backported to the following ansible-runner version - 2.2.2 - 2.3.1 added `RECEPTOR_KUBE_SUPPORT_RECONNECT` with following options: - “enabled”: this option will use timestamp with the log and enable our new code path - “disabled”: this option will not use timestamp and use the original code path - “auto”: auto detect if it's appropriate to enable timestamp base on kube version Co-Authored-By: Shane McDonald <me@shanemcd.com> Co-Authored-By: Seth Foster <fosterseth@users.noreply.github.com>
resume kube job log using kube log timestamp require fix for kubernetes/kubernetes#77603 in kubernetes/kubernetes#113481 fix is backported to the following kubernetes version - 1.23.14 - 1.24.8 - 1.25.4 required fix for ansible/ansible-runner#1161 fix is backported to the following ansible-runner version - 2.2.2 - 2.3.1 added `RECEPTOR_KUBE_SUPPORT_RECONNECT` with following options: - “enabled”: this option will use timestamp with the log and enable our new code path - “disabled”: this option will not use timestamp and use the original code path - “auto”: auto detect if it's appropriate to enable timestamp base on kube version Co-Authored-By: Shane McDonald <me@shanemcd.com> Co-Authored-By: Seth Foster <fosterseth@users.noreply.github.com> Signed-off-by: Hao Liu <haoli@redhat.com>
resume kube job log using kube log timestamp require fix for kubernetes/kubernetes#77603 in kubernetes/kubernetes#113481 fix is backported to the following kubernetes version - 1.23.14 - 1.24.8 - 1.25.4 required fix for ansible/ansible-runner#1161 fix is backported to the following ansible-runner version - 2.2.2 - 2.3.1 added `RECEPTOR_KUBE_SUPPORT_RECONNECT` with following options: - “enabled”: this option will use timestamp with the log and enable our new code path - “disabled”: this option will not use timestamp and use the original code path - “auto”: auto detect if it's appropriate to enable timestamp base on kube version Co-Authored-By: Shane McDonald <me@shanemcd.com> Co-Authored-By: Seth Foster <fosterseth@users.noreply.github.com> Signed-off-by: Hao Liu <haoli@redhat.com>
The Kubernetes version we were using is no longer supported by AKS. Update: Also needed to remove a workaround we had put in for handling long log lines. This was fixed in k8s [here](kubernetes/kubernetes#113481).
What type of PR is this?
/kind bug
What this PR does / why we need it:
This PR resurrects #77822 which didn't merge and improves upon it.
When using "--timestamps" from the API to get a pod's logs the Kubelet will use a default Golang Bufferr (4096 bytes) to transfer the log to the API stream. This causes timestamps to be intermingled in the output because the code arbitrarily injects the timestamp after every buffer iteration. [source]
The fix is to check the line ending of the buffer to see if the timestamp should be appended.
Which issue(s) this PR fixes:
Fixes #77603
Special notes for your reviewer:
logs.yml:
Fetch the logs:
Does this PR introduce a user-facing change?
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: