kubernetes helm: "lost connection to pod" and "transport is closing" errors
I run helm upgrade --install to modify the state of my kubernetes cluster and I sometimes get an error like this:
22:24:34 StdErr: E0126 17:24:28.472048 48084 portforward.go:178] lost connection to pod
22:24:34 Error: UPGRADE FAILED: transport is closing
It seems that I am not the only one, and it seems to happen with many different helm commands. All of these github issues have descriptions or comments mentioning "lost connection to pod" or "transport is closing" errors (usually both):
- https://github.com/kubernetes/helm/issues/1183
- https://github.com/kubernetes/helm/issues/2003
- https://github.com/kubernetes/helm/issues/2025
- https://github.com/kubernetes/helm/issues/2288
- https://github.com/kubernetes/helm/issues/2560
- https://github.com/kubernetes/helm/issues/3015
- https://github.com/kubernetes/helm/issues/3409
While it can be educational to read through hundreds of github issue comments, usually it's faster to cut to the chase on stackoverflow, and it didn't seem like this question existed yet, so here it is. Hopefully some quick symptom fixes and eventually one or more root cause diagnoses end up in the answers.
Answer
Memory limits were causing this error for me. The following fixed it:
kubectl set resources deployment tiller-deploy --limits=memory=200Mi