coreos - Kubernetes pods crash after a few hours, restarting kubelet fixes -


i'm running insecure test kubernetes v1.7.5 in bare metal setup running coreos 1409.7.0. i've installed api-server, controller, scheduler, proxy , kubelet on master node, , kubelet , proxy on 3 other worker nodes, flanneld using systemd service files provided in contrib/init k8s project.

everything running when cluster starts up. can deploy dashboard , deploys i've customized (consul clients/server, nginx, etc) , work great. however, if leave cluster running few hours come , every pod in crashloopbackup, being restarted many times. thing solves problem restart kubelet on each machine. problem goes away , goes normal.

logs kubelet after it's gone bad state:

sep 10 19:09:06 k8-app-2.example.com kubelet[1025]: , failed "startcontainer" "nginx-server" crashloopbackoff: "back-off 5m0s restarting failed container=nginx-server pod=nginx-deployment-617048525-mgf0v_default(f6dff9f2-95db-11e7-b533-02c75fb65df0)" sep 10 19:09:06 k8-app-2.example.com kubelet[1025]: ] sep 10 19:09:07 k8-app-2.example.com kubelet[1025]: i0910 19:09:07.286367    1025 kuberuntime_manager.go:457] container {name:nginx-server image:nginx command:[] args:[] workingdir: ports:[{name:http hostport:0 containerport:80 protocol:tcp hostip:}] envfrom:[] env:[{name:node_ip value: valuefrom:&envvarsource{fieldref:&objectfieldselector{apiversion:v1,fieldpath:status.hostip,},resourcefieldref:nil,configmapkeyref:nil,secretkeyref:nil,}} {name:pod_ip value: valuefrom:&envvarsource{fieldref:&objectfieldselector{apiversion:v1,fieldpath:status.podip,},resourcefieldref:nil,configmapkeyref:nil,secretkeyref:nil,}}] resources:{limits:map[] requests:map[]} volumemounts:[] livenessprobe:&probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/,port:80,host:,scheme:http,httpheaders:[],},tcpsocket:nil,},initialdelayseconds:10,timeoutseconds:1,periodseconds:10,successthreshold:1,failurethreshold:3,} readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} dead, restartpolicy says should restart it. sep 10 19:09:07 k8-app-2.example.com kubelet[1025]: i0910 19:09:07.286795    1025 kuberuntime_manager.go:457] container {name:regup image:registry.hub.docker.com/spunon/regup:latest command:[] args:[] workingdir: ports:[] envfrom:[] env:[{name:service_name value:nginx valuefrom:nil} {name:service_port value:80 valuefrom:nil} {name:node_ip value: valuefrom:&envvarsource{fieldref:&objectfieldselector{apiversion:v1,fieldpath:status.hostip,},resourcefieldref:nil,configmapkeyref:nil,secretkeyref:nil,}} {name:pod_ip value: valuefrom:&envvarsource{fieldref:&objectfieldselector{apiversion:v1,fieldpath:status.podip,},resourcefieldref:nil,configmapkeyref:nil,secretkeyref:nil,}}] resources:{limits:map[] requests:map[]} volumemounts:[] livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} dead, restartpolicy says should restart it. sep 10 19:09:07 k8-app-2.example.com kubelet[1025]: i0910 19:09:07.287071    1025 kuberuntime_manager.go:741] checking backoff container "nginx-server" in pod "nginx-deployment-617048525-mgf0v_default(f6dff9f2-95db-11e7-b533-02c75fb65df0)" sep 10 19:09:07 k8-app-2.example.com kubelet[1025]: i0910 19:09:07.287376    1025 kuberuntime_manager.go:751] back-off 5m0s restarting failed container=nginx-server pod=nginx-deployment-617048525-mgf0v_default(f6dff9f2-95db-11e7-b533-02c75fb65df0) sep 10 19:09:07 k8-app-2.example.com kubelet[1025]: i0910 19:09:07.287601    1025 kuberuntime_manager.go:741] checking backoff container "regup" in pod "nginx-deployment-617048525-mgf0v_default(f6dff9f2-95db-11e7-b533-02c75fb65df0)" sep 10 19:09:07 k8-app-2.example.com kubelet[1025]: i0910 19:09:07.287863    1025 kuberuntime_manager.go:751] back-off 5m0s restarting failed container=regup pod=nginx-deployment-617048525-mgf0v_default(f6dff9f2-95db-11e7-b533-02c75fb65df0)

edit: here logs kubelet when issue seems start


Comments

Post a Comment

Popular posts from this blog

resizing Telegram inline keyboard -

telegram's inline keyboard great feature lots of different use cases. inline buttons added list of items this: inline_keyboard = [[inlinekeyboardbutton(text="button", callback_data="button"), inlinekeyboardbutton(text="reset",callback_data="reset")]] inline_keyboard_markup = inlinekeyboardmarkup(inline_keyboard) update.message.reply_text("hi", reply_markup=inline_keyboard_markup) the above code adds 2 buttons each half width of chat screen. i know the normal keyboard button there resize_keyboard parameter somehow can used. my question is there way resize inline buttons ? example make full width or quarter width. unfortunately, can't now. :( you can suggest @botsupport , might add feature next version.
Read more

command line - How can a Python program background itself? -

consider following python command-line program. doesn't work because incomplete, think illustrates problem: #!/usr/bin/python3 import multiprocessing pool = multiprocessing.pool() task in expensive_iterator_with_user_interaction(): pool.call_async(expensive_computation, (task,)) pool.close() print("dear user, may press ctrl-z , bg script.") pool.join() thus, problem scripts needs user input while it's launching expensive parallel processes. there moment when user interaction not needed anymore. then, ask user background process shell actions. however, can achieved program itself? (side note: core of problem entanglement of user input firing “expensive computations”. therefore, cannot separate both in 2 scripts, , letting 1 spawning other daemon.) the command screen may you. use screen -l create new session , run application. after inputting data, can press " ctrl-a " , press " d " on keyboard detach session. now...
Read more

php - "cURL error 28: Resolving timed out" on Wordpress on Azure App Service on Linux -

Image
my wordpress installation on azure app service on linux using custom docker container has slow response times. pages take around 20-40 seconds load. i have troubleshooting plugin installed indicates problem "curl error 28: resolving timed out after n milliseconds" when making requests following urls https://api.wordpress.org/core/version-check/1.7/ http://api.wordpress.org/core/version-check/1.7/ https://api.wordpress.org/plugins/update-check/1.1/ http://api.wordpress.org/plugins/update-check/1.1/ https://api.wordpress.org/themes/update-check/1.1/ http://api.wordpress.org/themes/update-check/1.1/ curl works fine on scm-site command line. example works ok. curl -x post http://api.wordpress.org/core/version-check/1.7/ edit if ssh container , run php code, works fine. <?php $url = 'http://api.wordpress.org/core/version-check/1.7/'; $fields = array( 'version' => urlencode('4.8.1'), 'php' => urlencode(...
Read more