30 juin 2014

Debugging a Docker container

So you can run your (put your favorite language/framework here) application within a Docker container, that's cool. Now, how do you debug it ?

This is feedback I got from my colleague Cyrile, trying to port his Tomcat 8 runtime from RUN@Cloud to a Docker-based environment. He use to integrate the tomcat appserver with syslog and capture thread dumps at runtime with a kill -3 signal sent to the JVM.

Few days ago I'd have suggested him to instrument his Docker image with tons of additional diagnostic tools and extra ports, but I've read Jerôme's blog in the meantime.

So, application is running inside a Docker container :

➜ ~  docker run -d dgageot/helloworld
1cbe9740baa45279eb775d2e0df5a12e5807c112ec382b3bdb20695df429854e

I'll use nsenter and Jerôme docker-enter to add a new diagnostic process to my container. Please remember a container is not a classic Virtual Machine, but "just" a group of process sharing a common Linux kernel namespace. nsenter let you add another process to the group, so can be used to attach a new shell session to the running container.

As I'm running boot2docker and this one don't have nsenter pre-installed (it should imho), I'm installing it directly from Jerôme Docker image :

➜  ~ docker run --rm -v /usr/local/bin:/target jpetazzo/nsenter
Unable to find image 'jpetazzo/nsenter' locally
Pulling repository jpetazzo/nsenter
...
Installing nsenter to /target
Installing docker-enter to /target

docker-enter script makes it even easier to use nsenter in a docker context, just need to know the docker image ID. So let's ssh into boot2docker and use this script to access the running container.

➜  ~ boot2docker ssh
Warning: Permanently added '[localhost]:2022' (RSA) to the list of known hosts.
                        ##        .
                  ## ## ##       ==
               ## ## ## ##      ===
           /""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
           \______ o          __/
             \    \        __/
              \____\______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
boot2docker: 1.0.0
             master : 16013ee - Mon Jun  9 16:33:25 UTC 2014


docker@boot2docker:~$ sudo docker-enter 1cbe9740baa45279
# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 14:12 ?        00:00:04 java -jar target/hello.jar
root        22     0  0 14:19 ?        00:00:00 -sh
root        25    22  0 14:19 ?        00:00:00 ps -ef

I now can use my favorite diagnostic tools to check application health.
# jstack 1
2014-06-30 14:20:17
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.5-b02 mixed mode):

"Attach Listener" #15 daemon prio=9 os_prio=0 tid=0x00007fbfa0001000 nid=0x25 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

Cool, isn't it ?
As you maybe noticed, this require to be root on host, so for your production server you might consider reading Jerôme's article carefully and consider using the ssh command= option with a dedicated ssh key for production diagnostic.

I'm considering using this exact same mechanism in a Jenkins plugin so user can provide arbitrary image for build environment, without requirement for a sshd server (like docker plugin does).


5 commentaires:

Shraddha a dit…

Thank you, this was exactly what I needed to debug my Docker application. I can now navigate the container file system. Your code snippets really helped.

An unrelated question: what utilty do you use to embed those code snippets in your blog post?

Thanks and regards,
Shraddha

harold425 a dit…

Thanks for this blog post, really useful!

@Shraddha: he is using syntaxhighlighter

Nyte a dit…
Ce commentaire a été supprimé par son auteur.
Nyte a dit…

If I understand correctly, why don't you simply use 'docker exec -it CONTAINER_NAME bash' to attach a bash session to your container?

Nicolas De Loof a dit…

Just read the date of this post and you'll notice docker-exec was introduced in docker 1.3, 6 months later.