This week is JenkinsWorld, and I won't be there ... but this doesn't mean I won't be part of it.
I'm pleased to announce Docker-slaves-plugin has reached 1.0.
Docker-slaves was created on year ago, as Yoann and I joined forces during DockerHackDay, with complementary ideas in mind to re-invent Jenkins in a Docker world. We are proud we were able to develop a working prototype within 24 hours and won 3rd prize on this challenge !
What makes Docker slaves such a revolution ?
Docker-related Jenkins plugins so far use a single docker container to provide a Jenkins agent (aka "slave"), which will host a build. Such a container is running a Jenkins remote agent and as such has to include a JRE, and allow connection from master or connect to master by itself. In both cases, this introduces significant constraints on the Docker image one can use. We want this to disappear.
To communicate with remote agent, Jenkins establish a communication channel with slave. On ssh slave, this communication uses the ssh connection as a tunnel. Other slave launcher uses various tunneling techniques or explicit port allocation to create this channel. But docker also has an interactive mode to run a container, which let us establish a stable (encrypted) connection with a remote container. Running `docker run -it` is very comparable to running a ssh client to connect to a remote host. So we use this capability to create the remoting channel directly on top of docker's interactive mode.
... and you can even define those containers using a Dockerfile you store in SCM ! If you love infrastructure as code, you'll be happy to get both your source code, continuous deployment pipeline script, and build environment stored all in your project SCM with plain Dockerfiles. Docker-slaves will build the required image on demand and run the build inside this container.
Last but not least, Jenkins docker-related plugins use to start with a fresh, new slave, which require to run a full SCM checkout and populate project dependencies. Docker-slaves uses a volume as build workspace, so this one is persistent between builds. There is huge room for improvement here, with assistance for docker's volume-drivers, typically to provide snapshot capability so you can browse workspace for a specific build (to diagnose a build failure) or replicate workspace on cluster.
If you have opportunity to give it a try, please send me feedback @ndeloof
UPDATE:
For some odd reason (who said "conspiracy" ?) docker-slaves 1.0 was not published in update center (yet). Will need to investigate with Jenkins infra team, but with Jenkins World they are all higly busy this week.
I'm pleased to announce Docker-slaves-plugin has reached 1.0.
Docker-slaves was created on year ago, as Yoann and I joined forces during DockerHackDay, with complementary ideas in mind to re-invent Jenkins in a Docker world. We are proud we were able to develop a working prototype within 24 hours and won 3rd prize on this challenge !
What makes Docker slaves such a revolution ?
Docker-related Jenkins plugins so far use a single docker container to provide a Jenkins agent (aka "slave"), which will host a build. Such a container is running a Jenkins remote agent and as such has to include a JRE, and allow connection from master or connect to master by itself. In both cases, this introduces significant constraints on the Docker image one can use. We want this to disappear.
With Docker-slaves, you can use ANY docker image to run your build.
To communicate with remote agent, Jenkins establish a communication channel with slave. On ssh slave, this communication uses the ssh connection as a tunnel. Other slave launcher uses various tunneling techniques or explicit port allocation to create this channel. But docker also has an interactive mode to run a container, which let us establish a stable (encrypted) connection with a remote container. Running `docker run -it` is very comparable to running a ssh client to connect to a remote host. So we use this capability to create the remoting channel directly on top of docker's interactive mode.
SSHD in docker container is evil. Docker-slaves don't need one, and does not require your master to expose JNLP on the Internet
Docker is not just about container, it's also about containers. docker-compose is a popular tool and demonstrate need to combine containers together to build useful stuff. We think a build environment has the same requirement. You need maven to build your project, but you also need a redis database and a web browser so you can run Selenium functional tests. Just create 3 containers, and link them together ! This is exactly what Docker-slaves does, so you can define your build environment as a set of containers.
Docker-slaves let you compose your build environment
... and you can even define those containers using a Dockerfile you store in SCM ! If you love infrastructure as code, you'll be happy to get both your source code, continuous deployment pipeline script, and build environment stored all in your project SCM with plain Dockerfiles. Docker-slaves will build the required image on demand and run the build inside this container.
Docker-slaves bring Continous-Delivery-as-Code to next level
Docker slave also adds a transparent "remoting" container so Jenkins plumbing is well setup and all Jenkins plugins work without any change. This includes Pipeline plugin. This plugin used to require a classic jenkins slave as node() to run build step, and this node to host a docker daemon if you want to run anything docker related. With docker slaves, just write your Jenkinsfile without need for a single classic Jenkins executor.
dockerNode("maven:3.3.3-jdk-8") {
sh "mvn -version"
}
How does this compare to docker-pipeline's docker.inside ? There's no need for a host jenkins Node here, nor a local Docker daemon installed on your jenkins slaves. You can rely on docker swarm to scale your build capacity.
Docker-slaves do embrace Pipeline
Last but not least, Jenkins docker-related plugins use to start with a fresh, new slave, which require to run a full SCM checkout and populate project dependencies. Docker-slaves uses a volume as build workspace, so this one is persistent between builds. There is huge room for improvement here, with assistance for docker's volume-drivers, typically to provide snapshot capability so you can browse workspace for a specific build (to diagnose a build failure) or replicate workspace on cluster.
Docker-slaves workspace is persistent
UPDATE:
For some odd reason (who said "conspiracy" ?) docker-slaves 1.0 was not published in update center (yet). Will need to investigate with Jenkins infra team, but with Jenkins World they are all higly busy this week.
@ndeloof w00t ! Best plugin ever !— Damien Duportal (@DamienDuportal) September 14, 2016
9 commentaires:
Great job ! Looks like this is turning to the right direction.
Unfortunatly this doesn't work in my pipeline setup i always get the following error :
java.lang.NoClassDefFoundError: org/jenkinsci/plugins/durabletask/executors/ContinuedTask
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at jenkins.util.AntClassLoader.defineClassFromData(AntClassLoader.java:1139)
at hudson.ClassicPluginStrategy$AntClassLoader2.defineClassFromData(ClassicPluginStrategy.java:884)
at jenkins.util.AntClassLoader.getClassFromStream(AntClassLoader.java:1310)
at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1366)
...
I'm using jenkins 2.7.4 with latest plugins versions. Any idea?
Here is my jenkinsfile, the full stack trace and the versions of plugins : https://gist.github.com/jeanlouisboudart/827123c819e55ae590d23f9d90709c47
Sounds great, I have a question thought.
It relies on docker-common plugins, that means there must be a docker binary accessible so if jenkins is already a docker container it doesn't work yet ?
docker-slaves uses Docker-CLI fro master, so you need this one in PATH.
if master is itself running inside a container, just bind mount docker.sock and add docker binary (from https://github.com/docker/docker/releases/tag/v1.12.1) to your image.
We plan to offer a plain java API alternative asap (to reduce external processes, so threads, inside master)
@Jean-Louis you are running 0.5, need to wait for update center to be rebuilt with latest 1.0
You can download it in the meantime from http://repo.jenkins-ci.org/simple/releases/it/dockins/docker-slaves/1.0/docker-slaves-1.0.hpi
Thanks @ndeloof
@Nicolas it works like a charm with 1.0 :)
I just noticed two small issues :
* When used in a pipeline, if you trigger a job you can can see the pipeline executor running and a few seconds later the "container / slave", perfect but if you try to click on the progress bar of this slave your redirected to the wrong url (/console instead of job/JOBNAME/BUILDNUMBER/console)
* For some reason the log "docker pull" phase is defered (i mean displayed at the end) and it can be disturbing 'cause you have no clue if the job is "stuck" or just downloading stuff
A part from that the plugin looks really promising, i already played with it for a while without pipeline and i can see how poweful it is. I'll need refactor some of my jenkins job now :)
console link issue is not managed by this plugin. I can't see how it could impact it
docker pull run with parallel download, I guess this doesn't well fit into jenkins sequential log mechanism
Yoann and I collaborated on Docker-slaves a year ago during DockerHackDay, with complimentary ideas in mind to re-invent Jenkins in a Docker world. We are happy to have developed a working prototype in 24 hours and won third place in this contest! Continue your fantastic effort! I hope to see more intriguing factoids from your perspective in the future. Thank you so much for your contribution!
motorcycle accident lawyer near me
Enregistrer un commentaire