Ease Your Jenkins Master Node Pains With Remote Agents

Jenkins is great because you can automate anything with it; Jenkins is terrible because you can automate anything with it. Anyone who has spent even a short amount of time wrangling Jenkins will understand that paradox. Configuring Jenkins can be tedious and error prone, and it’s easy to slip into the depths of plug-in hell, as there is a plug-in for everything. And then once you get it configured correctly, it will meet your needs, at least for a little while. Eventually jobs will begin to fail, plug-ins will stop working, those undocumented configuration changes will bite you, Groovy idiosyncrasies inside Jenkinsfiles will annoy you, and the weight of this outdated monolith will keep you up at night thinking about the duct tape holding it all together.

But I’m not here to harp about why Jenkins can be painful, as there are plenty of online resources supporting the pros and cons of Jenkins. I can think of a few CI/CD tools I prefer over Jenkins, but it’s not going away any time soon and it’s quite common to come across organizations running numerous bloated Jenkins instances overloaded with excessive, long-running, concurrent jobs. Scaling up can go past the point of ridiculousness, but other measures should be taken to prevent a total crash and burn. JenkinsX offers a cloud-native solution, but it’s not as easy as simply extending Jenkins pipelines into Jenkin X, and migrating everything to an alternative like GitLab or Azure DevOps might be unfeasible for the organization at the moment. So this is where using Jenkins Agents (also called Jenkins Slaves) come into the picture, to alleviate the pains of a bloated Jenkins Master node, with minimal effort.

There are two notable ways to use Jenkins Agents, which I will explain in detail in the tutorial below:

1. Java Network Launch Protocol (JNLP) connection
2. SSH connection

SSH is the preferred method for using Jenkins Agents. With JNLP, a .jar file executes on the Agent node and talks back to the Master node. If the Java app stops running or the Master node restarts, or something else occurs to break the connection, then the Agent will not be accessible from the Master. SSH is much cleaner and reliable.

I’m using Jenkins running in AWS, so this procedure is geared towards AWS, but you can apply the same basic steps for any other cloud provider. I’m assuming you have some basic AWS EC2 and networking knowledge, know how to use the Linux command line and SSH, and have used Jenkins in the past. So let’s get to it.

JNLP Connection

Step 1 — Set up Jenkins Master instance

Details for setting up Jenkins on an AWS EC2 can be found here. Unless you already have a Jenkins server set up, you will need to follow the instructions to install and unlock Jenkins for its initial use.

Spin up an AWS instance for the Jenkins Master. For both the Master and Agent nodes I used the AWS public image debian-stretch-hvm-x86_64-gp2–2018–11–10–63975

In my case the Master and Agent instances are running in a private subnet with a security group allowing SSH access to the instances and web access to load the Jenkins Master node UI in the browser. You can configure the instances and their networking how you see fit, but you will at least need a security group providing the required access to launch the Jenkins UI and to SSH into the Agent instance.

Step 2 — Set up Jenkins Agent instance

Spin up another AWS instance for the Jenkins Agent. Obviously the more Agent nodes the better, but we’re only setting up one here for simplicity.

SSH into the Agent instance and install the Java JDK. These instructions should work fine. https://docs.datastax.com/en/jdk-install/doc/jdk-install/installOpenJdkDeb.html

Create a folder for the Jenkins workspace on the Agent. I provided full access to that folder but it could be tweaked down appropriately.

$ sudo mkdir /var/lib/jenkins
$ sudo chmod 777 /var/lib/jenkins

Step 3 — Configure Agent Node in Jenkins

Proceed to the URL for the Jenkins Master to load the UI in the browser.

Under Manage Jenkins, choose Configure Global Security to set the TCP port for inbound agents to random.

Under Manage Jenkins, choose Manage Nodes and Clouds to configure the Agent Node connection. The remote root directory must match the directory created in step 2 above and the launch method ‘Launch agent by connecting it to the master’ is for using JNLP to connect to the Agent.

After saving the Agent configuration, you will be given a set of instructions for connecting the Agent to the master. Click on the agent.jar link to download the file locally.

Copy the agent.jar file to the Agent instance using scp or whatever means at your disposal.

$ scp agent.jar david@10.10.12.259:/home/david

Step 4 — Execute agent.jar on Agent instance

Copy the command provided to you after saving the Agent configuration in Jenkins, and execute it on the Agent instance.

$ sudo java -jar agent.jar -jnlpUrl http://<host-name-for-your- jenkins-master-node>/computer/<host-name-for-your-jenkins-agent-node>/slave-agent.jnlp -secret 3da59d99210f736b9b2e1b7dee7e9ad266d67a453d567eae5e4094dab2a8f26d -workDir "/var/lib/jenkins"

If successful, you will see the Agent connected to the Master.

Step 5 — Run a Job in Jenkins to test the Agent node

Back in the Jenkins UI, create a simple job to use the Agent node. In my case I tested with a simple freestyle project executing some small bash script. Just check the box to restrict where the project can run, which in this case was agent-1 to match the label for the Agent Node configuration, and then run the job.

SSH Connection (the better way)

Step 1 — Set up Jenkins Master instance

If not already set up, follow the same steps as step 1 for using JNLP.

Step 2 — Set up Jenkins Agent instance

If not already set up, follow the same steps as step 2 for using JNLP.

After setting up the Agent instance and creating the /var/lib/jenkins folder, create a Jenkins user on the Agent instance.

$ useradd -m -d /home/jenkins jenkins

Then create a key pair for this user.

And then cat the private key file and copy the entire contents to the clipboard to use later in the Jenkins UI.

$ cat ~/.ssh/id_rsa

Step 3 — Create credentials in Jenkins

Proceed to the URL for the Jenkins Master to load the UI in the browser.

Under Manage Jenkins, choose Manage Credentials and then select the credentials store link for global and then choose Add Credentials. Create credentials for ‘SSH Username with private key’ and paste in the private key contents copied from step 2 above.

Step 4 — Configure Agent Node in Jenkins

Under Manage Jenkins, if not already done choose Configure Global Security to set the TCP port for inbound agents to random.

Under Manage Jenkins, choose Manage Nodes and Clouds to configure the Agent Node connection. If you followed along and set up the Agent Node to use JLNP, then be sure to label this Agent Node configuration something different, as Jenkins will allow you to create duplicate labels. The remote root directory must match the directory created on the Agent instance and the launch method should be ‘Launch agents via SSH’. Provide the IP or host name for the Agent instance and select the credential created in Step 3 above. For Host Key Verification Strategy, it’s easier to choose ‘Manually trusted key Verification Strategy’, and then check the box for ‘Require manual verification of initial connection’. Non-verifying Verification Strategy should not be used, as that leaves you vulnerable to a man-in-the-middle attack.

Step 5— Relaunch the Agent in Jenkins

After saving the Agent configuration, relaunch it and select Trust SSH Host Key.

Relaunch the Agent again and you will notice an authentication error. This is expected because the server is rejecting the private key specified in the credentials you set up.

Step 6— Add public key to the authorized_keys on Agent instance

Back in the Agent instance, the public key for the jenkins user created in step 2 above needs to be added to the list of authorized_keys. This will allow the Master node to talk to the Agent node over SSH using the credentials set up in Jenkins.

$ su jenkins
$ cat id_rsa.pub > authorized_keys

Step 7— Relauch the Agent again in Jenkins

Just like in Step 5 above, relaunch the agent, but this time you should see a successful connection.

Step 8— Run a Job in Jenkins to test the Agent node

Create a simple job to use the Agent node. In my case I tested with a simple pipeline script and specified the agent label to use the Agent set up in step 4 above.

Conclusion

Adding Jenkins Agents can provide some relief when a Jenkins Master node is just plain overwhelmed, unreliable, and too fragile. My suggestion is by no means a silver bullet, and there are numerous things that can be done to improve upon what I presented, but I hope you found this article useful and find some sanity when using Jenkins. Happy coding!