Skip to content

Ultimate Guide to Deploying a Jenkins Multinode Cluster on AWS with Linux and Windows Slave Nodes

Avatar photo

https://www.linkedin.com/in/kumar-nikhil811/

Jenkins is an open-source automation server that helps automate parts of the software development process, including building, testing, and deploying software. It provides a platform for continuous integration and delivery, allowing developers to quickly and easily build, test, and deploy their code. Jenkins can help improve software quality, reduce manual errors, and increase productivity for development teams.

Benefits of Using Jenkins

  1. Continuous Integration: Jenkins automates the process of building and testing code changes as soon as they are committed to a shared repository. This helps identify and fix issues early in the development cycle, reducing the risk of bugs and errors in the final product.
  2. Continuous Delivery: Jenkins automates the deployment of software to production, making it faster and easier to release new features and updates. This helps organizations stay competitive by delivering new functionality to customers more quickly.
  3. Customizable: Jenkins provides a vast library of plugins and integrations that allow developers to customize the platform to their specific needs. This includes plugins for source control management, testing frameworks, and deployment tools.
  4. Open-Source: Jenkins is free and open-source software, meaning that anyone can use, modify, and distribute it without any licensing fees. This makes Jenkins accessible to small and large organizations alike and helps foster a vibrant community of contributors who can improve and enhance the platform over time.

Deploying Jenkins Multinode Cluster on AWS

Jenkins Multinode Cluster Design

In this we are going to create 3 instances, of which one will be a master node on which Jenkins is deployed for this am using a Linux instance and the other two will be windows and Linux, both will be slave nodes that will be attached to the master.

Here I have already deployed three AWS instances two Linux and one Windows

AWS instances

Steps to deploy Jenkins on Linux VM:

  1. First, we need to configure the yum repository to install Jenkins and the java package, using the below command:

these commands will configure the yum repository and import the rpm keys.

[root@ip-172-31-53-78 ~]# wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
--2023-03-19 09:34:47--  https://pkg.jenkins.io/redhat-stable/jenkins.repo
Resolving pkg.jenkins.io (pkg.jenkins.io)... 146.75.34.133, 2a04:4e42:78::645
Connecting to pkg.jenkins.io (pkg.jenkins.io)|146.75.34.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 85
Saving to: '/etc/yum.repos.d/jenkins.repo'

/etc/yum.repos.d/jenkins.repo         100%[=========================================================================>]      85  --.-KB/s    in 0s

2023-03-19 09:34:47 (5.42 MB/s) - '/etc/yum.repos.d/jenkins.repo' saved [85/85]

[root@ip-172-31-53-78 ~]# rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
[root@ip-172-31-53-78 ~]# cat /etc/yum.repos.d/jenkins.repo
[jenkins]
name=Jenkins-stable
baseurl=http://pkg.jenkins.io/redhat-stable
gpgcheck=1
[root@ip-172-31-53-78 ~]#

2. Once, yum got configured we can install java (openjdk) and Jenkins Package. We must need java installed on all the nodes whether it’s a master or slave node as Jenkins is developed using Java.

[root@ip-172-31-53-78 ~]# yum install fontconfig java-11-openjdk
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Jenkins-stable                                                                                                           90 kB/s |  26 kB     00:00
Dependencies resolved.
========================================================================================================================================================
 Package                                      Architecture     Version                                       Repository                            Size
========================================================================================================================================================
Installing:
 fontconfig                                   x86_64           2.14.0-2.el9_1                                rhel-9-appstream-rhui-rpms           301 k
 java-11-openjdk                              x86_64           1:11.0.18.0.10-2.el9_1                        rhel-9-appstream-rhui-rpms           443 k
Installing dependencies:
 ModemManager-glib                            x86_64           1.18.2-3.el9                                  rhel-9-baseos-rhui-rpms              304 k
 adwaita-cursor-theme                         noarch           40.1.1-3.el9                                  rhel-9-appstream-rhui-rpms           655 k
 adwaita-icon-theme                           noarch           40.1.1-3.el9                                  rhel-9-appstream-rhui-rpms            12 M
 alsa-lib                                     x86_64           1.2.7.2-1.el9                                 rhel-9-appstream-rhui-rpms           522 k
 at-spi2-atk                                  x86_64           2.38.0-4.el9  
                                rhel-9-appstream-rhui-rpms            90 k
[root@ip-172-31-53-78 ~]# yum install jenkins -y
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:01:49 ago on Sun 19 Mar 2023 09:38:09 AM UTC.
Dependencies resolved.
========================================================================================================================================================
 Package                            Architecture                      Version                                  Repository                          Size
========================================================================================================================================================
Installing:
 jenkins                            noarch                            2.387.1-1.1                              jenkins                             94 M

Transaction Summary
========================================================================================================================================================
Install  1 Package

Total download size: 94 M
Installed size: 94 M
Downloading Packages:
jenkins-2.387.1-1.1.noarch.rpm                                                                                           11 MB/s |  94 MB     00:08
--------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                    11 MB/s |  94 MB     00:08
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                1/1
  Running scriptlet: jenkins-2.387.1-1.1.noarch                                                                                                     1/1
  Installing       : jenkins-2.387.1-1.1.noarch                                                                                                     1/1
  Running scriptlet: jenkins-2.387.1-1.1.noarch                                                                                                     1/1
  Verifying        : jenkins-2.387.1-1.1.noarch                                                                                                     1/1
Installed products updated.

Installed:
  jenkins-2.387.1-1.1.noarch

Complete!
[root@ip-172-31-53-78 ~]#

To verify whether java is installed or not. We can use the below command

[root@ip-172-31-53-78 ~]# java --version
openjdk 11.0.18 2023-01-17 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.18.0.10-2.el9_1) (build 11.0.18+10-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.18.0.10-2.el9_1) (build 11.0.18+10-LTS, mixed mode, sharing)
[root@ip-172-31-53-78 ~]#

Once the java and Jenkins got installed we need to start the Jenkins service

[root@ip-172-31-53-78 ~]# systemctl start jenkins
[root@ip-172-31-53-78 ~]# systemctl status jenkins
● jenkins.service - Jenkins Continuous Integration Server
     Loaded: loaded (/usr/lib/systemd/system/jenkins.service; disabled; vendor preset: disabled)
     Active: active (running) since Sun 2023-03-19 09:44:05 UTC; 13s ago
   Main PID: 11896 (java)
      Tasks: 43 (limit: 5704)
     Memory: 347.1M
        CPU: 56.627s
     CGroup: /system.slice/jenkins.service
             └─11896 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080

Mar 19 09:43:18 ip-172-31-53-78.ec2.internal jenkins[11896]: a1283d0925c64152a2a907d84bd0bd19
Mar 19 09:43:18 ip-172-31-53-78.ec2.internal jenkins[11896]: This may also be found at: /var/lib/jenkins/secrets/initialAdminPassword
Mar 19 09:43:18 ip-172-31-53-78.ec2.internal jenkins[11896]: *************************************************************
Mar 19 09:43:18 ip-172-31-53-78.ec2.internal jenkins[11896]: *************************************************************
Mar 19 09:43:18 ip-172-31-53-78.ec2.internal jenkins[11896]: *************************************************************
Mar 19 09:44:05 ip-172-31-53-78.ec2.internal jenkins[11896]: 2023-03-19 09:44:05.602+0000 [id=29]        INFO        jenkins.InitReactorRunner$1#onAtta>
Mar 19 09:44:05 ip-172-31-53-78.ec2.internal jenkins[11896]: 2023-03-19 09:44:05.639+0000 [id=22]        INFO        hudson.lifecycle.Lifecycle#onReady>
Mar 19 09:44:05 ip-172-31-53-78.ec2.internal systemd[1]: Started Jenkins Continuous Integration Server.
Mar 19 09:44:05 ip-172-31-53-78.ec2.internal jenkins[11896]: 2023-03-19 09:44:05.742+0000 [id=44]        INFO        h.m.DownloadService$Downloadable#l>
Mar 19 09:44:05 ip-172-31-53-78.ec2.internal jenkins[11896]: 2023-03-19 09:44:05.743+0000 [id=44]        INFO        hudson.util.Retrier#start: Perform>
[root@ip-172-31-53-78 ~]#

To connect with Jenkins Server we can use any browser and Jenkins uses Port 8080 by default

Jenkins is up and running, we are able to connect with Jenkins but we need to pick the Admin password from the location which is mentioned by Jenkins

[root@ip-172-31-53-78 ~]# cat /var/lib/jenkins/secrets/initialAdminPassword
a1283d0925c64152a2a907d84bd0bd19
[root@ip-172-31-53-78 ~]#

By using this Credential we can log in to the Jenkins server and it will ask you to set up an account

Now, Jenkins Master is deployed.

To Add the Linux node as a slave node in the cluster we need to go to “manage Jenkins”, then “Manage Nodes and Cloud”

Here Master node information is shown and to add a new node need to click on the “New Node” option

We need to provide the node name and click on create then we need to provide information number of executors, Remote root directory, launch method etc.

Java needs to be installed on the Linux node as Jenkins will send a java agent which will be launched on the slave node to establish a connection and performs an action on the slave node.

In the above screenshots, Jenkins sent a Java agent on the Linux node to establish a connection, Once a node got connected it will show like this in Jenkins master.

Now Linux node is connected to the master and it has two executors just like the master.

Executors mean how many jobs can be run on the node parallelly so now both the nodes master and slave node can run 2 jobs each.

Now, it’s time to attach the Windows node. Before adding the node let me explain to you the use of adding two different OS nodes in the Jenkins cluster.

Sometimes there might be a use case in which we need a windows machine to perform some action for that kind of action we can’t perform on Linux nodes so we can bind any job to that specific node to perform the action.

To add a windows node we need to go into manage Jenkins and manage node & cloud just like we did for the Linux node but we need to change the launch method for Linux SSH will be used to transfer the agent but in the case of windows we can’t use SSH so for that we need to use JNLP protocol and we need to manually run the command on windows node to connect it with the master.

For windows, I have selected 3 executors and the launch method will be “Launch agent by connecting it to the controller

Once we will click on save it will show us the command to download the agent and launch it on the windows node on windows also we need to install Java before running these commands.

windows node command
C:\Users\Administrator>java -version
openjdk version "11.0.18" 2023-01-17
OpenJDK Runtime Environment OpenLogic-OpenJDK (build 11.0.18+10-adhoc..jdk11u)
OpenJDK 64-Bit Server VM OpenLogic-OpenJDK (build 11.0.18+10-adhoc..jdk11u, mixed mode)

C:\Users\Administrator>curl -sO http://54.237.19.140:8080/jnlpJars/agent.jar

C:\Users\Administrator>java -jar agent.jar -jnlpUrl http://54.237.19.140:8080/manage/computer/windowsnodes/jenkins-agent.jnlp -workDir "/jenkinws"
Mar 19, 2023 10:23:13 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using \jenkinws\remoting as a remoting work directory
Mar 19, 2023 10:23:13 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to \jenkinws\remoting
Mar 19, 2023 10:23:14 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: windowsnodes
Mar 19, 2023 10:23:14 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3107.v665000b_51092
Mar 19, 2023 10:23:14 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using C:\jenkinws\remoting as a remoting work directory
Mar 19, 2023 10:23:14 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://54.237.19.140:8080/]
Mar 19, 2023 10:23:14 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Mar 19, 2023 10:23:14 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: 54.237.19.140
  Agent port:    38871
  Identity:      ff:94:99:ff:3b:ba:77:4f:74:54:7a:a9:63:ca:0a:06
Mar 19, 2023 10:23:14 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Mar 19, 2023 10:23:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 54.237.19.140:38871
Mar 19, 2023 10:23:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Mar 19, 2023 10:23:15 AM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
INFO: Waiting for ProtocolStack to start.
Mar 19, 2023 10:23:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: ff:94:99:ff:3b:ba:77:4f:74:54:7a:a9:63:ca:0a:06
Mar 19, 2023 10:23:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

It shows node is now connected with the Jenkins master.

Now all the nodes are connected and we can see different executors for each node on the left side.

It’s time to test the cluster by running different jobs on each 

Here am creating a freestyle job for testing the Linux node.

Here am restricting the job to run on only the Linux node.

adding some shell commands to test the job

In the same way, I am creating a job for windows that will be restricted to running on windows node only and there, I will run the windows batch commands.

Now both the jobs are ready and will trigger at the same time and both jobs will be running on their respective nodes.

Here we can see both jobs are running on their respective restricted node.

Console output of job which ran on Linux node

Linux job output

In the above screenshot, we can verify that this job runs on a Linux node

Windows job output

This job got failed timeout command got failed.

Here is the windows command output and we can see that it also runs on the windows node only.

Advantages of a multimode cluster of Jenkins:

  • A Jenkins multinode cluster with Linux and Windows OS provides greater flexibility for developers to build, test and deploy software across a wider range of environments.
  • By adding more nodes to the Jenkins cluster, teams can increase their computing power and speed up build and deployment times, leading to faster releases and improved productivity.
  • With both Linux and Windows nodes in the cluster, developers can build and test their code on multiple platforms simultaneously, which helps ensure that their software works correctly across different environments.
  • The cluster architecture also provides greater reliability and fault tolerance, as the workload can be distributed across multiple nodes, reducing the risk of downtime or failure.

By deploying a Jenkins multinode cluster with Linux and Windows slave nodes in AWS, development teams can achieve faster release cycles, improved software quality, and greater flexibility in building and testing their code. With Jenkins, teams can automate their software delivery pipeline and optimize their development process, helping them stay competitive in today’s fast-paced technology landscape

Disclaimer:

The views expressed and the content shared in all published articles on this website are solely those of the respective authors, and they do not necessarily reflect the views of the author’s employer or the techbeatly platform. We strive to ensure the accuracy and validity of the content published on our website. However, we cannot guarantee the absolute correctness or completeness of the information provided. It is the responsibility of the readers and users of this website to verify the accuracy and appropriateness of any information or opinions expressed within the articles. If you come across any content that you believe to be incorrect or invalid, please contact us immediately so that we can address the issue promptly.

Tags:

Avatar photo


https://www.linkedin.com/in/kumar-nikhil811/
Nikhil Kumar is a DevOps Engineer with 5years of experience in the field. Alongside a successful career in technology, he has also cultivated a passion for writing, having authored several articles and blogs on the subjects of DevOps and the Cloud. With a keen interest in exploring the intersection of technology and the written word, he brings a unique perspective to the conversation.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.