How to Create Dynamic Stages in a Jenkins Pipeline

1. Overview

Jenkins pipelines have become an indispensable tool for automating software delivery processes.

Firstly, let’s consider why dynamic stages are crucial. In traditional Jenkins pipelines, we often define static stages that remain constant across different builds. However, as projects grow into complexity, we frequently encounter scenarios where we need the pipeline to adapt on the fly. This is where dynamic stages come into play. By leveraging dynamic stages, we can create pipelines that adjust their structure and behavior on various factors, such as Git branches, configuration files, or even external system inputs.

In this tutorial, we’ll look at the process of implementing dynamic stages in a Jenkins pipeline.

2. Understanding Jenkins Pipelines

Before we go into how we can create dynamic stages, it’s essential we have a solid grasp of Jenkins pipelines in general. Jenkins pipelines provide a powerful way to define the entire build process as code, enabling greater flexibility and version control.

2.1. Basic Pipeline Structure

At its core, a Jenkins pipeline is defined in a file called Jenkinsfile, which is typically stored in the project’s source code repository. Moreover, this file uses a Domain Specific Language (DSL) based on Groovy. The basic structure of a pipeline consists of stages, which in turn contain steps.

Let’s see an example:

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                echo 'Building the project...'
            }
        }
        stage('Test') {
            steps {
                echo 'Running tests...'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying the application...'
            }
        }
    }
}

In the code snippet above, we can see three stages:

build
test
deploy

Each stage contains a single step that simply echoes a message.

2.2. Static and Dynamic Stages

Now that we understand the basic structure, let’s explore the difference between static and dynamic stages.

On one hand, static stages are predefined and remain constant for every pipeline run. They’re straightforward to implement and work well for simple, consistent workflows. However, they lack flexibility when the build process needs to adapt to different scenarios.

On the other hand, dynamic stages are generated at runtime based on various conditions or inputs. This approach offers several advantages:

flexibility: we can create stages based on the current state of the project or environment
reusability: we can generate similar stages without duplicating code
maintainability: changes to the stage generation logic can be made in one place, affecting multiple stages

For instance, we might create stages based on which Git branches have changes, or generate test stages dynamically based on the types of tests detected in a project.

3. Setting Up the Environment

Before we proceed, it’s crucial that we have the environment properly configured. Let’s walk through the necessary steps to ensure we’re ready to start implementing the dynamic Jenkins pipelines.

3.1. Jenkins Installation and Configuration

First and foremost, we need to have Jenkins installed and running on the system. If not already deployed, we can download Jenkins from the official website and follow the installation instructions for the current operating system.

At this point, once Jenkins is up and running, let’s navigate to the Jenkins dashboard. Here, we ensure that we have the necessary permissions to create and modify pipelines. Typically, we want to log in with an admin account or a user account with sufficient privileges.

3.2. Required Plugins

To work effectively with dynamic stages, we install a few essential plugins. These plugins extend the functionality of Jenkins and provide us with the tools we need.

Let’s navigate to Manage Jenkins > Manage Plugins and ensure we have the required plugins installed:

Pipeline
Pipeline: Stage View
Git (if working with Git repositories)

To install these plugins, we can search for them in the Available tab and select each for installation. Once installed, we restart Jenkins to apply the changes.

3.3. Verification

At this point, let’s create a simple pipeline to test the setup. We can do this by creating a new item in the Jenkins dashboard and selecting Pipeline as the project type.

Additionally, in the pipeline configuration, we can add a simple test script:

pipeline {
    agent any
    stages {
        stage('Test Setup') {
            steps {
                echo 'Our Jenkins pipeline setup is working!'
            }
        }
    }
}

After saving and running this pipeline, we see a successful build with a test message.

Moreover, with the environment now set up and tested, we can move on to the exciting part, how to create dynamic stages in a Jenkins pipeline.

4. Creating Dynamic Stages

Dynamic stages enable us to adapt a pipeline structure based on various conditions or inputs, making the CICD process more flexible and efficient.

4.1. Using Groovy Scripts

To begin with, one of the most powerful ways to create dynamic stages is by leveraging Groovy scripts within the Jenkinsfile. The flexibility of Groovy enables us to generate stages programmatically based on any logic we define.

Let’s start with a simple example where we generate stages based on a list of environments:

def environments = ['dev', 'qa', 'staging', 'production']

pipeline {
    agent any
    stages {
        stage('Dynamic Deployment') {
            steps {
                script {
                    environments.each { env ->
                        stage("Deploy to ${env}") {
                            echo "Deploying to ${env} environment"
                        }
                    }
                }
            }
        }
    }
}

In this example, we iterate over a list of environments and create a deployment stage for each one. This approach enables us to easily add or remove environments without modifying the pipeline structure.

4.2. Leveraging Jenkins DSL

While Groovy scripts offer great flexibility, Jenkins also provides a Domain Specific Language (DSL) that allows us to create dynamic stages. Moreover, the Jenkins DSL offers a more declarative approach to pipeline creation, which can be easier to read and maintain.

Here’s an example of how we can use the Jenkins DSL to create dynamic parallel stages:

pipeline {
    agent any
    stages {
        stage('Parallel Tests') {
            parallel {
                stage('Unit Tests') {
                    steps {
                        echo 'Running unit tests...'
                    }
                }
                stage('Integration Tests') {
                    steps {
                        echo 'Running integration tests...'
                    }
                }
                stage('UI Tests') {
                    steps {
                        echo 'Running UI tests...'
                    }
                }
            }
        }
    }
}

In this case, we define parallel stages for different types of tests. While this example uses static stages, we can combine the approach with Groovy scripts to generate these parallel stages dynamically.

For instance, we can read a configuration file to determine which types of tests to run:

def testTypes = readYaml file: 'test-config.yaml'

pipeline {
    agent any
    stages {
        stage('Parallel Tests') {
            steps {
                script {
                    def parallelStages = [:]
                    testTypes.each { test ->
                        parallelStages["${test.name}"] = {
                            echo "Running ${test.name}..."
                            // Add actual test execution here
                        }
                    }
                    parallel parallelStages
                }
            }
        }
    }
}

In this more advanced example, we read a YAML configuration file to determine which test stages to create. We then dynamically generate parallel stages based on this configuration.

Thus, by combining these techniques, we can create highly flexible and adaptable pipelines.

5. Implementing Common Use Cases

Now that we’ve covered the basics of creating dynamic stages, let’s explore some common scenarios where dynamic stages can significantly enhance a Jenkins pipeline. In particular, these practical examples demonstrate how we can apply these techniques to real-world situations.

5.1. Generating Stages Based on Git Branches

One of the most useful applications of dynamic stages is adapting a pipeline based on the Git branch being built. This way, we can create different workflows for feature branches, hotfixes, and main branches.

Let’s see an example of how we can achieve this:

pipeline {
    agent any
    stages {
        stage('Determine Build Type') {
            steps {
                script {
                    if (env.BRANCH_NAME == 'main') {
                        stage('Production Build') {
                            echo 'Building for production...'
                        }
                        stage('Deploy to Production') {
                            echo 'Deploying to production...'
                        }
                    } else if (env.BRANCH_NAME.startsWith('feature/')) {
                        stage('Feature Build') {
                            echo 'Building feature...'
                        }
                        stage('Deploy to Dev') {
                            echo 'Deploying to dev environment...'
                        }
                    } else if (env.BRANCH_NAME.startsWith('hotfix/')) {
                        stage('Hotfix Build') {
                            echo 'Building hotfix...'
                        }
                        stage('Deploy to Staging') {
                            echo 'Deploying to staging for urgent testing...'
                        }
                    }
                }
            }
        }
    }
}

Here, we use conditional logic to create different stages based on the branch name. Thus, we have a single Jenkinsfile that adapts to different scenarios.

5.2. Creating Stages from Configuration Files

Another powerful technique is generating stages based on external configuration files. This approach enables us to modify a pipeline structure without changing the Jenkinsfile itself.

Let’s consider a scenario where we have a YAML file defining build steps:

# build-config.yaml
steps:
  - name: Compile
    command: mvn compile
  - name: Test
    command: mvn test
  - name: Package
    command: mvn package

Furthermore, we can then use this configuration to dynamically generate pipeline stages:

def buildConfig = readYaml file: 'build-config.yaml'

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                script {
                    buildConfig.steps.each { step ->
                        stage(step.name) {
                            sh step.command
                        }
                    }
                }
            }
        }
    }
}

This approach gives us the flexibility to easily modify a build process by updating the YAML file, without the need to change the Jenkinsfile.

5.3. Dynamically Parallel Stages

Lastly, let’s explore how we can create dynamic parallel stages. This technique is particularly useful when we want to run multiple independent tasks simultaneously, improving an overall build time.

Let’s see an example where we dynamically create parallel stages for different test suites:

def testSuites = ['unit', 'integration', 'e2e']

pipeline {
    agent any
    stages {
        stage('Parallel Tests') {
            steps {
                script {
                    def parallelStages = testSuites.collectEntries {
                        ["${it.capitalize()} Tests" : {
                            stage("Running ${it} tests") {
                                echo "Executing ${it} test suite..."
                                // Add actual test execution here
                            }
                        }]
                    }
                    parallel parallelStages
                }
            }
        }
    }
}

In this case, we dynamically create parallel stages for each test suite in the testSuites list. This way, we can easily add or remove test suites by modifying the list, without changing the pipeline structure.

At this point, by implementing common use cases, we can create highly flexible and efficient Jenkins pipelines that adapt to the project’s specific needs.

6. Conclusion

In this article, we’ve explored the powerful concept of dynamic stages in Jenkins pipelines. We started by understanding the basics of Jenkins pipelines and the distinction between static and dynamic stages. Then, we delved into the practical aspect of creating dynamic stages using Groovy scripts and Jenkins DSL.

Furthermore, we examined several common use cases for dynamic stages, including generating stages based on Git branches, creating stages from configuration files, and implementing dynamic parallel stages. These techniques demonstrate how we can significantly enhance our CICD workflows by making them more flexible and adaptable to various scenarios.

Full Archive

About Baeldung