CI/CD #05. Jenkins: trigger a Pipeline via Git webhook.

We’re setting up a Jenkins “Pipeline script from SCM”, which uses a generic Bash script file to ⓵ create a virtual environment, ⓶ run editable install, and ⓷ run Pytest for all tests. And this Jenkins job can be triggered remotely when we push some file(s) onto the target Git repo.


CI/CD #05. Jenkins: trigger a Pipeline via Git webhook.

This post combines material we’ve gone through in the following posts:

In the above last post (post 4), as a concluding remark, I’ve written:

I started this post using both Pipeline implementations discussed in CI/CD #03. Jenkins: using Pipeline and proper Bash script to run Pytest. I could not get Jenkins to trigger the build.

I still have not been able to find any documentation or tutorial discussing this issue. Through repeated experimentations, I seem to get a hang of it:

🚀 After creating the Jenkins Pipeline, WE MUST FIRST DO a manual build using the ▷ Build Now link.

This behaviour has been consistent during my experimentations. But since I do not have any documentation to back it up, please treat it with caution.

Let’s get to it. The issues we’ll have to attend to are: ⓵ the Bash script, ⓶ the Jenkinsfile, ⓷ the Jenkins Pipeline, ⓸ using ngrok to make our local Jenkins server publicly accessible, ⓹ set up Git webhook, and finally ⓺ test the entire set up.

❶ The Bash script.

The Bash script used in this post has been checked in at https://github.com/behai-nguyen/linux-scripts/blob/main/pytest.sh. We’ve previously discussed this script in this section – please refer to it for set up and usage.

These two are essentially identical. The GitHub version has its comments (documentation) fixed, the codes are identical.

❷ The Jenkinsfile.

We’re also using the https://github.com/behai-nguyen/app-demo.git repo for this new Pipeline, the same Jenkinsfile in this section is also used for this new Pipeline. It has been checked into the repo at
https://github.com/behai-nguyen/app-demo/blob/main/Jenkinsfile.

❸ The Jenkins Pipeline.

As mentioned at the beginning, we’re setting up a Pipeline script from SCM project.

⓵ Log into Jenkins, remove any existing app_demo project.

⓶ Click on + New Item. On the next page, under Enter an item name, enter app_demo – at runtime, Jenkins will create a directory with this name under Jenkins’ work directory, i.e. /var/lib/jenkins/workspace/app_demo.

And /var/lib/jenkins/workspace/app_demo is the value of the WORKSPACE Jenkins environment variable.

⓷ Select Pipeline, then click on the OK button to move to the Configuration page.

⓸ On the Configuration page, for Description, write something meaningful, e.g. CI/CD #05. Jenkins: trigger a Pipeline via Git webhook.

For Build Triggers check GitHub hook trigger for GITScm polling. Without checking this option, Jenkins will not process the push events from Git, the on-screen explanation should sufficiently explain the purpose of this option.

Under the heading Advanced Project Options, click on the Advanced ⌄ button, for Display Name, enter something descriptive, e.g. Jenkins Pipeline and GitHub Webhooks.

Under the heading Pipeline, for Definition, select Pipeline script from SCM.

For SCM, select Git.

Under Repositories, for Repository URL, enter the address of the repo; e.g. https://github.com/behai-nguyen/app-demo.git.

Under Branches to build, for Branch Specifier (blank for 'any'), enter */main – you can try some other branch, I’ve only tested with */main.

Finally, Script Path – leave the default value of Jenkinsfile.

Our new app_demo Pipeline should look like the following screenshots:

Click on the Save button: we’re taken to the app_demo project page.

❹ Using ngrok to make our local Jenkins server publicly accessible.

Please refer to this section of a previous post. It’s exactly the same for this post.

❺ Set up Git webhook for repo https://github.com/behai-nguyen/app-demo.git.

First, if there is any existing webhook on https://github.com/behai-nguyen/app-demo.git, we might need to remove it, since the payload URL might have become invalid, because we terminated ngrok since our last run, etc.

And we’ve have also done this exact same process before. Please refer to this section of a previous post.

❻ Now, we’re testing the Pipeline and hence the entire set up.

⓵ Remove app_demo/ from /var/lib/jenkins/workspace/:

jenkins@hp-pavilion-15:~/workspace$ rm -rf app_demo/

⓶ 🚀 WE MUST DO a manual build first:

– Click on the ▷ Build Now link on the left hand side to do a the first build for this Pipeline.

This build should just pass. Sub-directory app_demo/ should now exist under /var/lib/jenkins/workspace/.

For testing purposes, we’ll again remove app_demo/ from /var/lib/jenkins/workspace/.

The build log should show the following output lines:

...
tests/functional/test_routes.py .....                                    [ 83%]
tests/jenkins_demo/test_push_01.py .                                     [100%]

=============================== warnings summary ===============================
src/app_demo/utils/functions.py:10
  /var/lib/jenkins/workspace/app_demo/src/app_demo/utils/functions.py:10: DeprecationWarning: invalid escape sequence '\p'
    """This method assumes the project has the following layout:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================= 6 passed, 1 warning in 0.05s =========================
...

Note that we have two (2) test modules, and six (6) tests. We’re ready to do automatic tests.

⓷ Add another test module D:\app_demo\tests\jenkins_demo\test_push_02.py, this module has only a single test method:

We’ve gone through this similar testing process in this post CI/CD #04. Jenkins: trigger a Freestyle project via Git webhook. This testing process is the same. We’ll go over the key points briefly, not in as much detail as the mentioned post.

It should now automatically trigger build 2 (two). The build log should show something similar to the following screenshot:

This screenshot shows the new test module, and the total tests increased by 1 (one), to seven (7): 7 passed, 1 warning.

We can see that, except for the Pipeline, every other processes described in this post are, more or less, identical to those in the Freestyle project, which we’ve already covered.

The main difference is:

🚀 After creating the Jenkins Pipeline, WE MUST FIRST DO a manual build using the ▷ Build Now link.

– And as I have stated before, please treat the above conclusion with caution!

✿✿✿

Prior to writing this post, I’ve spent almost two (2) days on this issue. I have carried out numerous tests. Let’s describe some tests, so that you can verify if interested. The tests that I’m going to describe could be regarded as “narrowed down tests” or “focused tests” – by which I mean the tests that I have singled out to prove that we must first do a manual build before auto builds can happen.

Please note, I did not carry out these tests on the https://github.com/behai-nguyen/app-demo.git repo, I don’t want to pollute it with testing commits, I created another repo and played with that.

⓵ Continue on with the environment above intact, stop and start Jenkins. Then do a push: this should trigger another build.

Commands to stop, start and check Jenkins status are:

sudo systemctl stop jenkins
sudo systemctl start jenkins
systemctl status jenkins	

⓶ Remove this Pipeline. Re-create it, but DO NOT RUN a build manually. Do a push: no build happens, even though ngrok receives the push notification with no problem. And Git also reports a successful delivery.

⓷ Continue on from ⓶ – do the first manual build. Then do a push: this should automatically trigger another build.

⓸ Remove app_demo/ from /var/lib/jenkins/workspace/, then push to verify that the Pipeline is responsible for cloning the repo. That is, as long as we did the first manual build, it does not matter if the repo is on disk or not, the Pipeline should always clone on each build. (How else can it get the latest code to run Pytest on?)

⓹ Replace webhook: restart ngrok to get a new URL, remove the existing webhook with the previous URL, then set up a new one with this new URL. Then do a push: this should trigger another automatic build.

That’s about it for this post. I do hope you find the information useful. And please kindly let me know if I’ve made any mistakes. Thank you for reading and stay safe as always.

✿✿✿

Feature image sources: