Integration of Machine Learning with DevOps

Github, Docker, and Jenkins

Here in this rapidly moving world, time is the most precious thing. The world of IT is a strict follower optimization and automation with pace, so to fulfill the acquisitiveness of time — DevOps came in play, vis-a-vis DevOps, it is a set of practices that works to automate and integrate the processes between software development.

After going through a few days of Machine Learning training integrating with DevOps. The vision of automation with reliability is getting the wings of fire.

JOB 1

In this project, we are making a fully automated chain of jobs to train a perfect Machine Learning model with the Jenkins.

Inside my RHEL 8 system, I created a Dockerfile, to create my own customized Docker images. The launched image has the environment to train the Machine Learning and Deep Learning models, Ex.

Numpy, Pandas, Tensor Flow, Kera

(On Jenkins)

JOB 2 —

1. Job 2 in Jenkins will pull the Github repository and copy the code to the /mlops1 directory in my base RedHat system.

JOB 3 —

1. This job will launch a container over Docker from the Docker Image that, I created initially, called mlopsimages.

1.a. First, we check the code type of main model file whether it is a Machine Learning model or not.

1.b. If the file is Machine Learning/Deep Learning model then we check whether the container named mltensorflow exist or not, if it exists then we print already exist and if not then we launch a container named mltensorflow over the Docker using our framed Docker Image.

1.c. The /mlops1 directory mounted with the /mlopsproject directory inside the container build inside it already.

JOB 4 —

JOB 4

1. Then we will run the mainmodel1.py file pulled from Github.
2. Also connected the trigger for the job mlops 3, it will run only when mlops 2 runs successfully, also called Upstream.

Code is:

sudo docker exec mltensorflow python3 /mlopsproject/mainmodel1.py

Epoch is in process and model get trained with accuracy op[o=op=p[oh=p[ohhhh=uiopiupo=[.99%

JOB 5 —

1. In this job, the task is to increase the accuracy of the model.

2. After model trained we saved its accuracy percentage in accuracy.txt

3. Using if condition we checked whether the accuracy is above 95 percent, if so then mlops 5 job will be triggered, otherwise, using the while loop we keep increasing the number of epochs and convolution layers, until the desired accuracy of 95% is achieved.

4. Finally, mlops 6 triggered.

JOB 6 —

1. Finally, this job tells that the model is good.
2. We will get notifications using email or notifications on slack.

Final Job

Setup for receiving mail

Github Triggers-

After creating a public IP using ngrok, we triggered a hook on Github, to trigger the mlops 1 as soon as we commit

Git Bash Automation —

Automating the push file command to deploy the file from the local repository to Github, using a bash file, known as post-commit.

Here post-commit file created for push automation.

Final Build Pipeline —

Build Pipeline Model vis-a-vis the jobs.

Github Link:

utkarsh161099/mlops

STRUCTURE OF JOB TO BE PERFORM

Conclusion

The DevOps is a culture, that is very beneficial for the developers either talk about accuracy, optimization, or talk about automation. This project taught us the industry use cases of DevOps to a great extent. Here our job is only to commit our file on the Github, the rest job is fully automated- remaining job Jenkins could do itself.

Deep gratitude to Vimal Daga Sir, for his beautiful steps towards making India Future Ready…