MLops Project: Automate the machine learning
Task Overview:
1. Create a container image that has Python3 and Keras or numpy installed using dockerfile.
2. When we launch this image, it should automatically starts train the model in the container.
3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins
4. Job1 : Pull the Github repo automatically when some developers push repo to Github.
5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the software required for the CNN processing).
6. Job3 : Train your model and predict accuracy or metrics. If metrics accuracy is less than 85%, then tweak the machine learning model architecture. Retrain the model and get the train model.
7. Job4: This job sent the notification to developer.
8. Create One extra job job5 for monitor : If container, where app is running, fails due to any reason then this job should automatically start the container again. And also sent a mail to developer.
Project Description:
1. Build Docker images for TensorFlow using Dockerfile:
I use Rhel8 as BaseOs and CentOS for creating my Dockerfile.
- Dockerfile for keras models:
Run command “docker run -t tensorflow_project .” for creating your image.
FROM centos
RUN yum install python3 -y
RUN python3 -m pip — no-cache-dir install — upgrade \pip \setuptools
RUN pip3 install tensorflow
RUN pip install numpy
RUN pip install keras
CMD [“python3”,”/code/project.py”]
2. Jobs in Jenkins:
- Job1(Copy github code):
I use MNIST dataset to deploy this model. You can check the code from here…
Whenever developer push any code in Github, this job automatically detect and copy in host OS.
This job creates a directory mlops and copy the GitHub code in that directory.
- Job2(Deploy container for model_train):
If my job1 is successfully built, it triggers job2 and launches the container. By looking at the code or program file, this job automatically launch the respective docker container(either Keras os Sklearn).
if sudo cat /root/devops/project.py | grep keras
then
if sudo docker ps -a | grep os
then
epoch=1
layer=1
sudo docker rm -f os
sudo docker run -t -v /root/devops:/code — name os tensorflow_project /code/project.py -e’epoch’ -l’layer’
else
epoch=1
layer=1
sudo docker run -t -v /root/devops:/code — name os tensorflow_project /code/project.py -e$epoch -l$layer
fi
fi
- Job3(Check the accuracy and tweak the code and again run until it found the required accuracy):
This job check the accuracy of model which I trained in job2 and if accuracy is below from required, this do some changes in code and run again the container to find until required accuracy.
accuracy=$(sudo cat /root/devops/accuracy.txt)
final_accuracy=85
compare=$(echo “$accuracy > $final_accuracy” | bc )
epoch=1
layer=1
while [[ $compare != 1 ]]
do
let epoch+=1
let layer+=1
sudo docker rm -f os
sudo docker run -t -v /root/devops:/code — name os tensorflow_project /code/project.py -e$epoch -l$layer
accuracy=$(sudo cat /root/devops/accuracy.txt)
compare=$(echo “$accuracy > $final_accuracy” | bc )
done
If anything goes wrong, it triggers to job5.
- Job4(This job sent a mail to the developer for successful train of model):
After getting the required accuracy, it sent a mail to developer.
import smtplib
s = smtplib.SMTP(‘smtp.gmail.com’, 587)
s.starttls()
s.login(“sender_email”, “password”)
message = “Hey Developer, Finally we got the model trained. “
s.sendmail(“sender_mail”, “developr_mail”, message)
s.quit()
- Job5(This job is for monitoring the job2 and job3, if any container failed, it rebuild the container):
This job relaunch the container, if any of container get failed and also sent a email for failure the job.
import smtplib
s = smtplib.SMTP(‘smtp.gmail.com’, 587)
s.starttls()
s.login(“sender-mail”, “password”)
message = “Hey Developer, you need to check your code once. Might be this have some error. “
s.sendmail(“sender-mail”, “developer_mail”, message)
s.quit()
