Students: Bas Holweg, Mike van Steekelenburg & Hans Verstraeten
1.1. Introduction
Promolding is a company specialized in producing high performance polymer products. One of the products it produces is a bundle of needles for tattoos. These tattoos are reference points for radiotherapy operations. The needles inject pigment at a depth of 0,25-0,75 mm compared to normal tattoos of 2-3 mm.
1.2. Assignment
Due to COVID we could not finish our previous project. Luckily we had the opportunity to do this needle project instead. These needles need to be checked whether they are the correct length and not bent. This is called quality control (QC). Promolding is doing this QC manually. This process can be automated and we are taking the first step toward achieving this.
Figure 1 Flowchart of the needle QC
1.3. Solution
Cropping program
Because of COVID it was not possible to physically work with a robot. Therefore we focused on the QC of the assembled needles. The raw footage is not good enough to work with. To clean these up, the images (fig.1) need to be cropped.
Figure 2 Example image of the given dataset
We wrote a program in Python to crop the images automatically. It uses machine vision and a multitude of filters to give a rough estimate where the needles are in the picture. The program then crops the image around the needles so they can be processed further.
Detection
For the detection of the needles we used a deep learning bounding box algorithm called YOLOv5. This is a pre-trained convolutional neural network used for object detection. We started with 326 annotated cropped images. These were augmented with a horizontal flip and blur to create 721 usable images, which were annotated using Roboflow. After 1500 epochs figure 3 is the best result we have. The resulting model has a high accuracy and we are very pleased with our result.
Figure 3 YOLOv5 results on test images
Main program
After the deep learning model has finished detecting the needles, it checks whether the needles are present, too long or too short. This is accomplished in our main python program. We extract the data from the YOLOv5 algorithm and calculate the length of the needle. With this information we can tell if a needle is too long, too short, bent, or not present.
Classifier
For the detection of bent needles we used a deep learning classifier from PyTorch. This is a non-pre-trained convolutional neural network model. The results we got from this process are mixed. This is because we have too few pictures of bent needles. Because of this our deep learning model cannot be trained very accurately with this amount of data. We trained a classifier model and it worked with the test images, but this part of the project can be improved to work in an industrial environment.
Figure 4 Classification code
Square program
The classifying program prefers squared images. Because of this we wrote a program that adds black bars to the image and resizes it to 250×250 pixels. This way, the ratio of the pixels stays the same. This task was relatively easy compared to the other challenges we faced.
1.4. Major decisions
Deep learning algorithm YOLOv5
A major decision we took was using the YOLOv5 algorithm. This algorithm can detect the needles with high precision and we are very proud of how this part of the project turned out.
Automatically cropping the images using machine vision
This was one of the more challenging problems to solve. The dataset exists of images that are mostly full of useless material. We only need a small part of the image (the needles). Furthermore the setup was unstable which caused the images to be shifted in the x and y directions. To solve this we created a reliable vision program that crops the images to the required size and position.
Main program with csv export
The data is exported to a csv file for later use. This will be necessary when the process is fully automated. Should the need arise a worker is able to check the data for a rejected needle.
Figure 5 Results in .csv file
1.5. Conclusion
We look back at four very productive weeks. Because our previous project was cancelled we had to make the most out of these last weeks of the SMR minor. We created a reliable QC-program for the assembled needles for Promolding. During this time we gained a deeper understanding of multiple subjects, like object detection-/classification-deep learning, machine vision and Python programming. We are ecstatic with the results we have achieved in this short period of time. However there is still room for improvement which will be covered in the recommendations chapter.