Students: Jarik Okkerman, Jochem Vliegenthart, Mart Reumer, Vincent van den Nouland
Introduction
Dero is a company which is situated in Nieuw-Vennep. It is a company which specializes in automation, mostly in the food sector, but also in post & parcel and end-of-line logistics. For one of their customers, they are looking into automating the packaging of fresh packets.
Currently these fresh meal boxes from grocery stores are hand-assembled on a production line. This requires between five and six people to work a production line. The customer wishes to automate this process and cut back on personnel.
Problem
All the fresh packets contain different ingredients. From bags to packages to natural products. These should all be picked up and placed in the carton boxes. In addition, the vegetables must be checked for unsightly spots/mould. (Quality inspection).
In this project we have decided to first tackle the top ten most sold fresh packets. Also, we left the quality control out of scope and decided to focus on the recognition and picking of the vegetables.
Solution
The solution we created a prototype consisting of two Intel RealSense D415 camera’s which provide an RGB image of the vegetables as well as a depth image. A Techman Robot TM12, which is used to pick and place the vegetables. And a small conveyor belt to feed the system the empty carton boxes.
Vision
To recognise the vegetables and determine a point for the robot to pick up a combination of the RGB and depth image from the RealSense camera’s is used. For each vegetable a colour range is defined in a database, this range is used to create a mask to filter the RGB image for the vegetable that is being searched. After this the centre of mass of each vegetable is found the pixel locations are saved. For vegetables with stem the pickup point is chosen on a spot where the suction cup can create a vacuum. The system then checks in the depth image what the height is for each found vegetable. Then the pixel coordinates of the vegetable that’s the highest are translated to real world coordinates which are sent to the robot.
Robot
The robot then moves to this location and picks up the vegetable using a pneumatic suction cup. After this the robot verifies if it has made a successful pick by checking the pneumatic tubes if there is indeed a vacuum. If this is not the case the system will request another pickup point from the vision software and try again. After the package is filled the robot gives a signal to the conveyor belt and loads a new and empty carton and the cycle begins a new.
Recommendations
While the vision system works, it is quite inflexible. In order to recognise the vegetables the lighting conditions need to be really close to the original conditions in our test setup. And the fact that the system relies on colour masks does not help because vegetables are natural products and therefore vary in colour. We would recommend training machine learning models for each vegetable. This way the system should be very good in detecting a lot of different variants of one type of vegetable. Also this would make it possible to perform quality checks on the vegetables.
Also we would suggest using a deltapicker instead of a robot arm. Deltapickers are very light and can reach much higher velocities. This would drastically improve the picks per minute to match the speed of human pickers.
