Students: Niels Westerduin, Mark van der Lee, Matthias van Zweeden, Erik Donkersloot
For questions please contact us:
applebinpicking@gmail.com
Business case
The assignment for the project came from Lacquey BV and is done at The Hague University of Applied Sciences.
Lacquey
Lacquey BV is a daughter company of FTNON BV and delivers complete robot systems for the fresh food industry. Lacquey is specialised in robotic food handling solutions and gripper technology.
Problem
Lacquey wants to develop a robot system for bin-picking apples directly from a crate. Previous attempt with the use of a stereo vision 3D camera did not give the desired results because the system was too inaccurate. It was not possible to get a proper 3D image from all the products in a crate from one point of view. Especially situations with a large height difference gave a distorted image.
Because of this Lacquey now wants to try to make a 3D image with laser triangulation. This technique uses a line laser in order to determine the height of an object. Moving the crate is undesirable because products can start rolling because of the acceleration.
The main goal of the project is to use laser triangulation to get an accurate 3D model of apples in a crate. From this 3D image the separate apples have to be identified in order to pick them up. The actual picking of the fruit is not the main goal of Lacquey for the project but it is included for demonstration purposes. After each picking movement the crate has to be scanned again because the 3D image will change.
Collaborations
Besides Lacquey there are other companies involved in the project too. Lacquey is the client for the project and provides different components. The most important component Lacquey provides is their gripper. This gripper combines a mechanical gripper with pneumatic suction cups. Furthermore, Lacquey provides the laser and the encoder which are necessary for the vision system. For the laser triangulation a special camera is used. This camera is provided by Sick BV. The provided Color Ranger camera as a custom sensor which is able to make a 3D image and a color image on the same sensor. A lens for this camera was provided by VS-technology. The vision software made by the project group is made in Halcon from MVtec. This software was provided for us by Batenburg Datavision.
Figure 1: Vision system with camera and laser
System
The system consists of a scanning setup and a robot. The software made in Halcon has to run in Visual Studio, therefore a PC is used. To make a 3D image, laser triangulation is used. A laser line and a camera are pointed to an object with a relative angle. In the sensor image the laser line has an offse determined by the height of the object. Because of this a 3D image can be made.
Figure 2: Laser triangulation
Mechanical solution
Because the crate has to be stationary and the 3D image is made with a laser line the vision system has to move. Therefore a gantry system is made which provides a linear movement for the vision setup.
Electrical components
The electrical components are best explained using the diagram:
Figure 3: control flow
On a HMI (touchscreen Human Machine Interface) the system can be started. It sends a signal to the PLC which then controls the laser and the motor controller. The motor controller is an Arduino which controls the stepper motor for the linear movement. The PLC also tells the vision software to run. The software, which runs on a PC, communicates with the camera and gives an output on the screen. This software determines the location of the which apple it wants to pick and sends this information to the PLC. The PLC on its turn uses these coordinates for the robot program so the robot can pick an apple from the crate and put it in the correct location. All control components like the PLC and Arduino are placed in a control box.
Figure 4: Control box
Results
The gantry system provides a stable linear movement system so the crate can be stationary. With this system accurate 3D images of the crate can be made. The vision software can detect apples from the scans and determine which apple has to be picked up. Also the gripper orientation is determined in the software to prevent a collision with the crate or other apples. For the robot the software generates the position for the apple it has to pick and the orientation of the gripper. The robot can place the apple in a predetermined location. Because of the color images which are also made with the camera different apples can be sorted.
Figure 5: Picking system
Conclusion
The main goal of this project was to make a 3D scan with color information of a crate with apples. This goal is accomplished. However, to put this prototype in operation some improvements can be made: A linear light beam can be added in order to make the color image lighter, now the brightness of the pictures has to be increased with software. Another option is to increase the brightness of the ambient light. However, with this option the laser power has to be increased which can cause safety issues.
Figure 6: Scanning setup