Students: Lars Hartman, Sven Engelchor, Martin Teekens. (2022)

Original situation

Afriflora is a business that produces roses in countries on the African continent and sell them internationally through Royal FloraHolland. Currently, when an aircraft pallet full of boxes arrives, it gets placed on a scissor lift which lower the pallet to a workable height. Then, three workers depalletize the boxes by hand, and place them on a conveyor with the label up front. It is important that the box does not lay on any of its sides, and that the label is on the front, so that the sorting system can successfully sort all the boxes.

The problem

Depalletizing is demanding work. The company has trouble finding personel that is willing to do the job. Because of this, Afriflora wants to automize this process. They reached out to Heemskerk Besturingstechniek. Heemkerk Besturingstechniek wanted to make sure it was possible, for that, they reached out to SMR, and so it was our job to design the automatized situation and make a working proof of principle.

The machine must recognize the boxes of one layer, pick them up one by one, and place them on the conveyor. The boxes on the pallet however may can lay flat, or they can lay on their sides. What ever the orientation, the boxes must be placed flat, with the label upfront, else, the sorting system will not be able to successfully sort the boxes. A requirement is that the solution can handle 400 boxes an hour. This is necessary, or else the company will miss revenue thanks to the solution.

Our solution

Vision

Detecting the boxes

As stated above, the boxes do not lay in a predefined grid, but can lay in a zigzag pattern, some on their side, some flat. For this it is useful to use a vision camera so that the outline of the boxes can be detected, and with that, the centre and orientation can be determined. The problem is that it is hard to detect boxes in a pile of boxes, it is like finding a grain in stack of hay. To solve this problem, we used an Intel RealSense D435. This is a depth sensing camera that uses stereovision to measure the distance from the camera. With this, we can remove everything below the upper layer of boxes from view (Figure 2). Then, by getting the grey values, and removing some noise, we can convert it to a binary image only containing black and white (Figure 4). With the binary image, a contour detection algorithm is used to detect the edges of the boxes (Figure 5). By drawing the smallest possible rectangle, we are able to get a rectangle that has the same centre and orientation as the box (Figure 6). Since the boxes all have the same colour, it is possible that in the binary conversion a small link is formed between two boxes. This creates a situation where two boxes are detected as one entity. To prevent this, in the binary image is strongly blurred, and then everything that is not fully white gets turned to black.

Detecting the label

There is a label with a barcode on one side of the box. This label is on the front of the box. We needed a second vision setup for finding the label. The label can fit on two sides. The camera will take a picture of one of these sides. The image will be converted to a black and white picture. If there is a white square on the picture the vision can detect the label with contour detection. At that moment the robot knows that is de front of the box. If there is no label detected the robot knows the other side is the front.

The End of Arm Tool (EOAT)

for the proof of principle, we move empty boxes. We have therefore made a simple gripper with 4 suction cups. We did research for a suitable gripper for the situation at afriflora and have 2 recommendations for the company. A gripper from Joulin or from Piab, both are foam suction grippers that can handle the weight.

Computer Assisted drawing (CAD)

We made a 3D model of the designed situation at afriflora. This model contains the robots we chose. We placed the Camra’s and shortened the conveyer belts to make room for the 2 robots.

Proof of Principle

For the prove of principle we used a Kawasaki robot. The boxes are placed on the ground within the reach of the robot. The vision programs are running on a separate laptop, which preforms all the calculations needed. As for the HMI, a plc is used to display the status of the bot, and to control the conveyor belt, and give a start and stop signal to the robot. The plc and the Kawasaki communicate with each other via digital I/O signals. The communication with the laptop is handled via an TCP server/client model, where the laptop is a server, and the Kawasaki the client. Then, when the Kawasaki is working, when ever needed, the robot can ask the server for coordinates and decide its path accordingly.

The major decisions we made during the project

During this project, there were many decisions made along the way to completion. The most important decision was to choose what tasks the robot will be performing. First, we decided that the robot will completely orientate the box to a correct orientation and place it on the belt. Later we chose a solution that takes away the tasks of correctly orientating the box from the robot and designed specialized parts that mechanically rotate the box. In the end we decided that all tasks are given back to the robot, but place two robots, because it is easier to maintain two robots that are the same, then many different contraptions.

We’d like to thank Guus Zwart for being a great client, Mathijs van der Vegt for the techsupport, and Thijs Brilleman for project support.