Students: Luc Van Eck | Max Tienhooven | Pedro Nicolas Cristiansen Hutabarat | Simon Gmeiner
Original situation description
Colruyt Smart Techniques, an innovative start-up within the Colruyt Group, follows the motto ‘from inspiration to implementation’. Providing many services, including engineering, designing, or even implementing technological solutions for retail environments, makes them reduce things to the essence to achieve maximum efficiency and effectiveness.
The Company have tried to develop a vision system to identify crate types and pick them up with a robot arm, but they still need help with edge cases. They were interested in changing the design of the EoAT to be able to grab more types of crates. Currently, the existing system uses a vacuum grip, which cannot grab hold of some crates with more significant side gaps. Another situation is that the containers have a closed top, so the crates can only be picked from one side.
Problem statement
Unloading crates is one of the work lines that this Belgian company must improve. The customer is looking for the idea of partially unloading the containers automatically. They want a vision system that can detect if a crate is pickable for the robot; if so, the crate is unloaded automatically. If the crate cannot be unloaded automatically, a worker must be there to move the crate. In addition, the customer wants an improved EoAT for their various crate kinds, as they are now only able to empty 40% of the pickable crates.
Our solution
We aim to create a versatile system that can unload various types of crates that Colruyt has. The process of this system begins with scanning the type of crates. The robot then moves towards the crates until the hooks at the end of the arm tool reaches the edge. These hooks needs to be angled to lift the crate. At the End of Arm Tool (EoAT), we are using pressure sensors that connects to the hooks that is used to monitor the presence of the crate. Just in case the box fails to lift it, or the crates slip down, then the robot will stop and alert a worker.
When placed on the conveyor, the photoelectric sensor detects the crates, and soon after, the conveyor will move. Then, the robot will repeat to pick the crates until there’s no more in the container. Our system comes with a perfect user interface, where we provide multiple buttons and information boxes that workers can use, like start, stop, and emergency buttons. Also, we print all the information in a box so the user can quickly know where the malfunction is. As one of the primary considerations is to create a system that allows manual unloading by workers, we provide an interface where they can see which box needs to be moved, as the robot cannot move it automatically.
Major decisions made during the project.
- EoAT (Hook)
The vacuum gripper has been the go-to tool for Colruyt in unloading crates, effectively handling 85 percent of their diverse crate range. Seeking improvement, we introduced a novel hook design. Before settling on the current design, our approach involves comprehensive research, including constructing various conceptual prototypes. While most of these prototypes initially featured suction cups, some stood out, including one with a top hook, another with a piston gripper, and a double pick-upside variant.
After successfully constructing prototypes, we realized the potential to improve the percentage of pickable crates without relying on suction. Our chosen design allows us to lift all boxes by adjusting the hook to fit existing crates and carefully calibrating force considerations. A significant improvement was adding rubber to the hook for increased robustness. This seemingly minor modification had a profound impact, as specifically tested rubber variants exhibited a potency that rendered the system inoperable.
Final end of arm tool
- EoAT (Safety System)
One of the most significant considerations of the client is to have an end Arm Tool (EoAT) with a proper safety system, so in our new design, we have a suitable mechanism to be called a safety system. Our safety system is hidden behind the front plate, which will slide the bottom plate up and down by the cylinder. Right inside the bottom plate, we also have a slider bracket that can move back and forth by the linear actuator; this thing will be exactly below the crate to hold the weight.
Final Safety System
- Computer vision architectures
Choosing our vision architecture and pipeline, demanded lots of planning, researching and testing, before finally settling on the solution proposed in the next section. We custom trained all our models on over 100 carrier images (>600 crates) and applied modern algorithms for estimating crate position and orientation.
Vision
Our vision setup consists of multiple steps, as shown in the image below.
We are using an intel depth-sensing camera to acquire the initial images, which are then first fed into a segmentation model which extracts the crate fronts and then into a classification model that is capable of differentiating between pickable/non-pickable crates. Finally, we calculate the pickup-location and estimate the crate height for the safety-system and drop-off.
Final Vision
Main code
Our main code consists of a Flask socketio server hosting our user-interface, of which commands can be input that will be handled by our main script. The main script communicates with our robot controlling script which controls all robot movements and also interfaces with our vision script, which is responsible for extracting image coordinates and all image transformations and calculations. Additionally, we have implemented extensive logging and error handling.