Students_spring_2020 : Es-saïdi Shaïma and Rocha João

Introduction

The year 2020 has been shattered by the coronavirus. The Hague University of Applied Sciences has closed its doors for the spring semester and the minor Smart Manufacturing and Robotics has been modify in consequence.

Given the exceptional circumstances of this event hitting the world, the minor was transformed into online projects. Initially, a minor robotic oriented become a programming project initiated by students.

For 4 weeks, a solution has been developed supervised by Guus Paris, Matthijs Van Der Vegt and Thijs Brilleman. The topic of our project is to count people entering and leaving a grocery shop in order to help the owner to deal with the coronavirus crisis. The tackled of this project has been made by Es-saïdi Shaïma and Rocha João, two mechanical engineering from France and Portugal.

Current situation

Faced with the advance of the coronavirus in recent months, many countries have decided to adopt safety restrictions and new rules have been put in place for the good of the population. It is on an island in Portugal, the island of Terceira, that we have carried out this project.

Since March, merchants have had to comply with government instructions to stop the spread of the coronavirus. These workers can no longer accept as many customers as before and are forced to have only a maximum quota inside the store. Indeed, they now have to count the number of people entering and leaving the store.

However, counting these people isn’t an easy task when working on several fronts, which is why this project was initiated. As the staff and the owner of the business are not able to monitor the flow of customers, we have carried out this project with machine vision.

Goal of this project

The goal of this project is to help employees and the owner in those difficult times to count people entering ang leaving the grocery store. According to the new political regulations, the store can only accommodate a maximum of 4 people at a time. To reach this goal, usb camera, raspberry pi, leds has to be used. The algorithm has to be able to operate 10 hours per day on a live stream video.

Our Solution

At the end of our 4 weeks project, we build a person counter algorithm with OpenCV and is working as demanding. The solution developed required machine vision principally and can be divided into two parts:

Detecting

During the detection phase we run our object detection to identify if new objects have entered our view in first place and then see if we can find objects that were lost during the tracking phase.
For each detected object we create or update an object tracker with the new bounding box coordinates using blob detection. Finally, we run this phase once every N frame.

 

Tracking

When we are not in the “detecting” phase we are in the “tracking” phase.

During the tracking phase, for each of our detected objects, we create an object tracker to track the object as it continuously moves around the frame. This tracker is more efficient and faster than the object detector.
To track people we remove the noise (erosion) and the shadows (background subtraction) and fill up the blank area (dilation) from the frames to be able to compute a bounding box for each person that is in the chosen area.

                                                        

Figure 1 : Without dilation                                                Figure 2: With dilation

From there, we extract the position coordinates and determine if someone is coming in or out of the shop. To do this, we drew two white lines and two crossing line (Up and Down) to visualize the people. By solving the problem in this way, we can determine with certainty whether the customer is entering or leaving the store and therefore the count become more accurate and rigorous.

Figure 3: The crossing lines

Therefore, every time a person will cross this line, we’ll increment our respective counter and finally count people entering or leaving the grocery shop.
Lastly, we’ll continue tracking until we’ve reached the last frame and then re-run our entire process.

To finish, at the entrance will be placed a luminous system composed of leds that will allow to regulate the flow of customers and let them enter or not inside the premises.

Figure 4: Leds stop or go

The major decisions

The first idea was to use the camera (CCTV) from the grocery shop to run the code directly on a live stream but unfortunately, the software proved to be inadequate for our use. Indeed, as the camera are working on a special security service, we couldn’t have access to it. As a result, we changed our approach and chose to develop our own live stream using a webcam and a raspberry pi which will be also used to run the code.  After receiving all the necessary material, we were able to create a live stream from our store, however, the insufficient capacity of the raspberry pi processor pushes us to give up on doing it because we couldn’t run our program.

In order to finish our project properly and obtain a correct result, we chose another code that no longer use machine learning. Indeed, we wanted in first place to use machine learning to avoid imposing restrictions to the clients but as we encounter this problem at the end of our project we switch back to a more basic code where we need to impose some constraints.

This allowed us to be able to present this work and seems to be a better solution than the initial one. However, on one hand, we are disappointed to not to have been able to complete this project but also happy to have managed to show something that works in certain conditions as long as the customer respect the constraints and its nighttime.

 

Conclusion

The main goal of our project has been achieved but due to time restrictions, it was not possible for the group to build a robust algorithm.
The original approach of using a live stream from the CCTV to count people was abandoned in favor of a homemade one.

Though the assignment was tough for two mechanical engineer without programming experience to tackle this problem successfully in the meantime. However, with dedication, hard work and motivation, the group has made a proof of concept within the set time.

This solution is a beginning and there must be change afterwards before this can be implemented but we have learned a lot in the course of this project.