Students: Zeynel Koca, André Hoogesteger, Matthijs Kenis, Martijn Posadinu. (2022)
Company
In hospitals every medical instrument will pass through a sterilisation process after being used in the operating room. In the Erasmus Medical Centre in Rotterdam, the Netherlands this amounts to hundreds of instrument sets per day.
To increase the speed of this process, save on packaging materials and lighten the physical workload on the staff, R-solution have developed a system that automatically packages the instrument trays which is part of the sterilisation process. This system uses multiple Techman cobots to fold and fasten the packaging material around the trays. More information on this system can be found by following this link: https://www.r-solution.nl/r-appit/
Problem
Instrument detection
In order to further automate the process of sterilisation, our team of students has investigated the application of a machine learning algorithm to classify medical instruments. Other methods of identification are available, such as “RFID”, but require physical modification of the medical instruments which is not desired. In the current situation, instrument sets are used in an operating room and afterwards placed back in the corresponding tray and sent to the sterilisation unit. However, instruments are sometimes misplaced, resulting in a (different) set being incomplete. During the sterilisation process these sets are rearranged to the recipes as described in a database. The instrument sets need to be complete when they leave the sterilisation process and enter the storage area where they can be taken to the operation rooms.
Tray dimensions
Instrument sets are packaged in stainless steel trays. Every model has standard dimensions for the length, width and height of the tray. These dimensions are used in the automatic packaging system. However, the actual measurements of the trays can vary multiple millimetres which impedes the packaging process where a tight fit is desired. To improve the quality of the packaging process the trays need to be measured individually.
Solution
To check whether an instrument set is complete or not, a neural network has been applied. Images from a camera on the Techman cobot are fed into a python script which uses the Keras library to classify the object in the images.
The classification result is then checked with a recipe for the instrument set in the database and moved to the correct location.
Step 1: A barcode located on the tray containing the instrument set is scanned with a handheld barcode scanner by the operator.
The barcode is linked to a recipe for the instrument set in the database. This recipe will later be used to determine where an instrument needs to go in this process. The barcode also identifies the tray of which the dimensions will be measured and saved in the database later in this process.
Step 2: The operator places the individual instruments on the conveyor between two red lines.
To ensure a high accuracy of the classification, we made the decision to analyse individual instruments rather than a whole set in its tray. Instruments are placed separately on a conveyor, on which they pass under the camera. This way the whole instrument will be visible to the camera rather than some instruments being covered by others.
Step 3: The instruments pass under the camera and the resulting frames are fed into a python script.
In the time it takes for one instrument to pass under the camera, multiple frames are sent by the camera to the script. These frames are first cropped to the red lines and subsequently cropped to the instrument itself.
Step 4: The cropped images are fed into the neural network. This model returns a class to which it thinks the object belongs and a confidence score of this classification.
The python script will either accept this classification when the confidence score is sufficient or reiterate the classification process up to three times.
Step 5: There are three options depending on the result of the classification process.
- The instrument remains on the conveyor when the classification result’s confidence score is sufficient and matches the recipe in the database for the instrument set being processed.
- The instrument is separated from the flow and placed into temporary storage when it is successfully classified, but does not occur in the recipe for the instrument set being processed.
- The instrument is not classified with a sufficient confidence score and is subsequently put aside for the operator to investigate the cause of the error.
Step 6: The system checks whether any instruments belonging to the instrument set are in temporary storage.
When all contents of the instrument tray have passed the camera, the operator will end the process via the user interface. The system will then check storage locations for instruments that belong to the set that has just been processed and place these within the set.
Step 7: Tray dimensions
After the classification process has ended the operator places the instrument tray on the measuring location. The cobot will then orient its camera in such a way that the height of the tray can be measured using the vision application in the cobot’s TM-Flow software. After measuring the height the cobot will change its orientation to measure the length and with of the tray using the same process. The dimensions are than stored in the database and could be send to the R-appit packaging system.
The result of this process is an output of complete instrument sets that are ready to be inspected for cleanliness and quality. It simplifies the process of sorting through different instrument sets to find and relocate missing instruments to simply feeding instrument sets onto the conveyor and taking out completed instrument sets. For it to be applied in the current process taking place at the Erasmus MC the dataset and storage capacity of the system would have to be extended for every instrument that is used in the hospital.