How is the Recall Prediction Model Project progressing?
In a previous post, we gave an update on the ongoing project on ‘Predicting Medical Device Recalls Using Publicly Available Data’. The hypothesis is based on using reported data on recalls to form a model to understand the probability of recalls. Earlier this year, we reported on the project data source, methodology and initial findings in a white paper. Below is a summary of the latest progress report, a full year later, which analyzes the effectiveness of the model and new insights.
Recall Prediction Model Update
On 05/28/2020, the recall prediction model predicted the likelihood of a medical device having at least one recall in the next 365 days. At that time, there were 104,356 medical devices in the Navigator™ for Medical Devices database. The model gave each device a score from 0-1, where 1 is the highest chance of having a recall, and 0 is the lowest chance of having a recall. To create the prediction, the model used data from 05/2017 – 05/2020. The model gave 13,760 out of the 104,356 ( 13.18%) products a score above 0.75.
After a full year, on May 28th, 2021, there was enough data to analyze how accurate the predictions were back in 2020. How accurate is the model and how are we progressing? The results are in!
What we found
Between 05/28/2020 and 05/28/2021 there were a total of 918 safety type recalls. 341 unique medical devices were linked to these recalls. Out of the 341, the model correctly identified 263 in the 05/28/2020 prediction.
The area under the curve (AUC) of the receiver operating characteristic (ROC) curve is 0.857. This is another performance metric for binary classification problems. AUC between 0.8 – 0.9 is considered excellent 
The results are encouraging, creating additional possibilities for the many use cases that are solved utilizing the cleansed and normalized source database, Reed Tech Navigator™ for Medical Devices. For hands-on, filterable product search and a unified view of products, Navigator™ places analysis at your fingertips.
Learn more about Navigator™ for Medical Devices or contact us to request a Predictive Recalls Custom Analysis, see a demo or just ask questions:
 Mandrekar, J. N. (2010). Receiver Operating Characteristic Curve in Diagnostic Test Assessment. Journal of Thoracic Oncology, 5(9), 1315–1316. doi: 10.1097/jto.0b013e3181ec173d