Fujitsu Laboratories announced the development of the world’s first “High Durability Learning” technology that estimates the accuracy of AI and automatically restores AI models.
In a business context, AI models often struggle to maintain accuracy over the course of operations due to frequent changes in input data caused by shifts in social conditions and market trends. The newly developed technology makes it possible to automatically estimate the accuracy of the AI model during operations while preventing deterioration of its accuracy.
This technology enables AI models to be maintained with high accuracy over a long period, and enables stable AI operations in various circumstances. This technology will be offered as part of Fujitsu’s Human Centric AI Zinrai, a comprehensive portfolio that encompasses Fujitsu’s wide range of AI technologies and techniques.
Development Background
As AI models built from training data find additional uses in business, shifts in social conditions, market trends, and the environment can lead to changes in input data from the time when the model was initially trained. This cause deterioration of the accuracy of an AI model. When evaluating corporate credit risk in the financial sector using an AI model trained with corporate financial data, for instance, the accuracy of credit risk estimates will gradually deteriorate. This is because tendencies of input data into the system will change over time after the model was initially trained due to broader trends in the economy.
This makes it essential to periodically check the accuracy of AI models being used for business. If deterioration of the accuracy is confirmed, the AI model must be retrained using the latest data to restore the accuracy of its predictions.
Issues
Measuring the accuracy of the AI model requires the latest input data (Example: Financial Data) and the labeled data set (Example: Credit Risk). Preparing the labeled data is costly and labor-intensive, and it sometimes requires experts to annotate correct labels to large amounts of data. It is also difficult to determine when AI models should be retrained and it may be difficult to detect even if accuracy decreases significantly until its deterioration has actually been confirmed (Figure 1).
Figure 1. Challenges in AI Operations (Examples of credit risk assessment in the financial sector)
About the Newly Developed Technology
Fujitsu has developed a “High Durability Learning” technology that can estimate the accuracy of an AI model at any time during its operation and automatically restore it when deterioration of its accuracy is detected. The features of the developed technology are as follows.
1. Estimation of AI model’s accuracy during operation
Correct labels can be automatically annotated to input data during operations by visually charting the distribution of training data used to train an AI model and the distribution of input data during operations, as well as by grasping tendencies of changes in data from the time of training to the time the model is used (Figure 2). This makes it possible to estimate the accuracy of the AI model at that point in time by comparing the estimated results of the original AI model against the input data with the automatically configured correct labels.
Figure 2. Automatically annotation of input data with High Durability Learning
2. Controlling reductions in accuracy by automatically restoring the AI model
Based on the correct labels given to the input data, the decision boundary of the AI model can be adjusted according to trends of the input data, so that the AI model can automatically adapt to the new input data. As a result, it is possible to effectively prevent decreases of accuracy of AI model without performing retraining.
Effects
Using financial data from 3,800 companies, Fujitsu has verified our assessment of credit risk in the financial sector. This technology has made it possible to estimate the accuracy of AI model within a 3% error margin. We also confirmed that this technology can maintain the accuracy of AI at 89%, which was at 69% with the conventional technology.
The new technology can maintain the accuracy of AI model over a long period, delivering reliable and stable operations for users. Fujitsu anticipates that its High Durability Learning technology will ultimately improve the efficiency of AI models not only in the financial sector but in various other businesses, as well, including for product classification in the retail sector and character recognition in the distribution and logistics sector.
Future Plans
Fujitsu aims to incorporate this technology into its goal-oriented ”Design the Trusted Future by Data × AI” processes and framework, as well as for machine learning technologies in its Zinrai portfolio, and aiming to deliver practical use cases by the end of FY 2020.
