Industry 4.0: How deep learning accelerates image analysis

If you casually overheard a work meeting between Julian von Lautz and Dr. Hendrik Kramer, you might well get the idea that the two engineers were planning a trip to a reptile zoo. You might hear them mention python and anaconda, and repeatedly talk about the world’s largest snake family, colubrids or Natter in German. The Natter, they say, is very fast indeed. People in IT will know that the first two are actually references to the programming language Python and its toolkit Anaconda—not real snakes. But what about Natter? “It stands for ‘Neuronal analysis tool to evaluate rapid’ and describes new deep learning software that takes image analysis in metallography to a completely new level for materials development, quality inspection and failure analysis,” explains materials expert Dr. Hendrik Kramer. He works in the Materials Analysis/Metallography team at MTU Aero Engines in Munich and image analysis falls under his area of responsibility. Dr. Kramer teamed up with Julian von Lautz, then Senior Manager Failure Analysis, and other colleagues to put the idea into practice. “Admittedly, we came up with the name Natter first and only then a description to fit it,” he says.

The acronym and its definition hit the nail on the head: similar to the way that colubrids use their lightning-fast bite to snatch their prey, the MTU software uses artificial intelligence (AI) to classify microscope images of metals at an unprecedented speed. This really comes into its own in materials development, quality control and failure analysis. Natter is already in use for nickel-based alloys, where the percentage of the ɣ’ phase in the metal is decisive for the quality of the material properties at high temperatures. The challenge was that analysis of these microscope images had to be performed manually in some cases, which is a very laborious task. In the first step, a computer program classifies each pixel as light or dark based on its grey value and calculates the area of each pixel.

A long time coming: automated image analysis

“Depending on the material, a few special rules apply, such as counting adjoining light or dark areas only if they reach a certain size and shape or depending on how far away other areas occur,” explains image analysis expert Dr. Kramer. The old software produces an image with a rough pre-selection of light and dark areas, which a metallographer then has to check and improve. To do this, the experts have to click to add, remove or even trace individual areas by hand, a process that can take half an hour or so. “Given that a full analysis job can comprise 60 images, our highly trained metallographers sometimes sat clicking away at the computer for almost a week,” Dr. Kramer explains. The idea to automate this process using AI came about years ago and had been an aspiration for just as long. However, for most of that time, the machine learning algorithms required for this use case just weren’t powerful enough.

It was only when von Lautz and Dr. Kramer came across a special, newly developed deep learning method around three years ago that automated image analysis became a potentially viable option. They immediately designed a research project for a master’s student as a swift way to clarify if the approach could actually solve their problem. “Following about two months of programming activities, it was already clear that the new method would work,” von Lautz recalls. It took the two engineers a good year to get all the proper documentation in place, optimize the program code and prepare the training data so that Natter was ready to use, they explain. “You shouldn’t underestimate the work involved in preparing the data for such methods. It takes a huge amount of effort,” Dr. Kramer says. To train their deep learning software, they used 2,400 pairs of images for which they already had the right solutions for the respective tasks. Around 2,000 of these were to train the software and the remaining 400 were to check the progress of the training.

Natter improves image analysis performance by at least 90 percent

According to Dr. Kramer, picking out the image pairs was only one part of the job. Getting the data into a standard format took just as long, if not longer. “The images come from different microscopes, vary in the way they are exposed and cropped and have different resolutions,” he says. It took several months before it was possible to start training Natter on a uniform dataset. But all the effort more than paid off. Automated image analysis using AI takes just five seconds per image. The results of the automated analyses correspond to those of human testers with just over 96 percent accuracy. “So Natter’s processing time is next to nothing,” Kramer says. Nevertheless, the MTU engineers are erring on the side of caution and expect that an expert will still have to visually inspect every automatically evaluated image and perhaps check every tenth image again by hand.

“But even then, it will take only three minutes per picture using Natter rather than half an hour,” von Lautz says. “That boosts performance by at least 90 percent.” And it massively reduces the metallographers’ workload, who now need only two to three hours for a job that used to take a week. This frees them up to focus on other tasks, such as the development of new materials. According to von Lautz, MTU is ahead of the game in its implementation of this automated image analysis software. “In the future, we’ll work on making Natter suitable for analyzing even more materials in addition to nickel-based alloys,” he says. But both the MTU engineers are already thinking even further ahead. “In principle, the method offers a good solution for all image processing work—especially when the problem can’t be described mathematically but a lot of training data is available,” Kramer says. And its use isn’t necessarily restricted to metallography. Von Lautz believes that in the medium to long term, such AI methods will take the form of virtual assistants for aerospace engineers.

“In the same way that modern AI diagnostics programs assist doctors by automatically indicating any metastases on an X-ray of the lungs, these devices will help us engineers to focus our attention on the relevant areas during analyses and tests,” von Lautz says. This will allow engineers to perform more tests in less time to the same level of quality as before, if not higher, he explains. “In the aviation industry, where safety comes above everything else, the future belongs to such AI solutions.”