Evaluation of Convolutional Neural Networks for Image Quality Classification based on Synthetic Data

Abstract

In camera production the image quality is of utter importance. Several tests during the production
ensure this high quality. In this thesis the possibility of creating a final test, that classifies the
image quality with the help of machine learning, specifically convolutional neural networks, was
investigated. The data used was made up of synthetic, simulated images with commonly observed
quality defects. Eight different network architectures were evaluated on four different types of
data sets; two data sets for binary classification, one for multi-class classification, and one data set
containing real, non-simulated images. The results were promising with a test accuracy of 0.999
for the binary case with a two-stream network with a DenseNet base. For the multi-class classifi-
cation the best test accuracy was 0.989 with the same network. The results showed that there is
a high potential for the use of convolutional neural networks for classifying image quality. For a
large enough data set a simple convolutional network would be sufficient, achieving similar results
as the best network. The networks could handle most of the investigated defects, but seemed
to have a problem with blemishes - dust/sensor defects, which is why it would be recommended
to have another test for this defect. It was also concluded that there is a need for gathering a
larger amount of real images for training since the results on the real data set were at best 0.594
for binary classification. This showed that the networks trained on the simulated images did not
translate well to real images.

Popular Abstract

Is AI the next big thing in camera production?

As the use of AI is increasing in all aspects of today’s society, camera production companies are starting to take an interest in it as well. These companies go to great lengths and apply multiple tests to ensure that the cameras shipped to their customers are of the highest quality. We have looked into replacing some of these tests with machine learning to make them more objective and standardized. Machine learning is a subsection of AI where a computer algorithm can learn to make decisions based on experience. Can this be used in production without compromising the quality? Early tests performed on synthetically created data show promise as 99 out of 100 cameras were correctly categorized as approved or disapproved, even though some quality defects were harder to detect than others.

After several tests have been performed, the last test in camera production is to examine the general image quality by taking a picture and evaluating it. However, imagine that you have looked at images all day, at one point you are bound to get tired and miss something. Additionally, two different people can choose to evaluate images differently, resulting in subjective results. This is a problem that could be solved by using machine learning, making the test objective and standardized. It will also ensure that the cameras that do go out to the customers are of the desired quality.

This is where neural networks come in. A neural network is a machine learning algorithm that aims to mimic the human brain. Using several connections via ”neurons” the network learns from experience by backtracking based on the results of its decisions. It can learn to recognize patterns which then can be used to make decisions, for example, classification. Classification is when the network can decide if an image belongs to one class or another depending on the contents of the image. In our case that would mean that the network classifies the image as ”passed” if the image is of good quality or ”failed” if it contains image quality defects. Moreover, the network could be trained to say what type of defect the image contains.

We were able to train several different networks of various complexity, with the help of annotated examples, to be able to recognize defects and classify whether an image was of good or bad quality i.e. passed or failed. The results were indeed very promising for synthetic data, with accuracies of up to 99.9%. The synthetic data was made up of good quality images from other tests that are used in production, on these images the varying defects were simulated. The simulation algorithms were created from scratch and the results were compared to a handful of real images. However, for the small amount of real images available the results were not as good, reaching only an accuracy of 54%, indicating that the synthetic data was not representative of the real data. It is clear that more real, authentic images are needed before these networks can be put to practical use.

Is machine learning to be used in camera production then? Yes, there is a high potential for the use, however, there is still work to be done. There are several possibilities of extending and further developing the algorithms that we have created. Last but not least it is a matter of implementing a functioning program for production.

• author: Nigård, Vendela and Karlgren Gustavsson, Joar
• supervisor: Karl Åström
• organization: Mathematics (Faculty of Engineering)
• course: FMAM05 20221
• year: 2022
• type: H2 - Master's Degree (Two Years)
• subject: Mathematics and Statistics
• keywords: Convolutional neural networks, Machine learning, transfer learning, image quality assessment, camera production, image analysis
• publication/series: Master’s Theses in Mathematical Sciences
• report number: LUTFMA-3477-2022
• ISSN: 1404-6342
• other publication id: 2022:E28
• language: English
• id: 9088484
• date added to LUP: 2022-06-17 16:49:38
• date last changed: 2022-06-17 16:49:38