Fusion: Towards a multispectral object classifier

Abstract

Object classification has made great progress lately and a lot of it has to do with the advances in machine learning, processing power, and deep convolution neural networks. Today a camera can detect and classify objects with high precision using trained neural networks, but there is still room for improvements. Cameras using different spectrum, like visual and thermal, have their strengths and weaknesses in classifying objects. They both provide complimentary and unique data about the object if one should fail to classify an object the other one could. This thesis studies techniques to combine the two data types to improve and get a more robust classifier. A dataset was collected with synchronized visual and thermal images, with various examples, including hard examples for both data types. MobileNetV2 were trained and tested on the dataset with three different techniques to fuse the data types together in the network. The results proved the techniques to produce 15 - 17% better accuracy compared to only using one of the data types.

Popular Abstract

Most of today’s classifiers and detectors are trained on good visual data and have a good accuracy on good test examples. But how does this translate to data collected in the real world? What if there is some kind of distortions in the data, like bad weather conditions or bad lighting? Can there be any gain in collecting additional data and combine it with visual data? You might be okay with missing some examples during bad conditions, but what if the classifier or detector works in a surveillance system? Where it is important to have good accuracy at all times to know when an alarm should be sent and not.

Over the last decades, the classifier and detector algorithms made great progress. This is mostly thanks to the progress done in machine learning, specifically the progress in deep convolution neural networks. Today’s state-of-the-art classifiers and detectors are trained on good data and when try- ing to predict an image with some kind of distortion it has a hard time to make correct predictions. In camera surveillance systems the data collected might not always be perfect, for example during the night most of the data collected by a visual camera would be dark, which makes it harder for a network to classify objects.

The consequences of missing a threat or alarming a false threat in a surveillance system can be costly. So, if a classifier or detector is going to be used, it needs to be as accurate as possible at all times. Network models using visual data and thermal data both has its own advantages and complement each other in a lot of cases, by using both to classify or detect an object would make it more accurate if one of them should fail to collect good data. However, there is a consequence in using both, using two networks takes more computational power and time than using only one of them. In this thesis, it will be investigated if there is a way to combine them in one network and if there is any value in doing it in terms of accuracy and speed. A dataset containing synchronized images of visual and thermal of three classes were created to test the theory.

• author: Arcombe, Alexander
• supervisor: Karl Åström; Martin Ahrnbom
• organization: Mathematics (Faculty of Engineering)
• course: FMAM05 20181
• year: 2018
• type: H2 - Master's Degree (Two Years)
• subject: Mathematics and Statistics
• keywords: Fusion, Object Classification, Convolutional Neural Network
• publication/series: Master's Theses in Mathematical Sciences
• report number: LUTFMA-3369-2018
• ISSN: 1404-6342
• other publication id: 2018:E74
• language: English
• id: 8964087
• date added to LUP: 2019-07-15 11:21:37
• date last changed: 2019-07-15 11:21:37