Stefan Bosse - Robust Underwater Image Classification - Overview

Overview

Stefan Bosse - Robust Underwater Image Classification - Overview

Overview

Stefan Bosse - Robust Underwater Image Classification - Overview

Challenges

• Underwater images pose low quality with respect to illumination conditions, sharpness, and noise.

• Finding ROIs automatically can help to identify relevant regions in the image quickly by humans, or they can be used as an input for automated inspection and structural health monitoring (SHM).

Stefan Bosse - Robust Underwater Image Classification - Overview

Challenges

• Currently, for the inspection of piles of sea windmill energy harvester, divers have to go under water.

• But even if humans inspect the underwater surfaces (underwater by the diver or remotely), the scenes are cluttered and the identification of surface coverage is a challenge.

• Automated visual inspection is desired to reduce maintenance and service times.

Stefan Bosse - Robust Underwater Image Classification - Overview

Goals

Stefan Bosse - Robust Underwater Image Classification - Overview

Goals

Stefan Bosse - Robust Underwater Image Classification - Overview

Summary and Highlights

• We propose and evaluate a hybrid approach with segmented classification using small-scaled CNN classifiers (with less than 20000 hyper parameters and less than 3 Million unity vector operations)

• A reconstruction of labelled ROIs is provided by using an iterative mean and expandable bounding box algorithm.

  • The iterative bounding box algorithm combined with bounding box overlap checking suppress spurious wrong segment classifications and represent the best and most accurate matching ROI for a specific classification label, e.g., surfaces with pocks coverage.

Stefan Bosse - Robust Underwater Image Classification - Overview

Images and Classes

The underwater inspection of technical structures, e.g., construction parts of off-shore wind turbines like piles, involves the identification of various parts in the underwater images:

1. Background with water, bubbles, and fishes, summarized as feature class B;
2. Technical structure, e.g., a mono pile of a wind turbine, summarized as feature class P;
3. Formation of coverage with marine vegetation or organisms on the surface of the structure, summarized as feature class C.

Stefan Bosse - Robust Underwater Image Classification - Overview

Images and Classes

• The images set consists of different RGB underwater images posing a

  • high variance in illumination conditions,
  • spatial orientation,
  • noise (bubbles, blurring), and
  • colour palettes.

• The images are snapshots taken from videos recorded by a human diver with an underwater camera.

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

Methods and Architecture

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

Methods

1. Manual image labelling with labelled polygon regions
2. Segmentation of raw input images in smaller segment images (segment size n × m pixels, e.g., 64 × 64)
3. Convolutional Neural Networks (CNN)
   • Input: Image Segment
   • Output: Class probabilities (B, P, C) and maximum likelihood selection of best candidate (or none)
4. Iterative rectangular bounding box approximation using density-based clustering (DBSCAN) and centre-of-mass (COM) computations; COM determines the centre of the bounding box.
   • Input: All labelled segments of input images
   • Output: Set of labelled ROI coordinates
5. Post correction of overlapping conflicting labelled ROIs

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

Architecture

Overview of the data flow architecture and the used algorithms

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

Software

Web browser-based software architecture with remote shell worker processes (Bosse, Appl. Sciences, 2022)

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

CNN Parameters

• Four different CNN model architectures were evaluetd:

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

CNN Parameters

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

Mean Bounding Box Algorithm (MBB)

• There is a set of class symbols Σ and a class matrix M consisting of elements labelling an image segment with a class, so that:

$$\Sigma={\left\lbrace{B},{P},{C},{U}\right\rbrace}\\ \sigma\in\Sigma\\ \hat{{M}}={\left(\matrix{\sigma_{{{1},{1}}}&..&\sigma_{{{1},{j}}}\\\sigma_{{{2},{1}}}&..&\sigma_{{{2},{j}}}\\..&..&..\\\sigma_{{{i},{1}}}&..&\sigma_{{{i},{j}}}}\right)}$$

• The matrix M is flattened to a point cloud list set P={p_σ}_σ∈Σ.
• Each class set p contains the matrix positions of the respective elements, i.e., p_σ={ ⟨i,j⟩}, with all points classified by the CNN to the same label class σ∈Σ.

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

Mean Bounding Box Algorithm (MBB)

• The DBSCAN clustering will return a group list of points that satisfy the clustering conditions, one point group list for each label class:

$${D}{B}{S}{C}{A}{N}:{P}\to{\left\lbrace{\left\lbrace{p}_{{j}}\right\rbrace}_{{j}},{\left\lbrace{p}_{{k}}\right\rbrace}_{{k}},{\left\lbrace{p}_{{l}}\right\rbrace}_{{l}},..\right\rbrace},{j}\ne{k}\ne{l}\\ {P}:{\left\lbrace{p}_{{i}}\right\rbrace}_{{i}},{i}={\left\lbrace{1},{2},{3},..,{n}\right\rbrace}\\ {p}_{{i}}=<{i},{j}>\in{R}^{{2}}$$

• It is assumed that a cluster will contain a majority of correctly classified points (segments), and a minority of scattered wrong classified points.

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

Mean Bounding Box Algorithm (MBB)

Iterative bounding box expansion with final conflict overlapping shrinking

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

Mean Bounding Box Algorithm (MBB)

• The MBB algorithm computes points ⟨x₁,y₁,x₂,y₂⟩ of a bounding box that is centred at the mass-of-centre point c of all points of a cluster and with outer sides given by the vectorial mean centred position of all points above or below, and left or right form the c point.

• The expansion of a previously computed bounding box is done by all points outside of the current bounding box, performing the next extension iteration.

  • Again, a spatial position averaging is performed, extending the boundary of the bound box.
  • The expansion is performed iteratively.
  • Each step includes more points, but increases the probability that the bound box is over-sized with respect to spurious outlier points that result from wrong CNN classifications.

Stefan Bosse - Robust Underwater Image Classification - Methods and Architecture

Mean Bounding Box Algorithm (MBB)

• In case of high iteration loop values, bounding boxes from different classes can overlap.

• To reduce overlapping conflicts, a class priority is introduced layering the class regions by relevance.

• After the ROI expansion is done, overlapping bounding boxes with lower priority are shrink until all overlap conflicts are resolved.

  • Commonly, more than one side of the bounding box can be shrunken to reduce the overlap conflict.
  • The possible candidates are evaluated and sorted with respect to the amount of shrinkage at each side.
  • The lowest shrinkage is applied first. If the conflict is not reduced by the selected side shrinking, the next side is shrink until the conflict (with one or more higher priority bounding boxes) is reduced.

Stefan Bosse - Robust Underwater Image Classification - Experiments and Results

Experiments and Results

Training and test data: Randomly chosen sub-set of about 10000 image segments taken from about 300 snap shot images

Stefan Bosse - Robust Underwater Image Classification - Experiments and Results

Experiments and Results

Training and test data: Randomly chosen sub-set of about 10000 image segments taken from about 300 snap shot images

Four different identical models were trained and applied in parallel (different random initialisation and sub-set of images)

Stefan Bosse - Robust Underwater Image Classification - Experiments and Results

Results

Classified bounding boxes for one image using four models trained in parallel (same parameters) but with different random initialisation and training data sub-set (Blue: class background, red: class coverage, green: class free construction surface)

Stefan Bosse - Robust Underwater Image Classification - Experiments and Results

Results

Accumulated prediction results for training, test, entire data set union with statistical features of the model ensemble trained in parallel (using different data sub-sets and random initialisation). All errors with 2σ standard deviation interval, and N=9000 samples, n=3000 for each class, and using CNN architecture A.

Stefan Bosse - Robust Underwater Image Classification - Experiments and Results

Results

Forward and backward (training) times for one 64 × 64 × 3 segment and different CNN architectures using the JavaScript ConvNet.js classifier¹ and TensorFlow (CPU)²

Stefan Bosse - Robust Underwater Image Classification - Conclusion

Conclusion

Stefan Bosse - Robust Underwater Image Classification - Conclusion

Stefan Bosse - Robust Underwater Image Classification - Conclusion

Stefan Bosse - Robust Underwater Image Classification - Conclusion

• Although, CNN are less suitable for low-resource embedded systems, the CNN architecture D (4/4) could be implemented in an embedded camera systems, expecting overall ROI extraction times for one image frame about 5 seconds, not suitable for real-time operation (maximal latency 100 ms). Using control-path parallelisation performing the image segment classifications in parallel, the ROI extraction could be reduced to 1 second using generic multi-core CPUs, or 100 ms using FPGA-based co-processors.