Learning to Align Neuroimages: A Weakly Supervised Model for Optical Flow using an Embedded Spatial Pyramid Network

Abstract

This work is about a model for aligning images; in particular, images of the brain. They are deformed in various ways that we would like to undo so these 2D images can successfully be processed into 3D representations. We frame the problem as that of finding optical flow: the vector field that describes how one image needs to be transformed to get the other. This type of problem naturally lends itself to convolutional neural networks. In particular, pyramidal convolutional networks have demonstrated success on it. We build upon this state of the art model in two key ways: we allow it to operate on learned feature maps and we train it in a weakly supervised manner. This new model learns to fix a varied set of deformations, among them cracks and brightness changes. Analysis of our model's errors and operation suggests it is robust and productively leverages its components.