LS-Net: Learning to Solve Nonlinear Least Squares
for Monocular Stereo

Most algorithms in computer vision use some form of optimization to obtain a solution that best satisfies some objective function for the problem at hand. The optimization method itself can be seen as simply an intelligent means of searching the solution space for the answer, possibly exploiting the specific structure of the objective function to guide the search.

One particularly interesting form of objective function is one that is composed of a sum of many squared residual terms.

where $r_{j}$ is the j-th residual term and $E$ is the optimization objective.

In most cases the residual terms are a nonlinear function of the optimization variables and problems with this type of objective function are called nonlinear least square (NLLS) problems (NLSPs). NLSPs can be efficiently solved using second-order methods [13].

However, the success in finding a good solution also depends on the characteristics of the problem itself. The set of residual functions can be likened to a system of equations with their solution at zero, $r_{j}(\mathbf{x})=0$. If the number of variables in this system is larger than the number of equations then the system is underdetermined, if they are equal then it is well-determined and if there are more equations than variables then it is overdetermined. Well-posed problems need to satisfy three conditions: 1) a solution must exist 2) there must be a unique solution and 3) the solution must be continuous as a function of its parameters [21].

Undetermined problems are ill-posed as they have infinitely many solutions and therefore no unique solution exists. To cope with this, traditional optimizers use hand-crafted regularizers and priors to make the ill-posed problem well-posed.

In this paper we aim to utilize strong and well-developed ideas from traditional nonlinear least squares solvers and integrate these with the promising new learning-based approaches. In doing so, we seek to capitalize on the ability of neural network-based methods to learn robust data-driven priors, and a traditional optimization-based approach to obtain refined solutions of high-precision. In particular, we propose to learn how to compute the update based on the current residual and Jacobian (and some extra parameters) to make the NLLS optimization algorithm more efficient and more robust to high noise.

We apply our optimizer to the problem of estimating the pose and depths of pairs of frames from a monocular image sequence known as monocular stereo as illustrated in Fig. 1.

To summarise, the contributions of our paper are the following:

1. 1.

   We propose an end-to-end trainable optimization method that builds upon the powerful approximate Hessian-based optimization approaches to NLLS problems.

2. 2.

   The implicit learning of priors and regularizers for least squares problems directly from data.

3. 3.

   The first approach to use a learned optimizer for efficiently minimizing photometric residuals for monocular stereo reconstruction.

Compared to existing learning-based approaches, our method is designed to produce predictions that are accurate and photometrically consistent.

The rest of the paper is structured as follows. First we outline related work on dense reconstruction using traditional and learning-based approaches. We then visit some preliminaries such as the structure of traditional Gauss-Newton optimizers for nonlinear least square problems. We then introduce our proposed system and finally carry out an evaluation of our method in terms of structure and motion accuracy on a number of sequences from publicly available datasets.

Optimization for SLAM In visual SLAM we are faced with the problem of estimating both the geometry of the scene and the motion of the camera. This is most often formulated as an optimization over the pixel depths and transformation parameters between pairs of frames. The cost function comprises some form of reprojection error which may be formulated either in terms of geometric or photometric residuals. Geometric residuals require the correspondence of points to be known and thus are only feasible for sparse recostructions. Photometric residuals are formulated in terms of intensity differences and can be computed across the entire image. However, this photometric optimization is difficult as the photometric residuals have high noise levels and various strategies have been proposed to cope with this. In DTAM [17], for example, this is accomplished by formulating a cost volume and integrating the residuals from multiple frames before performing the optimization. Even then, the residuals need to be combined with a TV-L1 regularization term to ensure noise does not dominate the reconstruction. Other approaches, such as LSD-SLAM [10], operate only on high-gradient pixels where the signal-to-noise ratio of the photometric residual is high. Even so, none of these systems are able to estimate the geometry and motion in a single joint optimization. Rather, they resort to an approach which swithches between independently optimizing the motion parameters and then the depths in an alternating fashion. CodeSLAM [2] overcomes this problem by using an autoencoder to compress the scene geometry into a small optimizable code, allowing for the joint optimization of both the geometry and motion.

Learning for Monocular Stereo There has been much interest recently in using end-to-end learning to estimate the motion of a camera [7, 6, 24] and reconstruct scenes from monocular images [9]. Most of these [9, 26] are based on feed-forward inference networks. The training signal for these networks can be obtained in many ways. The first approaches were based on a fully-supervised learning signal where labelled depth and pose information were used. Subsequent works have shown that the networks can be learned in a self-supervised manner using a learning signal derived, for example, from photometric error of pixel-wise reprojection [26], from the consistency of rays projected into a common volume [22] or even using an adversarial signal by modelling the image formation process in a GAN framework [4]. Even so, these approaches only utilize the photometric consistency in an offline manner, i.e. during training, and do not attempt to optimize it online as is common in traditional dense reconstruction methods.

To this extent, some works such as [23], have demonstrated that it is beneficial to include multiple views and a recurrent refinement procedure in the reconstruction process. Their network, comprising three stages, is closely related to the structure which we build on in this work. The first stage consists of a bootstrap network which produces a rough low-resolution prediction; the second stage consists of an iterative network which iteratively refines the bootstrap prediction; and finally a refinement network which computes a refined and upscaled depth map.

In this paper, we adopt the same structure but formalize the iterative network as an optimization designed to enforce multiview photometric consistency where the bootstrap network acts as an initialization of the optimization and the refinement acts as an upscaling. In essence, our reconstruction is based on an optimization procedure that is itself optimized using data. This is commonly referred to in the machine-learning literature as a meta-learned optimizer.

Meta-learning and Learning to Optimize A popular and very promising avenue of research which has been receiving increasing attention is that of meta-learned optimizers. Such approaches have shown great utility in performing few-shot learning without overfitting [19, 5], for optimizing GANS which are traditionally very difficult to train [16], for optimizing general black box functions [3] and even for solving difficult combinatorial problems [8]. Perhaps the most important advantage is to learn data-driven regularization as demonstrated in [18] where the authors use a partially learned optimization approach for solving ill-posed inverse problems. In [15], the authors train through a multi-step inverse compositional Lukas Kanade algorithm for aligning 2D images. In our method, we utilize a learned multi-step optimization model by using a recurrent network to compute the update steps for the optimization variables. While most approaches that attempt to learn optimization updates use either fully learned update steps [25, 14] about the objective and first-order gradient information [3], we exploit the least-square structure of our problem and forward the full Jacobian matrix to provide the network with richer information. Our approach is – to the best of our knowledge – the first to use second-order approximations of the objective to learn optimization updates.

The model is built around the optimization of the photometric consistency of the depth and motion predictions for a short sequence of input images. Each sequence of images has a single “target” keyframe (which we choose as the first frame) for which we optimize the depth values. In all cases, we operate on inverse depths, $z=\frac{1}{d}$ for better handling of large depths values. Our model additionally seeks to optimize for the relative transformations between each source frame $s$ in the sequence and the target keyframe $t$, $\mathbf{p}_{t\rightarrow s}$. The full model consists of three stages. All iterative optimization procedures require an initial starting point and thus the initialization stage serves the purpose of predicting a good initial estimate. The optimization stage consists of a learned optimizer which benefits from explicitly computed residuals and Jacobians. To make the optimization computationally tractable, the optimization network operates on a down-sampled version of the input and exploits the sparsity of the problem. The final stage of the network upsamples the prediction to the original resolution. The networks (including those of the optimizer) are trained using a supervised loss. We now describe each of the three network components in detail.

In this section we describe the loss function which we use to train the network weights of all three stages of our model.

The current state-of-the-art depth and motion prediction networks still rely on labelled images to provide a strong learning signal. We include a loss term based on labelled ground truth inverse depth images $\tilde{\mathbf{z}}$,

with image width $w$ and height $h$, and where $\mathbf{z}$ is the predicted inverse depth image.

We also use a loss term based on the relative pose between the source (s) and target (t) frame, $\tilde{\mathbf{p}}=(\tilde{\mathbf{t}}_{t\rightarrow s},\tilde{\boldsymbol{\alpha}}_{t\rightarrow s})$ with translation $\tilde{\mathbf{t}}_{t\rightarrow s}$ and rotation vector $\tilde{\boldsymbol{\alpha}}_{t\rightarrow s}$ from ground-truth data,

Note that this loss function need not be a sum of squares and can be computed using any other form using eg. L1 etc. The final loss function consists of a weighted combination of the individual loss terms:

Note that our objective here includes the ground-truth inverse depth which we do not have access to when computing the residuals $\mathbf{r}$ (and then the Jacobian $\mathbf{J}$) in the recurrent optimization network in Section 4.2.

The optimization network is never directly privy to the ground truth depth and poses, it only benefits from these by what is learned in the network parameters during training. In this manner, we have a system which is trained offline to best minimize our objective online. During the offline training phase, our system learns robust priors for the optimization by using the large amounts of labelled data. During the online phase our system optimizes for photometric consistency only but is able to utilize the knowledge it has learned during the offline training to better condition the optimization process.

During the training, we unroll our iterative optimization network for a set number of steps and backpropogate the loss through the network weights, $\theta$. In order to find the parameters of the optimizer network, the meta-loss, $L_{tot}(\mathbf{\theta})$, is minimized using the ADAM optimizer where the total meta-loss is computed as the loss summed over the $N$ iterations of the learned optimization (see Eq. 13). For each step $i$ in the optimization process we update the state $\mathbf{x}_{i}$ of the optimization network according to Eqn. 8.

As our loss depends on variables which are updated recurrently over a number of timesteps, we use backpropogation through time to train the network. Backpropogation through time unrolls each step and updates the parameters by computing the gradients through the unrolled network. In our experiments we unroll our optimization for 15 steps.

We find that training the whole network at once is difficult and thus train the initialization network first before adding the optimization stage.

In this section we evaluate the proposed method on both synthetic and real datasets. We aim to determine the efficiency of our approach i.e. how quickly it converges to an optimum and how it compares to a network which does not explicitly incorporate the problem structure in its iterations.

In the context of optimisation, our network-based updates accomplish something which a classical optimisation approach cannot in that it is able to reliably optimise a large under-determined system with implicitly learned priors.

For a large under-determined problem like in the depth and motion case, standard Levenberg-Marquadt (LM) fails to improve the objective and the required sparse matrix inversion for a $J^{T}J$ with $\approx 91K$ non-zero elements (128 $\times$ 96 size image) takes 532ms, compared to our network forward pass which takes 25ms. For small, overdetermined problems LM does work and for this reason, in Section 7.1, we have compared our approach to LM on a small curve fitting problem and found that our approach significantly outperforms it in terms of accuracy and convergence rate. For the small problem, the matrix inversion in the standard approach (LM) is very quick but we are also able to use a smaller network so our time per-iteration is tie with LM. This is summarised in Table 2.

In this paper we have presented an approach for robustly solving nonlinear least squares optimization problems by integrating deep neural models with traditional knowledge of the optimization structure. Our method is based on a novel nonlinear least squares optimizer which is trained to robustly optimize the residuals. Although it is generally applicable to any least squares problem, we have demonstrated the proposed method on the real-world problem of computing depth and egomotion for frames of a monocular video sequence. Our method can cope with images captured from a wide baseline. In future work we plan to investigate means of increasing the number of residuals that are optimized and thereby achieve an even more detailed prediction. We also plan to further study the interplay between the recurrent neural network and optimization structure and want to investigate the use of predicted confidence estimates in the learned optimization.