Abstract
The quantitative evaluation of optical flow algorithms by Barron et al. (1994) led to significant advances in performance. The challenges for optical flow algorithms today go beyond the datasets and evaluation methods proposed in that paper. Instead, they center on problems associated with complex natural scenes, including nonrigid motion, real sensor noise, and motion discontinuities. We propose a new set of benchmarks and evaluation methods for the next generation of optical flow algorithms. To that end, we contribute four types of data to test different aspects of optical flow algorithms: (1) sequences with nonrigid motion where the ground-truth flow is determined by tracking hidden fluorescent texture, (2) realistic synthetic sequences, (3) high frame-rate video used to study interpolation error, and (4) modified stereo sequences of static scenes. In addition to the average angular error used by Barron et al., we compute the absolute flow endpoint error, measures for frame interpolation error, improved statistics, and results at motion discontinuities and in textureless regions. In October 2007, we published the performance of several well-known methods on a preliminary version of our data to establish the current state of the art. We also made the data freely available on the web at http://vision.middlebury.edu/flow/. Subsequently a number of researchers have uploaded their results to our website and published papers using the data. A significant improvement in performance has already been achieved. In this paper we analyze the results obtained to date and draw a large number of conclusions from them.
Article PDF
Similar content being viewed by others
References
Adiv, G. (1985). Determining three-dimensional motion and structure from optical flow generated by several moving objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 7(4), 384–401.
Aggarwal, J., & Nandhakumar, N. (1988). On the computation of motion from sequences of images—a review. Proceedings of the IEEE, 76(8), 917–935.
Anandan, P. (1989). A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2(3), 283–310.
Anandan, P., & Weiss, R. (1985). Introducing smoothness constraint in a matching approach for the computation of displacement fields. In Proceedings of the DARPA image understanding workshop (pp. 186–196).
Baker, S., & Matthews, I. (2004). Lucas-Kanade 20 years on: a unifying framework. International Journal of Computer Vision, 46(3), 221–255.
Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., & Szeliski, R. (2007). A database and evaluation methodology for optical flow. In Proceedings of the IEEE international conference on computer vision.
Barron, J., Fleet, D., & Beauchemin, S. (1994). Performance of optical flow techniques. International Journal of Computer Vision, 12(1), 43–77.
Battiti, R., Amaldi, E., & Koch, C. (1991). Computing optical flow across multiple scales: an adaptive coarse-to-fine strategy. International Journal of Computer Vision, 6(2), 133–145.
Beier, T., & Neely, S. (1992). Feature-based image metamorphosis. In Annual conference series: Vol. 26(2). ACM computer graphics, SIGGRAPH (pp. 35–42).
Bergen, J., Anandan, P., Hanna, K., & Hingorani, R. (1992). Hierarchical model-based motion estimation. In Proceedings of the European conference on computer vision (pp. 237–252).
Black, M., & Anandan, P. (1991). Robust dynamic motion estimation over time. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 296–302).
Black, M., & Anandan, P. (1996). The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 63(1), 75–104.
Black, M., & Jepson, A. (1996). Estimating optical flow in segmented images using variable-order parametric models with local deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(10), 972–986.
Blake, A., & Zisserman, A. (1987). Visual reconstruction. Cambridge: MIT Press.
Bleyer, M., & Chambon, S. (2010). Does color really help in dense stereo matching? In Proceedings of the international symposium 3D data processing, visualization and transmission.
Bleyer, M., Rother, C., & Kohli, P. (2010). Surface stereo with soft segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Boykov, Y., Veksler, O., & Zabih, R. (2001). Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11), 1222–1239.
Brox, T., Bregler, C., & Malik, J. (2009). Large displacement optical flow. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Brox, T., Bruhn, A., Papenberg, N., &Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In Proceedings of the European Conference on Computer Vision (Vol. 4, pp. 25–36).
Bruhn, A., Weickert, J., & Schnörr, C. (2005). Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. International Journal of Computer Vision, 61(3), 211–231.
Bruhn, A., Weickert, J., Kohlberger, T., & Schnörr, C. (2006). A multigrid platform for real-time motion computation with discontinuity-preserving variational methods. International Journal of Computer Vision, 70(3), 257–277.
Burt, P., Yen, C., & Xu, X. (1982). Local correlation measures for motion analysis: a comparative study. In Proceedings of the IEEE conference on pattern recognition and image processing (pp. 269–274).
Burt, P., Yen, C., & Xu, X. (1983). Multi-resolution flow-through motion analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 246–252).
Cassisa, C., Simoens, S., & Prinet, V. (2009). Two-frame optical flow formulation in an unwarped multiresolution scheme. In Proceedings of the Iberoamerican congress on pattern recognition (pp. 790–797).
Cooke, T. (2008). Two applications of graph-cuts to image processing. In Proceedings of digital image computing: techniques and applications (pp. 498–504).
DNA Research (2008). 3Delight rendering software. http://www.3delight.com/.
Enkelman, W. (1986). Investigations of multigrid algorithms for the estimation of optical flow fields in image sequences. In Proceedings of the workshop on motion: representations and analysis (pp. 81–87).
Everingham, M., Van Gool, L., Williams, C., Winn, J., & Zisserman, A. (2009). The PASCAL visual object classes challenge 2009. http://www.pascal-network.org/challenges/VOC/voc2009/workshop/index.html
Fei-Fei, L., Fergus, R., & Perona, P. (2006). One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4), 594–611.
Fleet, D., & Jepson, A. (1990). Computation of component image velocity from local phase information. International Journal of Computer Vision, 5(1), 77–104.
Fuh, C., & Maragos, P. (1989). Region-based optical flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 130–135).
Georghiades, A., Belhumeur, P., & Kriegman, D. (2001). From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6), 643–660.
Glazer, F., Reyonds, G., & Anandan, P. (1983). Scene matching by hierarchical correlation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 432–441).
Glocker, B., Paragios, N., Komodakis, N., Tziritas, G., & Navab, N. (2008). Optical flow estimation with uncertainties through dynamic MRFs. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Golland, P., & Bruckstein, A. (1997). Motion from color. Computer Vision and Image Understanding, 68(3), 346–362.
Gross, R., Matthews, I., Cohn, J., Kanade, T., & Baker, S. (2008). Multi-PIE. In Proceedings of the international conference on automatic face and gesture recognition.
Hanna, K. (1991). Direct multi-resolution estimation of ego-motion and structure from motion. In Proceedings of the IEEE workshop on visual motion (pp. 156–162).
Haussecker, H., & Fleet, D. (2000). Computing optical flow with physical models of brightness variation. In Proceedings of the IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 760–767).
Herbst, E., Seitz, S., & Baker, S. (2009). Occlusion reasoning for temporal interpolation using optical flow. Technical report UW-CSE-09-08-01, Department of Computer Science and Engineering University of Washington.
Horn, B. (1986). Robot vision. Cambridge: MIT Press.
Horn, B., & Schunck, B. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203.
Jepson, A., & Black, M. (1993). Mixture models for optical flow computation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 760–761).
Ju, S. (1998). Estimating image motion in layers: the skin and bones model. PhD thesis, Department of Computer Science, University of Toronto.
Ju, S., Black, M., & Jepson, A. (1996). Skin and bones: multi-layer, locally affine, optical flow and regularization of transparency. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 307–314).
Jung, H., Lee, K., & Lee, S. (2008). Toward global minimum through combined local minima. In Proceedings of the European conference on computer vision (Vol. 4, pp. 298–311).
Landis, H. (2002). Production-ready global illumination. In L. Gritz (Ed.), RenderMan in production: SIGGRAPH 2002 course 16 (pp. 87–100). New York: ACM.
Le Besnerais, G., & Champagnat, F. (2005). Dense optical flow by iterative local window registration. In Proceedings of the international conference on image processing (Vol. 1, pp. 137–140).
Lei, C., & Yang, Y. (2009). Optical flow estimation on coarse-to-fine region-trees using discrete optimization. In Proceedings of the IEEE international conference on computer vision.
Lempitsky, V., Roth, S., & Rother, C. (2008). Fusion flow: discrete-continuous optimization for optical flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Levoy, M. (1988). Display of surfaces from volume data. IEEE Computer Graphics and Applications, 8(3), 29–37.
Li, Y., & Huttenlocher, D. (2008). Learning for optical flow using stochastic optimization. In Proceedings of the European conference on computer vision (Vol. 2, pp. 373–391).
Liu, C., Freeman, W., Adelson, E., & Weiss, Y. (2008). Human-assisted motion annotation. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Liu, C., Yuen, J., Torralba, A., Sivic, J., & Freeman, W. (2008). SIFT flow: dense correspondence across difference scenes. In Proceedings of the European conference on computer vision (Vol. 3, pp. 28–42).
Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application in stereo vision. In Proceedings of the international joint conference on artificial intelligence (pp. 674–679).
Mahajan, D., Huang, F., Matusik, W., Ramamoorthi, R., & Belhumeur, P. (2009). Moving gradients: a path-based method for plausible image interpolation. In Annual conference series. ACM computer graphics, SIGGRAPH.
Markandey, V., & Flinchbaugh, B. (1990). Multispectral constraints for optical flow computation. In Proceedings of the IEEE international conference on computer vision (pp. 38–41).
McCane, B., Novins, K., Crannitch, D., & Galvin, B. (2001). On benchmarking optical flow. Computer Vision and Image Understanding, 84(1), 126–143.
Mitiche, A., & Bouthemy, P. (1996). Computation and analysis of image motion: a synopsis of current problems and methods. International Journal of Computer Vision, 19(1), 29–55.
Mova LLC (2004). Contour reality capture. http://www.mova.com/.
Murray, D., & Buxton, B. (1987). Scene segmentation from visual motion using global optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(2), 220–228.
Nagel, H.-H., & Enkelmann, W. (1986). An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(5), 565–593.
Negahdaripour, S. (1998). Revised definition of optical flow: integration of radiometric and geometric cues for dynamic scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(9), 961–979.
Nir, T., Bruckstein, A., & Kimmel, R. (2008). Over-parameterized variational optical flow. International Journal of Computer Vision, 76(2), 205–216.
Ohta, N. (1989). Optical flow detection by color images. In International conference on image processing (pp. 801–805).
Otte, M., & Nagel, H.-H. (1994). Optical flow estimation: advances and comparisons. In Proceedings of the European conference on computer vision (pp. 51–60).
Philips, P., Scruggs, W., O’Toole, A., Flynn, P., Bowyer, K., Schott, C., & Sharpe, M. (2005). Overview of the face recognition grand challenge. In Proceedings of the IEEE conference on computer vision and pattern recognition (Vol. 1, pp. 947–954).
Pock, T., Pock, M., & Bischof, H. (2007). Algorithmic differentiation: application to variational problems in computer vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(7), 1180–1193.
Pratt, W. (1974). Correlation techniques of image registration. IEEE Transactions on Aerospace and Electronic Systems, AES-10, 353–358.
Ramnath, K., Baker, S., Matthews, I., & Ramanan, D. (2008). Increasing the density of active appearance models. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Rannacher, J. (2009). Realtime 3D motion estimation on graphics hardware. Undergraduate thesis, Heidelberg University.
Ren, X. (2008). Local grouping for optical flow. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Roth, S., & Black, M. (2007). On the spatial statistics of optical flow. International Journal of Computer Vision, 74(1), 33–50.
Scharstein, D., & Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(13), 7–42.
Scharstein, D., & Szeliski, R. (2003). High-accuracy stereo depth maps using structured light. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 195–202).
Seitz, S., & Baker, S. (2009). Filter flow. In Proceedings of the IEEE international conference on computer vision.
Seitz, S., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006). A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proceedings of the IEEE conference on computer vision and pattern recognition (Vol. 1, pp. 519–526).
Shade, J., Gortler, S., He, L.-W., & Szeliski, R. (1998). Layered depth images. In Annual conference series. ACM computer graphics, SIGGRAPH (pp. 231–242).
Shizawa, M., & Mase, K. (1991). A unified computational theory for motion transparency and motion boundaries based on eigenenergy analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 289–295).
Sim, T., Baker, S., & Bsat, M. (2003). The CMU pose, illumination, and expression database. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(12), 1615–1618.
Stiller, C., & Konrad, J. (1999). Estimating motion in image sequences: a tutorial on modeling and computation of 2D motion. IEEE Signal Processing Magazine, 16(4), 70–91.
Sun, C. (1999). Fast optical flow using cross correlation and shortest-path techniques. In Proceedings of digital image computing: techniques and applications (pp. 143–148).
Sun, J., Shum, H.-Y., & Zheng, N. (2003). Stereo matching using belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7), 787–800.
Sun, D., Roth, S., Lewis, J., & Black, M. (2008). Learning optical flow. In Proceedings of the European conference on computer vision (Vol. 3, pp. 83–97).
Sun, D., Roth, S., & Black, M. (2010). Secrets of optical flow estimation and their principles. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Szeliski, R. (1999). Prediction error as a quality metric for motion and stereo. In Proceedings of the IEEE international conference on computer vision (pp. 781–788).
Tappen, M., Adelson, E., & Freeman, W. (2006). Estimating intrinsic component images using non-linear regression. In Proceedings of the IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 1992–1999).
Trobin, W., Pock, T., Cremers, D., & Bischof, H. (2008). Continuous energy minimization via repeated binary fusion. In Proceedings of the European conference on computer vision (Vol. 4, pp. 677–690).
Trobin, W., Pock, T., Cremers, D., & Bischof, H. (2008). An unbiased second-order prior for high-accuracy motion estimation. In Proceedings of pattern recognition, DAGM (pp. 396–405).
Valgaerts, L., Bruhn, A., & Weickert, J. (2008). A variational model for the joint recovery of the fundamental matrix and the optical flow. In Proceedings of pattern recognition, DAGM (pp. 314–324).
Vedula, S., Baker, S., Rander, P., Collins, R., & Kanade, T. (2005). Three-dimensional scene flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3), 475–480.
Wang, J., & Adelson, E. (1993). Layered representation for motion analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 361–366).
Wedel, A., Pock, T., Braun, J., Franke, U., & Cremers, D. (2008). Duality TV-L1 flow with fundamental matrix prior. In Proceedings of image and vision computing, New Zealand.
Wedel, A., Pock, T., Zach, C., Cremers, D., & Bischof, H. (2008). An improved algorithm for TV-L1 optical flow. In Proceedings of the Dagstuhl motion workshop.
Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009). Structure- and motion-adaptive regularization for high accuracy optic flow. In Proceedings of the IEEE international conference on computer vision.
Weiss, Y. (1997). Smoothness in layers: motion segmentation using nonparametric mixture estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 520–526).
Werlberger, M., Trobin, W., Pock, T., Bischof, H., Wedel, A., & Cremers, D. (2009). Anisotropic Huber-L1 optical flow. In Proceedings of the British machine vision conference.
Xu, L., Chen, J., & Jia, J. (2008). A segmentation based variational model for accurate optical flow estimation. In Proceedings of the European conference on computer vision (Vol. 1, pp. 671–684).
Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., & Seidel, H.-P. (2009). Complementary optic flow. In Proceedings of seventh international workshop on energy minimization methods in computer vision and pattern recognition.
Zitnick, C., Kang, S., Uyttendaele, M., Winder, S., & Szeliski, R. (2004). High-quality video view interpolation using a layered representation. In Annual conference series: Vol. 23(2). ACM computer graphics, SIGGRAPH (pp. 600–608).
Author information
Authors and Affiliations
Corresponding author
Additional information
A preliminary version of this paper appeared in the IEEE International Conference on Computer Vision (Baker et al. 2007).
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Baker, S., Scharstein, D., Lewis, J.P. et al. A Database and Evaluation Methodology for Optical Flow. Int J Comput Vis 92, 1–31 (2011). https://doi.org/10.1007/s11263-010-0390-2
Received:
Accepted:
Published:
Issue date:
DOI: https://doi.org/10.1007/s11263-010-0390-2