{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T22:33:38Z","timestamp":1783377218004,"version":"3.54.6"},"reference-count":54,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2021,5,27]],"date-time":"2021-05-27T00:00:00Z","timestamp":1622073600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Seoul National University of Science and Technology","award":["2018-0779"],"award-info":[{"award-number":["2018-0779"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Motion in videos refers to the pattern of the apparent movement of objects, surfaces, and edges over image sequences caused by the relative movement between a camera and a scene. Motion, as well as scene appearance, are essential features to estimate a driver\u2019s visual attention allocation in computer vision. However, the fact that motion can be a crucial factor in a driver\u2019s attention estimation has not been thoroughly studied in the literature, although driver\u2019s attention prediction models focusing on scene appearance have been well studied. Therefore, in this work, we investigate the usefulness of motion information in estimating a driver\u2019s visual attention. To analyze the effectiveness of motion information, we develop a deep neural network framework that provides attention locations and attention levels using optical flow maps, which represent the movements of contents in videos. We validate the performance of the proposed motion-based prediction model by comparing it to the performance of the current state-of-art prediction models using RGB frames. The experimental results for a real-world dataset confirm our hypothesis that motion plays a role in prediction accuracy improvement, and there is a margin for accuracy improvement by using motion features.<\/jats:p>","DOI":"10.3390\/s21113722","type":"journal-article","created":{"date-parts":[[2021,5,27]],"date-time":"2021-05-27T11:07:02Z","timestamp":1622113622000},"page":"3722","update-policy":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["A Driver\u2019s Visual Attention Prediction Using Optical Flow"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0003-2537-7720","authenticated-orcid":false,"given":"Byeongkeun","family":"Kang","sequence":"first","affiliation":[{"name":"Department of Electronic and IT Media Engineering, Seoul National University of Science and Technology, Seoul 01811, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0002-3439-5042","authenticated-orcid":false,"given":"Yeejin","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Electrical and Information Engineering, Seoul National University of Science and Technology, Seoul 01811, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1016\/j.patcog.2016.08.029","article-title":"Are All Objects Equal? Deep Spatio-temporal Importance Prediction in Driving Videos","volume":"64","author":"Trivedi","year":"2017","journal-title":"Pattern Recognit."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"24972","DOI":"10.1073\/pnas.1820676116","article-title":"Social Behavior for Autonomous Vehicles","volume":"116","author":"Schwarting","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kim, I.H., Bong, J.H., Park, J., and Park, S. (2017). Prediction of Driver\u2019s Intention of Lane Change by Augmenting Sensor Information Using Machine Learning Techniques. Sensors, 17.","DOI":"10.3390\/s17061350"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Mart\u00ednez-Garc\u00eda, M., and Gordon, T. (2018, January 7\u201310). A New Model of Human Steering Using Far-point Error Perception and Multiplicative Control. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, Miyazaki, Japan.","DOI":"10.1109\/SMC.2018.00218"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Chang, S., Zhang, Y., Zhang, F., Zhao, X., Huang, S., Feng, Z., and Wei, Z. (2020). Spatial Attention Fusion for Obstacle Detection Using MmWave Radar and Vision Sensor. Sensors, 20.","DOI":"10.3390\/s20040956"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Martinez-Garcia, M., Kalawsky, R.S., Gordon, T., Smith, T., Meng, Q., and Flemisch, F. (2020). Communication and Interaction with Semiautonomous Ground Vehicles by Force Control Steering. IEEE Trans. Cybern.","DOI":"10.1109\/TCYB.2020.3020217"},{"key":"ref_7","unstructured":"Bylinskii, Z., Judd, T., Borji, A., Itti, L., Durand, F., Oliva, A., and Torralba, A. (2021, May 12). MIT Saliency Benchmark. Available online: https:\/\/2.zoppoz.workers.dev:443\/http\/saliency.mit.edu\/."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1109\/34.730558","article-title":"A Model of Saliency-based Visual Attention for Rapid Scene Analysis","volume":"20","author":"Itti","year":"1998","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_9","unstructured":"Judd, T., Durand, F., and Torralba, A. (2012). A Benchmark of Computational Models of Saliency to Predict Human Fixations, MIT Technical Report; MIT."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1109\/TPAMI.2009.112","article-title":"Spatiotemporal Saliency in Dynamic Scenes","volume":"32","author":"Mahadevan","year":"2009","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zhong, S.H., Liu, Y., Ren, F., Zhang, J., and Ren, T. (2013, January 14\u201318). Video Saliency Detection via Dynamic Consistent Spatio-temporal Attention Modelling. Proceedings of the AAAI Conference on Artificial Intelligence, Bellevue, WA, USA.","DOI":"10.1609\/aaai.v27i1.8642"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"4185","DOI":"10.1109\/TIP.2015.2460013","article-title":"Consistent Video Saliency Using Local Gradient Flow Optimization and Global Refinement","volume":"24","author":"Wang","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1109\/TPAMI.2012.89","article-title":"State-of-the-Art in Visual Attention Modeling","volume":"35","author":"Borji","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1937","DOI":"10.1016\/0042-6989(93)90020-W","article-title":"The Role of Features in Preattentive Vision: Comparison of Orientation, Motion and Color Cues","volume":"33","author":"Nothdurft","year":"1993","journal-title":"Vis. Res."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"725","DOI":"10.1016\/S0896-6273(00)81208-8","article-title":"Segregation of Object and Background Motion in Visual Area MT: Effects of Microstimulation on Eye Movements","volume":"26","author":"Born","year":"2000","journal-title":"Neuron"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Kang, B., and Lee, Y. (2020). High-Resolution Neural Network for Driver Visual Attention Prediction. Sensors, 20.","DOI":"10.3390\/s20072030"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5424","DOI":"10.1109\/TVT.2015.2487826","article-title":"How Much of Driving Is Preattentive?","volume":"64","author":"Pugeault","year":"2015","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Tawari, A., and Kang, B. (2017, January 11\u201314). A Computational Framework for Driver\u2019s Visual Attention Using a Fully Convolutional Architecture. Proceedings of the IEEE Intelligent Vehicles Symposium, Redondo Beach, CA, USA.","DOI":"10.1109\/IVS.2017.7995828"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully Convolutional Networks for Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1109\/TPAMI.2016.2572683","article-title":"Fully Convolutional Networks for Semantic Segmentation","volume":"39","author":"Shelhamer","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_21","unstructured":"Yu, F., and Koltun, V. (2016, January 2\u20134). Multi-Scale Context Aggregation by Dilated Convolutions. Proceedings of the International Conference on Learning Representations, Caribe Hilton, San Juan, Puerto Rico."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2481","DOI":"10.1109\/TPAMI.2016.2644615","article-title":"SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation","volume":"39","author":"Badrinarayanan","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Navab, N., Hornegger, J., Wells, W.M., and Frangi, A.F. (2015, January 5\u20139). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24571-3"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Noh, H., Hong, S., and Han, B. (2015, January 7\u201313). Learning Deconvolution Network for Semantic Segmentation. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.178"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"834","DOI":"10.1109\/TPAMI.2017.2699184","article-title":"DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs","volume":"40","author":"Chen","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_26","unstructured":"Sun, K., Zhao, Y., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang, X., Liu, W., and Wang, J. (2019). High-Resolution Representations for Labeling Pixels and Regions. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21\u201326). Pyramid Scene Parsing Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.660"},{"key":"ref_28","unstructured":"Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018, January 8\u201314). ICNet for Real-Time Semantic Segmentation on High-Resolution Images. Proceedings of the European Conference on Computer Vision, Munich, Germany."},{"key":"ref_29","unstructured":"Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018, January 8\u201314). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Proceedings of the European Conference on Computer Vision, Munich, Germany."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Lin, G., Milan, A., Shen, C., and Reid, I. (2017, January 21\u201326). RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.549"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Pohlen, T., Hermans, A., Mathias, M., and Leibe, B. (2017, January 21\u201326). Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.353"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Yang, M., Yu, K., Zhang, C., Li, Z., and Yang, K. (2018, January 18\u201323). DenseASPP for Semantic Segmentation in Street Scenes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00388"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"766","DOI":"10.1037\/0033-295X.113.4.766","article-title":"Contextual Guidance of Eye Movements and Attention in Real-world Scenes: The Role of Global Features in Object Search","volume":"113","author":"Torralba","year":"2006","journal-title":"Psychol. Rev."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"802","DOI":"10.1109\/TIP.2011.2162740","article-title":"Combining Head Pose and Eye Location Information for Gaze Estimation","volume":"21","author":"Valenti","year":"2012","journal-title":"IEEE Trans. Image Process."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Dobnikar, A., Lotri\u010d, U., and \u0160ter, B. (2011). Using Pattern Recognition to Predict Driver Intent. Adaptive and Natural Computing Algorithms, Springer.","DOI":"10.1007\/978-3-642-20267-4"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1016\/S1364-6613(00)01513-8","article-title":"Steering with or without the Flow: Is the Retrieval of Heading Necessary?","volume":"4","author":"Wann","year":"2000","journal-title":"Trends Cogn. Sci."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Perko, R., Schwarz, M., and Paletta, L. (2014, January 27\u201330). Aggregated Mapping of Driver Attention from Matched Optical Flow. Proceedings of the IEEE International Conference on Image Processing, Paris, France.","DOI":"10.1109\/ICIP.2014.7025042"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1016\/j.ifacol.2019.12.079","article-title":"Theoretical Interpretation of Driver\u2019s Gaze Considering Optic Flow and Seat Position","volume":"52","author":"Okafuji","year":"2019","journal-title":"IFAC PapersOnLine"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Tawari, A., and Kang, B. (2018). Systems and Methods of a Computational Framework for a Driver\u2019s Visual Attention Using a Fully Convolutional Architecture. (US20180225554A1), U.S. Patent.","DOI":"10.1109\/IVS.2017.7995828"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1720","DOI":"10.1109\/TPAMI.2018.2845370","article-title":"Predicting the Driver\u2019s Focus of Attention: The DR (eye) VE Project","volume":"41","author":"Palazzi","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Xia, Y., Zhang, D., Kim, J., Nakayama, K., Zipser, K., and Whitney, D. (2018, January 2\u20136). Predicting Driver Attention in Critical Situations. Proceedings of the Asian Conference on Computer Vision, Perth, Australia.","DOI":"10.1007\/978-3-030-20873-8_42"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Rudoy, D., Goldman, D.B., Shechtman, E., and Zelnik-Manor, L. (2013, January 23\u201328). Learning Video Saliency from Human Gaze Using Candidate Selection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.152"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Wang, W., Shen, J., Guo, F., Cheng, M.M., and Borji, A. (2018, January 18\u201323). Revisiting Video Saliency: A Large-scale Benchmark and a NewMmodel. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00514"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Gorji, S., and Clark, J.J. (2018, January 18\u201323). Going from Image to Video Saliency: Augmenting Image Salience with Dynamic Attentional Push. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00783"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/0004-3702(81)90024-2","article-title":"Determining Optical Flow","volume":"17","author":"Horn","year":"1981","journal-title":"Artif. intell."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Bigun, J., and Gustavsson, T. (2003). Two-Frame Motion Estimation Based on Polynomial Expansion. Image Analysis, Springer.","DOI":"10.1007\/3-540-45103-X"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Sun, K., Xiao, B., Liu, D., and Wang, J. (2019, January 15\u201320). Deep High-Resolution Representation Learning for Human Pose Estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00584"},{"key":"ref_48","unstructured":"Pereira, F., Burges, C.J.C., Bottou, L., and Weinberger, K.Q. (2012, January 3\u20136). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_49","unstructured":"Simonyan, K., and Zisserman, A. (2015, January 7\u20139). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA."},{"key":"ref_50","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_52","unstructured":"Harel, J., Koch, C., and Perona, P. (2006, January 4\u20137). Graph-Based Visual Saliency. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_53","unstructured":"Leon-Garcia, A. (2017). Probability, Statistics, and Random Processes for Electrical Engineering, Pearson Education."},{"key":"ref_54","unstructured":"Williams, R.H. (2003). Probability, Statistics, and Random Processes for Engineers, Cl-Engineering."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/www.mdpi.com\/1424-8220\/21\/11\/3722\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:08:49Z","timestamp":1760162929000},"score":1,"resource":{"primary":{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/www.mdpi.com\/1424-8220\/21\/11\/3722"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,27]]},"references-count":54,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["s21113722"],"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.3390\/s21113722","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,27]]}}}