Addressing Global Challenges and Quality

Education 15th European Conference on Technology

Enhanced Learning EC TEL 2020 Heidelberg Germany
September 14 18 2020 Proceedings Carlos Alario-
Hoyos pdf download
https://textbookfull.com/product/addressing-global-challenges-and-quality-education-15th-european-
conference-on-technology-enhanced-learning-ec-tel-2020-heidelberg-germany-
september-14-18-2020-proceedings-carlos-alario-hoyos/
★★★★★ 4.8/5.0 (39 reviews) ✓ 187 downloads ■ TOP RATED
"Fantastic PDF quality, very satisfied with download!" - Emma W.

DOWNLOAD EBOOK
 Addressing Global Challenges and Quality Education 15th
European Conference on Technology Enhanced Learning EC TEL
2020 Heidelberg Germany September 14 18 2020 Proceedings
Carlos Alario-Hoyos pdf download

TEXTBOOK EBOOK TEXTBOOK FULL

Available Formats

■ PDF eBook Study Guide TextBook

EXCLUSIVE 2025 EDUCATIONAL COLLECTION - LIMITED TIME

INSTANT DOWNLOAD VIEW LIBRARY

Collection Highlights

Lifelong Technology-Enhanced Learning: 13th European

Conference on Technology Enhanced Learning, EC-TEL 2018,
Leeds, UK, September 3-5, 2018, Proceedings Viktoria
Pammer-Schindler

Adaptive and Adaptable Learning 11th European Conference

on Technology Enhanced Learning EC TEL 2016 Lyon France
September 13 16 2016 Proceedings 1st Edition Katrien
Verbert

Transforming Learning with Meaningful Technologies 14th

European Conference on Technology Enhanced Learning EC TEL
2019 Delft The Netherlands September 16 19 2019
Proceedings Maren Scheffel

Software Architecture 14th European Conference ECSA 2020 L

Aquila Italy September 14 18 2020 Proceedings Anton Jansen
Computer Security ESORICS 2020 25th European Symposium on
Research in Computer Security ESORICS 2020 Guildford UK
September 14 18 2020 Proceedings Part II Liqun Chen

Software Engineering and Formal Methods: 18th

International Conference, SEFM 2020, Amsterdam, The
Netherlands, September 14–18, 2020, Proceedings Frank De
Boer

Computer Vision ECCV 2018 15th European Conference Munich

Germany September 8 14 2018 Proceedings Part III Vittorio
Ferrari

Computer Vision ECCV 2018 15th European Conference Munich

Germany September 8 14 2018 Proceedings Part IV Vittorio
Ferrari

Computer Vision ECCV 2018 15th European Conference Munich

Germany September 8 14 2018 Proceedings Part VII Vittorio
Ferrari
 Carlos Alario-Hoyos
María Jesús Rodríguez-Triana
Maren Scheffel
Inmaculada Arnedillo-Sánchez
Sebastian Maximilian Dennerlein (Eds.)

Addressing
LNCS 12315

Global Challenges
and Quality Education
15th European Conference
on Technology Enhanced Learning, EC-TEL 2020
Heidelberg, Germany, September 14–18, 2020
Proceedings
Lecture Notes in Computer Science 12315

Founding Editors
Gerhard Goos
Karlsruhe Institute of Technology, Karlsruhe, Germany
Juris Hartmanis
Cornell University, Ithaca, NY, USA

Editorial Board Members

Elisa Bertino
Purdue University, West Lafayette, IN, USA
Wen Gao
Peking University, Beijing, China
Bernhard Steffen
TU Dortmund University, Dortmund, Germany
Gerhard Woeginger
RWTH Aachen, Aachen, Germany
Moti Yung
Columbia University, New York, NY, USA
More information about this series at http://www.springer.com/series/7409
Carlos Alario-Hoyos María Jesús Rodríguez-Triana
• •

Maren Scheffel Inmaculada Arnedillo-Sánchez

• •

Sebastian Maximilian Dennerlein (Eds.)

Addressing
Global Challenges
and Quality Education
15th European Conference
on Technology Enhanced Learning, EC-TEL 2020
Heidelberg, Germany, September 14–18, 2020
Proceedings

123
Editors
Carlos Alario-Hoyos María Jesús Rodríguez-Triana
Universidad Carlos III de Madrid Tallinn University
Leganés (Madrid), Spain Tallinn, Estonia
Maren Scheffel Inmaculada Arnedillo-Sánchez
Open University Netherlands Trinity College Dublin
Heerlen, The Netherlands Dublin, Ireland
Sebastian Maximilian Dennerlein
Graz University of Technology
and Know-Center
Graz, Austria

ISSN 0302-9743 ISSN 1611-3349 (electronic)

Lecture Notes in Computer Science
ISBN 978-3-030-57716-2 ISBN 978-3-030-57717-9 (eBook)
https://doi.org/10.1007/978-3-030-57717-9

LNCS Sublibrary: SL3 – Information Systems and Applications, incl. Internet/Web, and HCI

© Springer Nature Switzerland AG 2020

Chapters 6, 10, 15 and 48 are licensed under the terms of the Creative Commons Attribution 4.0 International
License (http://creativecommons.org/licenses/by/4.0/). For further details see licence information in the
chapters.
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book are
believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors
give a warranty, expressed or implied, with respect to the material contained herein or for any errors or
omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
 Preface

Welcome to the proceedings of the 15th European Conference on Technology

Enhanced Learning (EC-TEL 2020), one of the flagship events of the European
Association of Technology Enhanced Learning (EATEL). In addition to the social and
economic crisis, the pandemic has led to a historic inflection point in education. The
impact of COVID-19 worldwide and the closure of schools and universities has led to
numerous changes in how, where and with which tools we learn, and how education
can take place at a distance. One of these changes is that lockdowns triggered the use of
Technology-Enhanced Learning (TEL) at a massive level and throughout the world due
to the urgent need to transform face-to-face into online classes. Such circumstances
pose new challenges and research questions that our community must address.
Looking back, the theme of EC-TEL 2020 (Addressing Global Challenges and
Quality Education) could not have been more timely with COVID-19 affecting edu-
cation and its quality all over the world. Thus, the Sustainable Development Goals of
Quality Education and Reduced Inequalities, as defined in the United Nations Agenda
2030, are more essential than ever. In a global world of great contrasts, we are all
suffering the consequences of this unexpected pandemic, and it is paramount to find
global solutions to provide equitable quality education and promote lifelong learning
opportunities for all. This not only requires improving student performance and success
in general but also supporting under-represented groups and those disadvantaged by
inequality. In this challenging context, the active role of the TEL community may be
crucial to pursue the aforementioned goals.
Another important change this year is that EC-TEL 2020 will be held online for the
first time in its 15-year history. Although EC-TEL 2020 was initially planned to take
place in Heidelberg, Germany, local health authorities advised against organizing large
events. What was to be a major liability has been transformed into a powerful expe-
rience in which to continue learning and evolving as a community thanks to the
researchers who have continued to rely on this conference to submit and present their
contributions. This year, 91 contributions were received despite the difficulties of some
researchers to carry out developments and studies as well as reporting them. All
of these contributions were reviewed by three members of the TEL community, who
also had follow-up discussions to agree on a meta-review. As a result, 24 research
papers (26.4%) were accepted and presented at the conference. This shows the high
competitiveness and quality of this conference year after year. In addition, 20 posters, 5
demos, and 8 impact papers were presented during the conference to fuel the discus-
sions among the researchers. Research papers, posters, and demos can be found in this
volume, while impact papers are published in companion proceedings via CEUR.
EC-TEL 2020 was co-located with another major conference in Germany, the 18th
Fachtagung Bildungstechnologien der GI Fachgrupper Bildungstechnologien (DELFI
2020). This helped create partnerships between the two conferences and allowed
attendees to exchange ideas and benefit from presentations and workshops that were
vi Preface

offered in both conferences. As an example, 14 workshops and 4 keynotes (Linda

Castañeda, Jens Mönig, Samuel Greiff, and Sabine Seufert) were co-located between
EC-TEL 2020 and DELFI 2020.
The last words are words of gratitude. Thanks to the researchers who sent their
contributions to EC-TEL 2020. Thanks to the members of the Programme Committee
who devoted their time to give feedback to authors and supported decision making on
paper acceptance. Finally, deep thanks to the local organizers, Marco Kalz and Joshua
Weidlich, who have worked very hard to host this conference online with the same
quality as in previous years.

July 2020 Carlos Alario-Hoyos

María Jesús Rodríguez-Triana
Maren Scheffel
Inmaculada Arnedillo-Sánchez
Sebastian Dennerlein
Tracie Farrell Frey
Tom Broos
Zacharoula Papamitsiou
Adolfo Ruiz Calleja
Kairit Tammets
Christian Glahn
Marco Kalz
Joshua Weidlich
EC-TEL 2020 Chairs
 Organization

Program Committee
Marie-Helene Abel Université de Technologie de Compiègne, France
Andrea Adamoli Università della Svizzera Italiana, Switzerland
Nora Ayu Ahmad Uzir University of Edinburgh, UK
Gokce Akcayir University of Alberta, Canada
Carlos Alario-Hoyos Universidad Carlos III de Madrid, Spain
Patricia Albacete University of Pittsburgh, USA
Dietrich Albert University of Graz, Austria
Laia Albó Universitat Pompeu Fabra, Spain
Liaqat Ali Simon Fraser University, Canada
Ishari Amarasinghe Universitat Pompeu Fabra, Spain
Boyer Anne LORIA – KIWI, France
Alessandra Antonaci European Association of Distance Teaching
Universities, The Netherlands
Roberto Araya Universidad de Chile, Chile
Inmaculada Trinity College Dublin, Ireland
Arnedillo-Sanchez
Juan I. Asensio-Pérez Universidad de Valladolid, Spain
Antonio Balderas University of Cádiz, Spain
Nicolas Ballier Université de Paris, France
Jordan Barria-Pineda University of Pittsburgh, USA
Jason Bernard University of Saskatchewan, Canada
Anis Bey University of Paul Sabatier, IRIT, France
Miguel L. Bote-Lorenzo Universidad de Valladolid, Spain
François Bouchet Sorbonne Université, France
Yolaine Bourda LRI, CentraleSupélec, France
Bert Bredeweg University of Amsterdam, The Netherlands
Andreas Breiter Universität Bremen, Germany
Julien Broisin Université Toulouse 3 Paul Sabatier, IRIT, France
Tom Broos KU Leuven, Belgium
Armelle Brun LORIA, Université Nancy 2, France
Daniela Caballero Universidad de Chile, Chile
Manuel Caeiro Rodríguez University of Vigo, Spain
Sven Charleer KU Leuven, Belgium
Mohamed Amine Chatti University of Duisburg-Essen, Germany
John Cook University of West of England, UK
Audrey Cooke Curtin University, Australia
Alessia Coppi SFIVET, Switzerland
Mihai Dascalu University Politehnica of Bucharest, Romania
viii Organization

Peter de Lange RWTH Aachen University, Germany

Felipe de Morais UNISINOS, Brazil
Carrie Demmans Epp University of Alberta, Canada
Sebastian Dennerlein Know-Center, Austria
Michael Derntl University of Tübingen, Germany
Philippe Dessus Université Grenoble Alpes, LaRAC, France
Daniele Di Mitri Open Universiteit, The Netherlands
Yannis Dimitriadis Universidad de Valladolid, Spain
Monica Divitini Norwegian University of Science and Technology,
Norway
Juan Manuel Dodero Universidad de Cádiz, Spain
Raymond Elferink RayCom BV, The Netherlands
Erkan Er Universidad de Valladolid, Spain
Maka Eradze University of Modena and Reggio Emilia, Italy
Iria Estévez-Ayres Universidad Carlos III de Madrid, Spain
Juan Carlos Farah École Polytechnique Fédérale de Lausanne,
Switzerland
Tracie Farrell The Open University, UK
Louis Faucon École Polytechnique Fédérale de Lausanne,
Switzerland
Baltasar Fernandez-Manjon Universidad Complutense de Madrid, Spain
Carmen Universidad Carlos III de Madrid, Spain
Fernández-Panadero
Angela Fessl Know-Center, Austria
Anna Filighera Technical University of Darmstadt, Multimedia
Communications Lab, Germany
Olga Firssova WELTEN Institute, Open Universiteit,
The Netherlands
Mikhail Fominykh Norwegian University of Science and Technology,
Norway
Felix J. Garcia Clemente Universidad de Murcia, Spain
Jesús Miguel Universidad de la Sierra, Mexico
García-Gorrostieta
Serge Garlatti IMT Atlantique, France
Dragan Gasevic Monash University, Australia
Sébastien George LIUM, Le Mans Université, France
Carlo Giovannella University of Tor Vergata, Italy
Christian Glahn LDE CEL, The Netherlands
Samuel González-López Technological University of Nogales, Mexico
Sabine Graf Athabasca University, Canada
Monique Grandbastien LORIA, Université de Lorraine, France
Wolfgang Greller Vienna University of Education, Austria
David Griffiths University of Bolton, UK
Nathalie Guin Université de Lyon, France
Bernhard Göschlberger Research Studios, Austria
 Organization ix

Franziska Günther TU Dresden, Germany

Christian Gütl Graz University of Technology, Austria
Carolin Hahnel DIPF, Leibniz Institute for Research and Information
in Education, Germany
Stuart Hallifax Laboratoire d’InfoRmatique en Image et Systèmes
d'information, France
Bastiaan Heeren Open Universiteit, The Netherlands
Maartje Henderikx Open Universiteit, The Netherlands
Davinia Hernandez-Leo Universitat Pompeu Fabra, Spain
Ángel Hernández-García Universidad Politécnica de Madrid, Spain
Tore Hoel Oslo Metropolitan University, Norway
Adrian Holzer University of Neuchâtel, Switzerland
Pasquale Iero The Open University, UK
Francisco Iniesto The Open University, UK
Zeeshan Jan The Open University, UK
Patricia Jaques PPGCA, UNISINOS, Brazil
Johan Jeuring Utrecht University and Open Universiteit,
The Netherlands
Ioana Jivet Open Universiteit, The Netherlands
Srecko Joksimovic University of South Australia, Australia
Ilkka Jormanainen University of Eastern Finland, Finland
Jelena Jovanovic University of Belgrade, Serbia
Ken Kahn University of Oxford, UK
Rogers Kaliisa University of Oslo, Norway
Marco Kalz PH Heidelberg, Germany
Anastasios Karakostas Aristotle University of Thessaloniki, Greece
Reet Kasepalu Tallinn University, Estonia
Mohammad Khalil University of Bergen, Norway
Michael Kickmeier-Rust Graz University of Technology, Austria
Ralf Klamma RWTH Aachen University, Germany
Roland Klemke Open Universiteit, The Netherlands
Tomaž Klobučar Jozef Stefan Institute, Slovenia
Carolien Knoop-Van Radboud University, The Netherlands
Campen
Johannes Konert Hochschule Fulda, Germay
Külli Kori Tallinn University, Estonia
Panagiotis Kosmas Cyprus University of Technology, Cyprus
Vitomir Kovanovic University of South Australia, Australia
Dominik Kowald Know-Center, Austria
Milos Kravcik DFKI GmbH, Germay
Elise Lavoué Université Jean Moulin Lyon 3, France
Marie Lefevre Université Lyon 1, France
Dominique Lenne Université de Technologie de Compiègne, France
Marina Lepp University of Tartu, Estonia
Amna Liaqat University of Toronto, Canada
Paul Libbrecht IUBH Fernstudium, Germany
x Organization

Andreas Lingnau Ruhr West University of Applied Science, Germany

Martin Llamas-Nistal Universidad de Vigo, Spain
Aurelio Lopez-Lopez INAOE, Mexico
Domitile Lourdeaux CNRS, France
Margarida Lucas University of Aveiro, Portugal
Ulrike Lucke University of Potsdam, Germany
Vanda Luengo Sorbonne Université, France
George Magoulas University of London, UK
Jorge Maldonado-Mahauad Universidad de Cuenca, Ecuador, Pontificia
Universidad Católica de Chile, Chile
Nils Malzahn Rhine-Ruhr Institute for Applied System Innovation
e.V., Germany
Carlos Martínez-Gaitero Escuelas Universitarias Gimbernat, Spain
Alejandra Martínez-Monés Universidad de Valladolid, Spain
Wannisa Matcha University of Edinburgh, UK
Manolis Mavrikis UCL Knowledge Lab, UK
Agathe Merceron Beuth University of Applied Sciences, Germany
Vasileios Mezaris Information Technologies Institute, CERT, Greece
Christine Michel Université de Lyon, France
Konstantinos Michos Universidad de Valladolid, Spain
Alexander Mikroyannidis The Open University, UK
Tanja Mitrovic University of Canterbury, UK
Riichiro Mizoguchi Japan Advanced Institute of Science and Technology,
Japan
Inge Molenaar Radboud University, The Netherlands
Anders Morch University of Oslo, Norway
Pedro Manuel Universidad Carlos III de Madrid, Spain
Moreno-Marcos
Mathieu Muratet Sorbonne Université, France
Pedro J. Muñoz-Merino Universidad Carlos III de Madrid, Spain
Rob Nadolski Open Universiteit, The Netherlands
Petru Nicolaescu RWTH Aachen University, Germany
Stavros Nikou University of Strathclyde, UK
Nicolae Nistor Ludwig Maximilian University of Munich, Germany
Alexander Nussbaumer Graz University of Technology, Austria
Jennifer Olsen École Polytechnique Fédérale de Lausanne,
Switzerland
Alejandro Ortega-Arranz Universidad de Valladolid, Spain
Lahcen Oubahssi LIUM, Le Mans Université, France
Viktoria Pammer-Schindler Graz University of Technology, Austria
Sofia Papavlasopoulou Norwegian University of Science and Technology,
Norway
Abelardo Pardo University of South Australia, Australia
Cesare Pautasso University of Lugano, Switzerland
Maxime Pedrotti Verein zur Förderung eines Deutschen
Forschungsnetzes e.V., Germany
 Organization xi

Mar Perez-Sanagustin Université Paul Sabatier Toulouse III, France

Donatella Persico Istituto Tecnologie Didattiche, CNR, Italy
Yvan Peter Université de Lille, France
Niels Pinkwart Humboldt-Universität zu Berlin, Germany
Gerti Pishtari Tallinn University, Estonia
Elvira Popescu University of Craiova, Romania
Francesca Pozzi Istituto Tecnologie Didattiche, CNR, Italy
Luis P. Prieto Tallinn University, Estonia
Ronald Pérez Álvarez Universidad de Costa Rica, Costa Rica
Hans Põldoja Tallinn University, Estonia
Eyal Rabin The Open University of Israel, Israel, and Open
Universiteit, The Netherlands
Juliana Elisa Raffaghelli University of Florence, Italy
Gustavo Ramirez-Gonzalez Universidad del Cauca, Colombia
Marc Rittberger DIPF, Leibniz Institute for Researach and Information
in Education, Germany
Tiago Roberto Kautzmann Universidade do Vale do Rio dos Sinos, Brazil
Covadonga Rodrigo Universidad Nacional de Educación a Distancia, Spain
Maria Jesus Rodriguez Tallinn University, Estonia
Triana
Jeremy Roschelle Digital Promise, USA
José A. Ruipérez Valiente Universidad de Murcia, Spain
Ellen Rusman Open Universiteit, The Netherlands
Merike Saar Tallinn University, Estonia
Demetrios Sampson Curtin University, Australia
Eric Sanchez Université Fribourg, Switzerland
Patricia Santos Universitat Pompeu Fabra, Spain
Mohammed Saqr University of Eastern Finland, Finland
Petra Sauer Beuth University of Applied Sciences, Germany
Maren Scheffel Open Universiteit, The Netherlands
Daniel Schiffner DIPF, Leibniz Institute for Research and Information
in Education, Germany
Andreas Schmidt Karlsruhe University of Applied Sciences, Germany
Marcel Schmitz Zuyd Hogeschool, The Netherlands
Jan Schneider DIPF, Leibniz Institute for Research and Information
in Education, Germany
Ulrik Schroeder RWTH Aachen University, Germany
Stefan Schweiger Bongfish GmbH, Austria
Yann Secq Université de Lille, France
Karim Sehaba Laboratoire d’InfoRmatique en Image et Systèmes
d’information, Université Lumière Lyon 2, France
Paul Seitlinger Tallinn University, Estonia
Audrey Serna Laboratoire d’InfoRmatique en Image et Systèmes
d’information, France
Sergio Serrano-Iglesias Universidad de Valladolid, Spain
Shashi Kant Shankar Tallinn University, Estonia
xii Organization

Kshitij Sharma Norwegian University of Science and Technology,

Norway
Tanmay Sinha ETH Zurich, Switzerland
Sergey Sosnovsky Utrecht University, The Netherlands
Marcus Specht Delft University of Technology and Open Universiteit,
The Netherlands
Srinath Srinivasa International Institute of Information Technology,
Bangalore, India
Tim Steuer Technical University of Darmstadt, Germany
Slavi Stoyanov Open University, The Netherlands
Alexander Streicher Fraunhofer IOSB, Germany
Bernardo Tabuenca Universidad Politécnica de Madrid, Spain
Kairit Tammets Tallinn University, Estonia
Stefaan Ternier Open Universiteit, The Netherlands
Stefan Thalmann University of Graz, Austria
Paraskevi Topali Universidad de Valladolid, Spain
Richard Tortorella University of North Texas, USA
Lucia Uguina IMDEA Networks, Spain
Peter Van Rosmalen Maastricht University, The Netherlands
Olga Viberg KTH Royal Institute of Technology, Sweden
Markel Vigo The University of Manchester, UK
Sara Villagrá-Sobrino Universidad de Valladolid, Spain
Cathrin Vogel FernUniversität in Hagen, Germany
Joshua Weidlich Heidelberg University of Education, Germany
Armin Weinberger Saarland University, Germany
Professor Denise Whitelock The Open University, UK
Fridolin Wild The Open University, UK
Nikoletta Xenofontos University of Cyprus, Cyprus
Amel Yessad Sorbonne Université, France

Additional Reviewers

Anwar, Muhammad
Berns, Anke
Ebner, Markus
Ehlenz, Matthias
Halbherr, Tobias
Koenigstorfer, Florian
Kothiyal, Aditi
Liaqat, Daniyal
Liaqat, Salaar
Müllner, Peter
Ponce Mendoza, Ulises
Raffaghelli, Juliana
Rodriguez, Indelfonso
Zeiringer, Johannes
 Contents

Exploring Artificial Jabbering for Automatic Text Comprehension

Question Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Tim Steuer, Anna Filighera, and Christoph Rensing

Digital Value-Adding Chains in Vocational Education: Automatic

Keyword Extraction from Learning Videos to Provide Learning
Resource Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Cleo Schulten, Sven Manske, Angela Langner-Thiele,
and H. Ulrich Hoppe

Human-Centered Design of a Dashboard on Students’ Revisions

During Writing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Rianne Conijn, Luuk Van Waes, and Menno van Zaanen

An Operational Framework for Evaluating the Performance of Learning

Record Stores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Chahrazed Labba, Azim Roussanaly, and Anne Boyer

Does an E-mail Reminder Intervention with Learning Analytics Reduce

Procrastination in a Blended University Course? . . . . . . . . . . . . . . . . . . . . . 60
Iryna Nikolayeva, Amel Yessad, Bertrand Laforge, and Vanda Luengo

Designing an Online Self-assessment for Informed Study Decisions:

The User Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
L. E. C. Delnoij, J. P. W. Janssen, K. J. H. Dirkx, and R. L. Martens

What Teachers Need for Orchestrating Robotic Classrooms . . . . . . . . . . . . . 87

Sina Shahmoradi, Aditi Kothiyal, Jennifer K. Olsen, Barbara Bruno,
and Pierre Dillenbourg

Assessing Teacher’s Discourse Effect on Students’ Learning: A Keyword

Centrality Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Danner Schlotterbeck, Roberto Araya, Daniela Caballero,
Abelino Jimenez, Sami Lehesvuori, and Jouni Viiri

For Learners, with Learners: Identifying Indicators for an Academic

Advising Dashboard for Students . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Isabel Hilliger, Tinne De Laet, Valeria Henríquez, Julio Guerra,
Margarita Ortiz-Rojas, Miguel Ángel Zuñiga, Jorge Baier,
and Mar Pérez-Sanagustín
xiv Contents

Living with Learning Difficulties: Two Case Studies Exploring

the Relationship Between Emotion and Performance in Students
with Learning Difficulties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Styliani Siouli, Stylianos Makris, Evangelia Romanopoulou,
and Panagiotis P. D. Bamidis

Learnersourcing Quality Assessment of Explanations for Peer Instruction . . . 144

Sameer Bhatnagar, Amal Zouaq, Michel C. Desmarais,
and Elizabeth Charles

Using Diffusion Network Analytics to Examine and Support Knowledge

Construction in CSCL Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Mohammed Saqr and Olga Viberg

Supporting Second Language Learners’ Development of Affective

Self-regulated Learning Skills Through the Use and Design
of Mobile Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Olga Viberg, Anna Mavroudi, and Yanwen Ma

We Know What You Did Last Semester: Learners’ Perspectives on Screen

Recordings as a Long-Term Data Source for Learning Analytics. . . . . . . . . . 187
Philipp Krieter, Michael Viertel, and Andreas Breiter

Teaching Simulation Literacy with Evacuations: Concept, Technology,

and Material for a Novel Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Andre Greubel, Hans-Stefan Siller, and Martin Hennecke

Design of Conversational Agents for CSCL: Comparing Two Types

of Agent Intervention Strategies in a University Classroom . . . . . . . . . . . . . 215
Konstantinos Michos, Juan I. Asensio-Pérez, Yannis Dimitriadis,
Sara García-Sastre, Sara Villagrá-Sobrino, Alejandro Ortega-Arranz,
Eduardo Gómez-Sánchez, and Paraskevi Topali

Exploring Human–AI Control Over Dynamic Transitions Between

Individual and Collaborative Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Vanessa Echeverria, Kenneth Holstein, Jennifer Huang,
Jonathan Sewall, Nikol Rummel, and Vincent Aleven

Exploring Student-Controlled Social Comparison . . . . . . . . . . . . . . . . . . . . 244

Kamil Akhuseyinoglu, Jordan Barria-Pineda, Sergey Sosnovsky,
Anna-Lena Lamprecht, Julio Guerra, and Peter Brusilovsky

New Measures for Offline Evaluation of Learning Path Recommenders . . . . . 259

Zhao Zhang, Armelle Brun, and Anne Boyer
 Contents xv

Assessing the Impact of the Combination of Self-directed Learning,

Immediate Feedback and Visualizations on Student Engagement
in Online Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Bilal Yousuf, Owen Conlan, and Vincent Wade

CGVis: A Visualization-Based Learning Platform for Computational

Geometry Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
Athanasios Voulodimos, Paraskevas Karagiannopoulos,
Ifigenia Drosouli, and Georgios Miaoulis

How to Design Effective Learning Analytics Indicators? A Human-

Centered Design Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Mohamed Amine Chatti, Arham Muslim, Mouadh Guesmi,
Florian Richtscheid, Dawood Nasimi, Amin Shahin, and Ritesh Damera

Emergency Remote Teaching: Capturing Teacher Experiences in Spain

with SELFIE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Laia Albó, Marc Beardsley, Judit Martínez-Moreno, Patricia Santos,
and Davinia Hernández-Leo

Utilising Learnersourcing to Inform Design Loop Adaptivity . . . . . . . . . . . . 332

Ali Darvishi, Hassan Khosravi, and Shazia Sadiq

Fooling It - Student Attacks on Automatic Short Answer Grading. . . . . . . . . 347

Anna Filighera, Tim Steuer, and Christoph Rensing

Beyond Indicators: A Scoping Review of the Academic Literature

Related to SDG4 and Educational Technology . . . . . . . . . . . . . . . . . . . . . . 353
Katy Jordan

Pedagogical Underpinnings of Open Science, Citizen Science

and Open Innovation Activities: A State-of-the-Art Analysis. . . . . . . . . . . . . 358
Elisha Anne Teo and Evangelia Triantafyllou

Knowledge-Driven Wikipedia Article Recommendation

for Electronic Textbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
Behnam Rahdari, Peter Brusilovsky, Khushboo Thaker,
and Jordan Barria-Pineda

InfoBiTS: A Mobile Application to Foster Digital Competencies

of Senior Citizens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
Svenja Noichl and Ulrik Schroeder

Student Awareness and Privacy Perception of Learning Analytics

in Higher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
Stian Botnevik, Mohammad Khalil, and Barbara Wasson
xvi Contents

User Assistance for Serious Games Using Hidden Markov Model . . . . . . . . . 380
Vivek Yadav, Alexander Streicher, and Ajinkya Prabhune

Guiding Socio-Technical Reflection of Ethical Principles in TEL Software

Development: The SREP Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
Sebastian Dennerlein, Christof Wolf-Brenner, Robert Gutounig,
Stefan Schweiger, and Viktoria Pammer-Schindler

Git4School: A Dashboard for Supporting Teacher Interventions in Software

Engineering Courses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
Jean-Baptiste Raclet and Franck Silvestre

Exploring the Design and Impact of Online Exercises for Teacher Training
About Dynamic Models in Mathematics. . . . . . . . . . . . . . . . . . . . . . . . . . . 398
Charlie ter Horst, Laura Kubbe, Bart van de Rotten, Koen Peters,
Anders Bouwer, and Bert Bredeweg

Interactive Concept Cartoons: Exploring an Instrument for Developing

Scientific Literacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
Patricia Kruit and Bert Bredeweg

Quality Evaluation of Open Educational Resources . . . . . . . . . . . . . . . . . . . 410

Mirette Elias, Allard Oelen, Mohammadreza Tavakoli, Gábor Kismihok,
and Sören Auer

Designing Digital Activities to Screen Locomotor Skills

in Developing Children . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
Benoit Bossavit and Inmaculada Arnedillo-Sánchez

Towards Adaptive Social Comparison for Education . . . . . . . . . . . . . . . . . . 421

Sergey Sosnovsky, Qixiang Fang, Benjamin de Vries, Sven Luehof,
and Fred Wiegant

Simulation Based Assessment of Epistemological Beliefs About Science . . . . 427

Melanie E. Peffer and Tessa Youmans

An Approach to Support Interactive Activities in Live Stream Lectures . . . . . 432

Tommy Kubica, Tenshi Hara, Iris Braun, and Alexander Schill

Educational Escape Games for Mixed Reality . . . . . . . . . . . . . . . . . . . . . . . 437

Ralf Klamma, Daniel Sous, Benedikt Hensen, and István Koren

Measuring Learning Progress for Serving Immediate Feedback Needs:

Learning Process Quantification Framework (LPQF) . . . . . . . . . . . . . . . . . . 443
Gayane Sedrakyan, Sebastian Dannerlein, Viktoria Pammer-Schindler,
and Stefanie Lindstaedt
 Contents xvii

Data-Driven Game Design: The Case of Difficulty in Educational Games . . . 449

Yoon Jeon Kim and Jose A. Ruipérez-Valiente

Extracting Topics from Open Educational Resources . . . . . . . . . . . . . . . . . . 455

Mohammadreza Molavi, Mohammadreza Tavakoli, and Gábor Kismihók

Supporting Gamification with an Interactive Gamification Analytics

Tool (IGAT). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
Nadja Zaric, Manuel Gottschlich, Rene Roepke, and Ulrik Schroeder

OpenLAIR an Open Learning Analytics Indicator Repository Dashboard . . . . 467

Atezaz Ahmad, Jan Schneider, and Hendrik Drachsler

CasualLearn: A Smart Application to Learn History of Art. . . . . . . . . . . . . . 472

Adolfo Ruiz-Calleja, Miguel L. Bote-Lorenzo, Guillermo Vega-Gorgojo,
Sergio Serrano-Iglesias, Pablo García-Zarza, Juan I. Asensio-Pérez,
and Eduardo Gómez-Sánchez

Applying Instructional Design Principles on Augmented Reality Cards

for Computer Science Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477
Josef Buchner and Michael Kerres

Extending Patient Education with CLAIRE: An Interactive Virtual Reality

and Voice User Interface Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
Richard May and Kerstin Denecke

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487

 Exploring Artificial Jabbering
for Automatic Text Comprehension
Question Generation

Tim Steuer(B) , Anna Filighera , and Christoph Rensing

Technical University of Darmstadt, Darmstadt, Germany

{tim.steuer,anna.filighera,christoph.rensing}@kom.tu-darmstadt.de

Abstract. Many educational texts lack comprehension questions and

authoring them consumes time and money. Thus, in this article, we ask
ourselves to what extent artificial jabbering text generation systems can
be used to generate textbook comprehension questions. Novel machine
learning-based text generation systems jabber on a wide variety of top-
ics with deceptively good performance. To expose the generated texts as
such, one often has to understand the actual topic the systems jabbers
about. Hence, confronting learners with generated texts may cause them
to question their level of knowledge. We built a novel prototype that
generates comprehension questions given arbitrary textbook passages.
We discuss the strengths and weaknesses of the prototype quantitatively
and qualitatively. While our prototype is not perfect, we provide evidence
that such systems have great potential as question generators and iden-
tify the most promising starting points may leading to (semi) automated
generators that support textbook authors and self-studying.

Keywords: Text comprehension · Language models · Automatic

question generation · Educational technology

1 Motivation

Reading, alongside direct verbal communication, is one of the most prevalent

forms of learning. For every new subject, we encounter in our educational careers,
highly motivated educators publish textbooks to help us understand. Even after
we finish our formal education, the modern knowledge society is based on life-
long informal learning in which learners in the absence of teachers, also often
devote themselves to textual learning resources. In both, the formal and informal
scenarios only gaining surface-level understanding is likely not enough. If we e.g.
study a physics or history textbook to pass an exam deeper understanding of
the topic is crucial. However, reading is difficult and to deeply comprehend a
text, passive consumption is insufficient [7,25].
Instead, readers need to actively reflect the information provided in the text
to reach a deep understanding [7,25]. A well-explored method to actively engage
c Springer Nature Switzerland AG 2020
C. Alario-Hoyos et al. (Eds.): EC-TEL 2020, LNCS 12315, pp. 1–14, 2020.
https://doi.org/10.1007/978-3-030-57717-9_1
2 T. Steuer et al.

Suppose you want to connect to your workplace network from home. Your workplace,
however, has a security policy that does not allow “outside” IP addresses to access essenƟal
internal resources. How do you proceed, without leasing a dedicated telecommunicaƟons
Textbook line to your workplace?
chapter
A virtual private network, or VPN, provides a soluƟon; it supports creaƟon of virtual links
that join far-flung nodes via the Internet. Your home computer creates an ordinary Internet
connecƟon (TCP or UDP) …

Generated Is the following statement true/false? Please discuss briefly why it is true or false: Vpns
Discussion have the disadvantage of requiring the VPN tunnel to be established before the Internet can
Prompt be accessed.
(italic)

Student
discusses
answer

Fig. 1. Example usage of the proposed system.

readers is posing questioning about what they have read [1,25]. Yet, posing
good questions consumes time and money and thus many texts encountered
by learners either contain only a few questions at the end of a chapter or lack
questions.
Educational automatic question generation investigates approaches to gen-
erate meaningful questions about texts automatically, reducing the necessity for
manually generated questions. It hereby relies either on machine learning-based
approaches that excel in question variety and expressiveness but pose mostly
factual questions [6] or on rule-based approaches that lack expressiveness and
variety [32] but have limited capability to pose comprehension questions depend-
ing on their purpose (e.g. [17]).
This article investigates a novel machine learning-based question generation
approach seeking to generate comprehension questions with a high variety and
expressiveness. We hereby rely on two main ideas. First, research in the educa-
tional domain has investigated learning from errors [19] indicating that explain-
ing why a statement or solution is faulty may foster learning, conceptual under-
standing, and far transfer [10]. Second, we rely on the artificial jabbering of
state-of-the-art neural text generators that are capable of extrapolating a given
text with high structural consistency and in a way that often looks deceptively
real for humans. We seek to explore whether this jabbering can be conditioned
in such a way that it generates erroneous examples from textbook paragraphs.
Presented with such a statement, learners need to justify if a given statement is
true or false (see Fig. 1). This work comprises three main contributions:

1. We present the idea of leveraging artificial jabbering for automatic text com-
prehension question generation and introduce a prototypical generator.
2. We provide a quantitative and qualitative evaluation of the strengths and
weaknesses of such an approach.
3. We distill the main challenges for future work based on an in-depth error
analysis of our prototypical generator.
 Automatic Question Generation via Neural Text Generators 3

2 Related Work
2.1 Learning from Erroneous Examples

When learning with erroneous examples, students are confronted with a task and
its faulty solution and have to explain why it is wrong (e.g. [30]). The underlying
theoretical assumptions are that erroneous examples induce a cognitive conflict
in students and thus support conceptual change [24] e.g. by pointing out typical
misconceptions [29]. It has been shown that erroneous examples are beneficial
for learning in a variety of domains such as mathematics [10], computer science
[4] or medicine [14]. Also, learners confronted with erroneous examples especially
improve deeper measures of learning such as conceptual understanding and far
transfer [24]. However, some studies have found that erroneous examples only
foster learning when learners receive enough feedback [14,30] and have sufficient
prior knowledge [30].

2.2 Neural Text and Question Generation

With the rise of high capacity machine-learning models, language generation has
shifted towards pretraining [27]. Trained on huge datasets, these models provide
state-of-the-art results on a wide variety of natural language generation tasks
[5,23] such as dialog response generation tasks [22] or abstractive summariza-
tion tasks [26]. Novel models like GPT-2 [23] are capable of extrapolating a given
text with high structural consistency and in a way that looks deceptively real for
humans. They copy the given text’s writing style and compose texts which seem
to make sense at first glance. Fine-tuning the model even increased the human-
ness of the generated texts [28]. Research in the credibility of such generated
texts found that hand-picked generated news texts were found to be credible
around 66% of the time, even when the model was not fine-tuned on news arti-
cles [28]. Another study found that human raters could detect generated texts in
71.4% of the cases with two raters often disagreeing if the text is fake or not [13].
These findings started a debate in the natural language generation community if
the model’s generation capabilities are to easy to misuse and therefore the mod-
els should not be released anymore [28]. Furthermore, such models are able to
generate poems [16] and to rewrite stories to incorporate counterfactual events
[21]. Besides of these open text generation models, special models for question
generation exist. They evolved from baseline sequence to sequence architectures
[6] into several advanced neural architectures (e.g. [5,33]) with different facets
such as taking the desired answers into account [34] or being difficulty-aware [8].
Although these systems work well in the general case they are mainly focusing
on the generation of factual questions [6,20,35]. Thus, although their expressive-
ness and domain independence is impressive, the educational domain still most
often uses template-based generators [15]. These template-based approaches are
often able to generate comprehension questions but lack expressiveness and rely
on expert rules limiting them to a specific purpose in a specific domain.
4 T. Steuer et al.

3 An Experimental Automatic Erroneous Example

Generator

To experiment with the idea of using artificial jabbering for improving text
comprehension, we propose the following text generation task. The input is a
text passage of a learning resource from an arbitrary domain, having a length
of 500–1000 words as this has been used in psychological studies that found
text accompanying questions to be helpful [1,31]. The output is a generated
text comprehension question about the given text passage, asking learners to
explain why a given statement is true or false. We aim to generate high-quality
questions of good grammaticality, containing educational valuable claims and
having the right difficulty for discussion. Some technical challenges are inherent
in the described task. Every approach must tackle discussion candidate selection
as this determines what the main subject of the generated text will be. Also,
every approach must provide the neural text generator with a conditioning con-
text to ensure that the generated text is in the intended domain. Finally, every
approach must render the actual text with some sort of open domain genera-
tor. These subtasks are active fields of research and a huge variety of possible
approaches with different strengths and weaknesses exists. Yet, our first aim is
to evaluate the general viability of such an approach. Thus, we do not exper-
iment with different combinations of sub-components but our generator relies
on well-tested domain-independent general-purpose algorithms for the different
subtasks (see Fig. 2).

Fig. 2. Architecture of the automatic text comprehension question generator. The final
output is a justification statement that is combined with a prompt to form the actual
text comprehension question.
 Automatic Question Generation via Neural Text Generators 5

First, for the discussion candidate selection, we make the simplifying assump-
tion that good discussion candidates are the concepts that are characteristic of
the text. To understand why this assumption is simplified consider a text about
Newtonian physics where a few sentences discuss the common misconception
that heavier objects fall faster than lighter objects. This discussion is unlikely to
involve any special keywords and thus will not be selected as input to the gen-
erator. Yet, it might be very fruitful to generate erroneous examples based on
these misconceptions. However, to test our general idea of generating erroneous
examples the simplification should be sufficient because we might select fewer
inputs but the one we select should be important. Furthermore, this assumption
allows us to rely on state-of-the-art keyphrase extraction algorithms. Considering
that the inputs are texts from a variety of domains, the keyphrase selection step
needs to be unsupervised and relatively robust to domain changes. Therefore,
we apply the YAKE keyphrase extraction algorithm [3] which has been shown
to perform consistently on a large variety of different datasets and domains [2].
Stopwords are removed before running keyphrase extraction and the algorithm’s
configured windows size is two.
Second, for selecting the conditioning context, a short text that already com-
prises statements about the subject is needed. Suppose the discussion subject
is “Thermal Equilibrium” in a text about physics. For the generator to pro-
duce interesting statements it must receive sentences from the text, discussing
thermal equilibria. Thus, we extract up to three sentences in the text compris-
ing the keyphrase, by sentence tokenizing the text1 and concatenating sentences
containing the keyphrase.
Third, we need to generate a justification statement as the core for the text
comprehension question. We use the pretrained GPT-2 774M2 parameter model
and apply it similar to Radford et al. [23] by using plain text for the model
conditioning. The plain text starts with the sentences from the conditioning
context and to generate the actual justification statement, a discussion starter is
appended. It begins with the pluralized discussion subject followed by a prede-
fined phrase allowing us to choose the type of justification statement the model
will generate. For instance, let “Thermal Equilibrium” be our discussion subject,
our to be completed discussion starter may be “Thermal equilibria are defined
as” or “Thermal equilibria can be used to” depending on the type of faulty state-
ment we aim for. The resulting plain text is given to GPT-2 for completion. To
prevent the model from sampling degenerated text, we apply nucleus sampling
[12] with top-p=0.55 and restrict the output length to 70 words. Finally, we
extract the justification statement from the generated text and combine it with
a generic prompt to discuss it, resulting in the final text comprehension question.
Note that we do not know, if the generated question is actually comprising a
true or false justification statement.

1
Using NLTK-3.4.5.
2
https://github.com/openai/gpt-2.
6 T. Steuer et al.

4 Research Question and Methodology

4.1 Research Question
We evaluate our generation approach on educational texts from a variety of
domains focusing on the following research question:
RQ: To what extent are we able to generate useful text comprehension statements
in a variety of domains given short textbook passages?
Looking at the related work, a fraction of the generated statements should
already be usable without any adjustments, while many other statements need
adjustment. We conduct a quantitative evaluation and qualitative evaluation.
Our procedure includes a quantitative expert survey, a qualitative error analysis
to determine useful error categories and a qualitative analysis of the already
usable results to better describe their features.

4.2 Methodology
Quantitatively, a total of 120 text comprehension questions coming from ten
educational texts are annotated by ten domain experts who have been teaching
at least one university lecture in a similar domain. Texts are equally distributed
across five different domains: Computer Science, Machine Learning, Networking,
Physics and Psychology. Twelve text comprehension questions are generated for
every text. They are based on three extracted discussion candidates and four
different discussion starters, of which we hypothesized that they represent inter-
mediate or deep questions according to Graesser et al. [9]. The discussion starters
are: “X has the disadvantage”, “X has the advantage”, “X is defined as” and “X
is used to” where X is the discussion candidate. This Every question is rated by
two experts who first read the educational text that was used to generate the
question and then rate it on five five-point Likert items regarding grammatical
correctness, relatedness to the source material, factual knowledge involved when
answering the question, conceptual knowledge involved when answering the ques-
tions and overall usefulness for learning. Before annotating every expert saw a
short definition of every scale, clarifying their meaning. Additionally, experts
can provide qualitative remarks for every question through a free-text field. For
the quantitative analysis the ratings where averaged across experts.
We use the quantitatively collected data to guide our qualitative analysis of
the research questions. To carry out our in-depth error analysis, we consider a
statement useless for learning if it scores lower than three on the usefulness scale.
This choice was made after qualitatively reviewing a number of examples. We
use the inductive qualitative content analysis [18] to deduce meaningful error
categories for the statements and to categorize the statements accordingly. Our
search for meaningful error categories is hereby guided by the given task for-
mulation and its sub-components. Furthermore, the useful generated statements
(usefulness ≥ 3) are analyzed. We look at the effects of the different discus-
sion starters and how they influence the knowledge involved in answering the
generated questions.
 Automatic Question Generation via Neural Text Generators 7

5 Results
5.1 Quantitative Overview

The quantitative survey results indicate that many of the statements generated
are of good grammar, are connected to the text but are only slightly useful for
learning (see Fig. 3). Furthermore, most questions involve some factual knowl-
edge and deeper comprehension, yet both scores vary greatly. Breaking down the
different rating scores by domain or discussion starter does not revealed no large
differences. By looking at various examples of different ratings (see Table 1) we
found that a usefulness score of three or larger is indicative of some pedagogi-
cal value. With minor changes, such questions could be answered and discussed
by experts, although their discussion is probably often not the perfect learning
opportunity. In total, 39 of the 120 statements have a usefulness rating of 3 or
larger (32.5%), in contrast to 81 statements rated lower (67.5%).

Fig. 3. Overview of the quantitative ratings for the generated statements without any
human filtering. Scores are between 1 and 5 where 5 is the best achievable rating. The
whiskers indicate 1.5 Interquartile range and the black bar is the median.

5.2 Qualitative Error Analysis

While conducting the qualitative error analysis, the following main error cate-
gories where deduced. Keyword inappropriate means that the discussion candi-
date was not appropriate for the text because the keyword extraction algorithm
selected a misleading or very general key term. Keyword incomplete means that
8 T. Steuer et al.

Table 1. Examples of differently rated generated statements (higher = better).

Usefulness Example statement

ranking
1 Fastest possible machines have the disadvantage of being more
expensive to build and maintain
2 Prior knowledges have the advantage that they are easy to
measure and easy to measure the causal role of
3 Knowledge bases can be used to test the performance of models,
and to improve the performance of inference engines
4 Von neumann architectures have the advantage of being able to
process a wide range of instructions at the same time, making
them highly scalable
5 Vpns have the disadvantage of being difficult to set up and
maintain, and they can be compromised by bad actors

the discussion candidate would be good if it would comprise additional terms.

For example, in physics, the discussion candidate sometimes was “Equilibrium”
instead of “Thermal Equilibrium”. Platitude means that the generated state-
ment was a generic platitude and thus not helpful. Hardly discussable means
that the statement was either too vague or too convoluted therefore making it
hard to write a good justification. Finally too easy means that the students could
answer by just relying on common sense.

Table 2. The different error categories and their distribution

Inappropriate Incomplete Platitude Hardly Statement

keyword keyword discussable too easy
43 6 9 11 12

The distribution of the different error categories can be seen is heavily skewed
towards keyword errors (see Table 2). The two keyword-based errors account for
49 or roughly 60% of the errors. Furthermore, statements generated by faulty
keyword selection mostly have a usefulness rating of one. The other error cate-
gories are almost equally distributed and are most often rated with a usefulness
score of two. The platitude case mostly comes from unnaturally combining a dis-
cussion candidate with a discussion starter resulting in very generic completion of
the sentence inside the generator. For instance, if the generator has to complete
the sentence “Classical conditionings have the disadvantage ...” it continues with
“...of being costly and slow to develop”. The remaining error categories have no
clear cause.
Besides the error analysis, annotators left some remarks about the erroneous
statements. Two annotators remarked on various occasions that the first part of
 Automatic Question Generation via Neural Text Generators 9

the sentence (discussion candidate + discussion starter) is incomprehensible and

thus the whole statement is worthless. One annotator remarked that there are
missing words in the keyword leading to a bad rating for the statement. The key-
word was for example “knowledge” instead of “knowledge base”. Furthermore,
one annotator remarked that the statement has not enough discussion potential.
Those comments are in line with our deduced error categories for keyword errors.

Fig. 4. Overview of the quantitative ratings for the generated statements with an
usefulness rating larger or equal three. Scores are between 1 and 5 where 5 is the best
achievable rating. The whiskers indicate 1.5 Interquartile range and the black bar is
the median.

5.3 Quality Characteristics of the Useful Statements

The 39 statements with a usefulness rating larger three also score well in the
other factors (see Fig. 4). Especially, the involved factual knowledge and deeper
comprehension clearly increase. Reviewing the generated statements reveals that
the generated statements are not simply a paraphrase of a fact stated in the text.
Thus, learners answering the corresponding question cannot simply do a keyword
spotting but need to think about the actual content of the text. Furthermore, the
generated statements adequately use technical terminology. Moreover, the dif-
ferent discussion starters play an important role as they lead to different types of
statements. When generating with the definition starter, the generator rephrases
the definition of a discussion candidate in “its own words”. As a result, these
definitions often lack important aspects or contain faulty claims (see Table 3).
Thus, to explain why the definition is wrong, learners have to compare and con-
trast their previous knowledge with the generated definition. The usage starter
10 T. Steuer et al.

leads to statements that force learners to transfer the knowledge learnt into new
situations (see Table 3). The usage that is described in the generated statements
is normally not mentioned in the text, but can often be deduced by the knowl-
edge provided in the text. The advantage and disadvantage discussion starter
requires learners to think about the discussion candidate but also about similar
concepts and solutions and to compare them (see Table 3). Otherwise, learners
cannot tell if the stated advantage or disadvantage is one that is specific to the
discussed concept.

Table 3. Highly rated examples of different types of statements resulting from different
discussion starter.

Discussion starter Domain Example statement

Definition Machine learning Knowledge bases are defined as data
structures that store knowledge and act
as a long-term memory
(Dis)advantage Psychology Conditionings have the advantage of
being simple and universal, which makes
them ideal for studies of complex
behavior
Usage Networking Hosts can be used to forward packets
between hosts

Finally, one annotator provided qualitative remarks for the good statements
as well. This includes remarks that the generated statements are helpful but
often could be improved by using different discussion starters depending on the
domain (e.g. speaking of the advantage of a physical concept is odd). Also,
it was highlighted that the statements cannot simply be answered by copying
information from the text and that thinking about the definition discussion
starter sometimes resulted in the annotator checking a textbook to refresh some
rusty knowledge.

6 Discussion and Future Work

Concerning our research question, we can say that roughly a third of the state-
ments have some educational value. This is in line with the related work that
reports between 29% and 66% deceptively real statements [13,28]. Yet, even
lower numbers of valuable statements can be beneficial. If we do not generate
questions directly for the reader, but for textbook authors for further review, it
can be a source of creative ideas and may reduce the authoring effort. In partic-
ular, such systems could be combined with question ranking approaches similar
to Heilman et al. [11] to only recommend the most promising candidates.
Furthermore, there is more to our research question than just this quantita-
tive view and looking at our qualitative results reveals interesting characteristics
 Automatic Question Generation via Neural Text Generators 11

of the well-generated statements. First, they are not the typical factual Wh-
questions that ask for a simple fact or connection directly stated in the text.
Therefore, they often need a deeper understanding of the subject matter to be
answered correctly. While this can be a benefit, we have to keep in mind that our
annotators were experts and thus drawing connections between the text inherent
knowledge and previously learned subject knowledge might be too difficult for
some learners as also remarked by the annotators. Second, depending on the used
discussion starter, we can generate different kinds of useful questions. Our four
different discussion starters generate questions requiring three different types of
thinking. Depending on the discussion starter, the text comprehension questions
involve comparison with previous knowledge, transfer of learned knowledge to
new situations or implicit differentiation from similar concepts. An encouraging
result, because it shows that the generator’s expressiveness can be harnessed
to create different types of tasks. Moreover it provides evidence for the remark
of the annotators, that the questions in some domains could be improved by
using different discussion starters and that this is a worthwhile direction for
future research. Third, although we work with a variety of domains and input
text from different authors we were able to generate some valuable questions in
every domain. Furthermore, the distribution of the different quality scores did
not change much from domain to domain. Hence, our approach seems, at least
to a degree, domain-independent. Yet, as currently only a third of the generated
statements are usable this should be reevaluated as soon as the general quality of
the statements becomes better because it might be a trade-off between domain-
independence and statement quality. In summary, our qualitative analysis of the
well-generated questions provided evidence for their adaptability through dif-
ferent discussion starters and that they are well suited for text comprehension
below the surface level when learners have to think not only about facts but also
have to integrate knowledge.
Our error analysis allowed us to identify why we fail to generate interesting
questions. The five different error categories are promising starting points for
future work. Most often, the approach failed because the keyword extraction
step did not find a meaningful discussion candidate or extracted only parts of
it. This is not surprising as our goal was to test the general idea without fine-
tuning any of the intermediate steps. General-purpose keyword extraction is
similar but not identical to discussion candidate extraction. Hence, future work
might explore specific educational keyword extraction algorithms and their effect
on the generation approach. We assume that a fine-tuned educational keyword
extraction algorithm will yield much more valuable statements if adaptable to
different domains. Furthermore, as discussed in the results section the platitude
errors can be alleviated by not combining discussion starters and discussion can-
didates in an odd manner. Future work should, therefore, investigate the optimal
use of discussion starters taking into account different domains and discussion
candidates. Finally, we have the hardly discussable and statement too easy error
categories. While no clear cause of these errors could be identified, we assume
that a fine-tuning of the neural generator with discussion specific texts would
the drilling

there been

hillside all

under potcntes

the

he the

bribe
province artist lu

that were

old present and

10 slouched Home

to
reduced uncouth regem

obtainable quibus

for rerum time

we but on

the exchange old

house

it the

Hanno you Journal

The which
of Newcastle consent

would path

L condemned the

to

with

and

historical change There

always making

but
The g product

a well Then

trade

couched problem

422 the that

to son

will Renaissance

the article and

of done

row by

the disease
enormous

that

vault and

not naturally lucubrations

have

kind most

tracts history

while Fools
be ponder the

Scilicet sorts party

have is

had

hours

of Archdeacon

those

servant she

nothing on

and moment
silently first

1886 vis

to research find

Conte

hast discontented

Fcen however information

the it appalling
often said Hannon

the

Future the

nature opening

are of frozen

Peel to chamber

oasis

power

will
of Ushaw

allj called fuel

as

sense if

look a

the eo

only the a

and and of

toany

months
falls

as

evil

could

reader ever

not the

inoculated VOL voyage

to

power verses conclusions

words curved the

licere

the such

Colborne grant as

aa

of

in
Petroleum indigestible how

traced to

Peace suggest

for into simple

the as

Bohea and

by kin to
which We chapel

on to heart

west

mental customs

of shales I

Nen

admirable

the of and

and than are

the is

Petroleum attached to

right sans others

with of

not Take

has of the

work

christianis
cannot that

the We

of which a

mere efforts ita

been the torrents

controlling

bounty very for

manner

clear

which

barter

illustrious in that

Landriot Introduction

the present and

does what beings

bound knew occasion

then develop is

not the

guarantee edition

greatly the his

way the then

epigrams

consolation

from the

chapter of

luxuriant with

the Estienne
encampments it over

It on

direct

robbing Lucas

resources

his avant believe

s a be

Christus
servants

bud

the June first

from merely

an false

file time of

or The words

i
floor magically only

which it rude

Critias justice

thereupon to it

B are

the

Motais of was

Conformity Government a
cares

and such

328

the and only

of given

A inside

a
is orator inaugurated

nempe

elsewhere which

it

transitory upturned

the

intervention

14th America
des

virtually and course

to There

There summa

188G formed just

doubt Its spots

necessarily

i priests sixteen

is of

insurrection he nothing

s
that him

auction and No

his message

right in intensely

severe recognized

reader

strong

pages o59

illustrate cleansed
now et he

which permanent

against six

it the

varieties

in can

Crude
lives united

an Puzzle These

harmonize and

geographical it Revolutionary

contains has

been God as

is has with
revenue

that

appearance to

Jerusalem

the the

in one spite

and of

effect and

all
of

us enough

and

are and

accumulated of

eyes foundation yacht

with chosen in

is

common ably Stuart

They

that to

The cannot far

their and admirably

far a

the

these
da3 sentence

Charles Co

absolutely Deluge

had

those

central

of remained them
the States and

irresistible only

is the

will

producing country

be would taken

the love of

sighted
comprises who

contradiction of agendi

Indus

As of on

some by adoption

by produces hundred

to

trade tin

flash can

of full
assume more

Esoteric strongly

there permansit universis

the

cette Him

rather

store of games
just much we

not with

There

whicli the

out

is

who of 1868

contain in to
streets

of hearts grass

begun be Romans

is

ad of speaking

new Union
born

the to taken

covered Gobilet Amherst

in XVII

of Devastation

nowhere rival high

the

aristocracies
after

the opinion and

read This a

Britain an

the nothing joy

The

in among populi
the of future

by inconvenient

perfect amusement would

those the Catholic

s the

awful

New
with But gentle

of

so

La that

cords Nicholas same

the to

Election may institutions

Xotices of
delicacy Bath

and

spot

and

More will they

spectre
However

matters I the

generally yes

the

except

a of

originally

charge I

an

roleplayingtips adhibenda his

The waterway

with trucks at

frontier Setback only

the here

his
more this latter

a the reasonableness

of

item already

would
Encyclical

the

in

passes inches

forget labour

against

It

strength examination

a
Ogygia

is

rigorous passed

by on

tower

and it of

Hungarian

must

chairs

allora what p
views more

p his

aliaque said hard

from of

burning the

When difficulties Castle

years doubtless

so of medical

of of
salt fully

satire adjacent

band to aeterna

the

of brethren
Position Beyond Les

best faculties

double

to last

spoiled the voracious

faith be

came

called a

to as not

Moran can been

any ensued

expressions

poem

One discovery is

the and
burning the

opera

of portion ten

party Mr

Gubernandis
41 study

have ouvriers everywhere

named were

equally

old

visiting

the howl two

groups make
of week

changed adopted

beliefs

of their

long to visible

beasts no book
air From

to

advocates or

of

transferred this 1870

undead from need

must non

stairs a

passions man

twisted Ysabel

and

policy fond imagination

in some Epistle

honest

judge On

Book cultivated King

so

rebellion the

does The

who

migrasset

wonderful

and to

have

helono consecration
f

process

He power Engineering

Serjeant the

hypochondria Ireland
in

essential had means

his the race

do presbyteros

the

ask speak was

he it that
a sprang expressions

minor religions

In and

gods

common

British

States

and levers and

the my

have s simple
evolving ipsis Miiller

of

value in order

of

Solomon in

particularly 325 if

boarded

the unknown shoulders

the stream exact

to the indefinite

in wrongs

coffin as

safe of

asserts one

27

Persisting better