{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T07:44:45Z","timestamp":1768981485414,"version":"3.49.0"},"reference-count":73,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T00:00:00Z","timestamp":1692144000000},"content-version":"vor","delay-in-days":227,"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,8,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Disagreement in natural language annotation has mostly been studied from a perspective of biases introduced by the annotators and the annotation frameworks. Here, we propose to analyze another source of bias\u2014task design bias, which has a particularly strong impact on crowdsourced linguistic annotations where natural language is used to elicit the interpretation of lay annotators. For this purpose we look at implicit discourse relation annotation, a task that has repeatedly been shown to be difficult due to the relations\u2019 ambiguity. We compare the annotations of 1,200 discourse relations obtained using two distinct annotation tasks and quantify the biases of both methods across four different domains. Both methods are natural language annotation tasks designed for crowdsourcing. We show that the task design can push annotators towards certain relations and that some discourse relation senses can be better elicited with one or the other annotation approach. We also conclude that this type of bias should be taken into account when training and testing models.<\/jats:p>","DOI":"10.1162\/tacl_a_00586","type":"journal-article","created":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T19:47:11Z","timestamp":1692215231000},"page":"1014-1032","update-policy":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":1,"title":["Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design"],"prefix":"10.1162","volume":"11","author":[{"given":"Valentina","family":"Pyatkin","sequence":"first","affiliation":[{"name":"Bar Ilan University, Ramat Gan, Israel. pyatkiv@biu.ac.il"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Frances","family":"Yung","sequence":"additional","affiliation":[{"name":"Saarland University, Saarbr\u00fccken, Germany. frances"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Merel C. J.","family":"Scholman","sequence":"additional","affiliation":[{"name":"Saarland University, Saarbr\u00fccken, Germany. m.c.j.scholman@coli.uni-saarland.de"},{"name":"Utrecht University, Utrecht, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Reut","family":"Tsarfaty","sequence":"additional","affiliation":[{"name":"Bar Ilan University, Ramat Gan, Israel. reut.tsarfaty@biu.ac.il"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ido","family":"Dagan","sequence":"additional","affiliation":[{"name":"Bar Ilan University, Ramat Gan, Israel. dagan@cs.biu.ac.il"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vera","family":"Demberg","sequence":"additional","affiliation":[{"name":"Saarland University, Saarbr\u00fccken, Germany. vera@coli.uni-saarland.de"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,8,15]]},"reference":[{"key":"2023081619464958400_bib1","doi-asserted-by":"publisher","first-page":"810","DOI":"10.18653\/v1\/2021.eacl-main.68","article-title":"Ellipsis resolution as question answering: An evaluation","volume-title":"16th Conference of the European Chapter of the Association for Computational Linguistics (EACL)","author":"Aralikatte","year":"2021"},{"issue":"2013","key":"2023081619464958400_bib2","article-title":"Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard","volume":"2013","author":"Aroyo","year":"2013","journal-title":"WebSci2013 ACM"},{"issue":"4","key":"2023081619464958400_bib3","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1162\/coli.07-034-R2","article-title":"Inter-coder agreement for computational linguistics","volume":"34","author":"Artstein","year":"2008","journal-title":"Computational Linguistics"},{"key":"2023081619464958400_bib4","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-011-1715-9","volume-title":"Reference to Abstract Objects in Discourse","author":"Asher","year":"1993"},{"key":"2023081619464958400_bib5","doi-asserted-by":"publisher","first-page":"15","DOI":"10.18653\/v1\/2021.bppf-1.3","article-title":"We need to consider disagreement in evaluation","volume-title":"Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future","author":"Basile","year":"2021"},{"key":"2023081619464958400_bib6","doi-asserted-by":"publisher","first-page":"632","DOI":"10.18653\/v1\/D15-1075","article-title":"A large annotated corpus for learning natural language inference","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Bowman","year":"2015"},{"key":"2023081619464958400_bib7","doi-asserted-by":"publisher","first-page":"4843","DOI":"10.18653\/v1\/2021.naacl-main.385","article-title":"What will it take to fix benchmarking in natural language understanding?","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Bowman","year":"2021"},{"key":"2023081619464958400_bib8","doi-asserted-by":"publisher","first-page":"578","DOI":"10.18653\/v1\/E17-2092","article-title":"Emobank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis","volume-title":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers","author":"Buechel","year":"2017"},{"key":"2023081619464958400_bib9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/W17-0801","article-title":"Readers vs. writers vs. texts: Coping with different perspectives of text understanding in emotion annotation","volume-title":"Proceedings of the 11th Linguistic Annotation Workshop","author":"Buechel","year":"2017"},{"key":"2023081619464958400_bib10","first-page":"1","article-title":"Discourse tagging reference manual","volume":"54","author":"Carlson","year":"2001","journal-title":"ISI Technical Report ISI-TR-545"},{"key":"2023081619464958400_bib11","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v3i1.13266","article-title":"Linguistic wisdom from the crowd","volume-title":"Third AAAI Conference on Human Computation and Crowdsourcing","author":"Chang","year":"2016"},{"key":"2023081619464958400_bib12","doi-asserted-by":"publisher","first-page":"8772","DOI":"10.18653\/v1\/2020.acl-main.774","article-title":"Uncertain natural language inference","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Chen","year":"2020"},{"issue":"CSCW","key":"2023081619464958400_bib13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3359164","article-title":"Efficient elicitation approaches to estimate collective crowd answers","volume":"3","author":"Young Chung","year":"2019","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"issue":"1","key":"2023081619464958400_bib14","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educational and Psychological Measurement"},{"issue":"1","key":"2023081619464958400_bib15","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1177\/1529100619850176","article-title":"Mapping the passions: Toward a high-dimensional taxonomy of emotional experience and expression","volume":"20","author":"Cowen","year":"2019","journal-title":"Psychological Science in the Public Interest"},{"issue":"2","key":"2023081619464958400_bib16","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1162\/COLI_a_00097","article-title":"Did it happen? The pragmatic complexity of veridicality assessment","volume":"38","author":"De Marneffe","year":"2012","journal-title":"Computational Linguistics"},{"issue":"1","key":"2023081619464958400_bib17","doi-asserted-by":"publisher","first-page":"87","DOI":"10.5087\/dad.2019.104","article-title":"How compatible are our discourse annotation frameworks? Insights from mapping RST-DT and PDTB annotations","volume":"10","author":"Demberg","year":"2019","journal-title":"Dialogue & Discourse"},{"key":"2023081619464958400_bib18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3173574.3173986","article-title":"Addressing age-related bias in sentiment analysis","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems","author":"D\u00edaz","year":"2018"},{"key":"2023081619464958400_bib19","doi-asserted-by":"publisher","first-page":"701","DOI":"10.1007\/978-3-319-18818-8_43","article-title":"Crowdsourcing disagreement for collecting semantic annotation","volume-title":"European Semantic Web Conference","author":"Dumitrache","year":"2015"},{"key":"2023081619464958400_bib20","first-page":"11","article-title":"CrowdTruth 2.0: Quality metrics for crowdsourcing with disagreement","volume-title":"1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing, and Short Paper 1st Workshop on Disentangling the Relation Between Crowdsourcing and Bias Management, SAD+ CrowdBias 2018","author":"Dumitrache","year":"2018"},{"issue":"3","key":"2023081619464958400_bib21","doi-asserted-by":"publisher","first-page":"403","DOI":"10.3233\/SW-200415","article-title":"Empirical methodology for crowdsourcing ground truth","volume":"12","author":"Dumitrache","year":"2021","journal-title":"Semantic Web"},{"key":"2023081619464958400_bib22","doi-asserted-by":"publisher","first-page":"764","DOI":"10.1162\/tacl_a_00488","article-title":"Text-based np enrichment","volume":"10","author":"Elazar","year":"2022","journal-title":"Transactions of the Association for Com putational Linguistics"},{"key":"2023081619464958400_bib23","first-page":"440","article-title":"Graded word sense assignment","volume-title":"Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing","author":"Erk","year":"2009"},{"key":"2023081619464958400_bib24","doi-asserted-by":"publisher","first-page":"1626","DOI":"10.18653\/v1\/2021.naacl-main.129","article-title":"Did they answer? Subjective acts and intents in conversational discourse","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Ferracane","year":"2021"},{"key":"2023081619464958400_bib25","doi-asserted-by":"publisher","first-page":"2051","DOI":"10.18653\/v1\/P18-1191","article-title":"Large-scale qa-srl parsing","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Fitzgerald","year":"2018"},{"key":"2023081619464958400_bib26","article-title":"Deberta: Decoding-enhanced bert with disentangled attention","volume-title":"International Conference on Learning Representations","author":"He","year":"2020"},{"key":"2023081619464958400_bib27","doi-asserted-by":"publisher","first-page":"1428","DOI":"10.18653\/v1\/2020.acl-main.132","article-title":"Bridging anaphora resolution as question answering","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Hou","year":"2020"},{"key":"2023081619464958400_bib28","first-page":"1120","article-title":"Learning whom to trust with MACE","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)","author":"Hovy","year":"2013"},{"key":"2023081619464958400_bib29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3290605.3300637","article-title":"Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments","volume-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","author":"Hube","year":"2019"},{"key":"2023081619464958400_bib30","first-page":"44","article-title":"The sensitivity of annotator bias to task definitions in argument mining","volume-title":"Proceedings of the 16th Lingusitic Annotation Workshop (LAW-XVI) within LREC2022","author":"Thorn Jakobsen","year":"2022"},{"key":"2023081619464958400_bib31","doi-asserted-by":"publisher","first-page":"1357","DOI":"10.1162\/tacl_a_00523","article-title":"Investigating reasons for disagreement in natural language inference","volume":"10","author":"Jiang","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023081619464958400_bib32","doi-asserted-by":"publisher","first-page":"103","DOI":"10.18653\/v1\/P17-2017","article-title":"Understanding task design trade-offs in crowdsourced paraphrase collection","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Jiang","year":"2017"},{"key":"2023081619464958400_bib33","first-page":"556","article-title":"Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Jurgens","year":"2013"},{"key":"2023081619464958400_bib34","first-page":"269","article-title":"Rapid development of a corpus with discourse annotations using two-stage crowdsourcing","volume-title":"Proceedings of the International Conference on Computational Linguistics (COLING)","author":"Kawahara","year":"2014"},{"key":"2023081619464958400_bib35","article-title":"Improving crowdsourcing-based annotation of Japanese discourse relations","volume-title":"LREC","author":"Kishimoto","year":"2018"},{"key":"2023081619464958400_bib36","article-title":"Discourse comprehension: A question answering framework to represent sentence connections","author":"Ko","year":"2021","journal-title":"arXiv preprint arXiv:2111.00701"},{"key":"2023081619464958400_bib37","first-page":"79","article-title":"Europarl: A parallel corpus for statistical machine translation","volume-title":"Proceedings of MT Summit X","author":"Koehn","year":"2005"},{"key":"2023081619464958400_bib38","doi-asserted-by":"publisher","first-page":"3296","DOI":"10.18653\/v1\/2020.findings-emnlp.296","article-title":"Detecting stance in media on global warming","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Luo","year":"2020"},{"issue":"3","key":"2023081619464958400_bib39","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1515\/text.1.1988.8.3.243","article-title":"Rhetorical Structure Theory: Toward a functional theory of text organization","volume":"8","author":"Mann","year":"1988","journal-title":"Text-Interdisciplinary Journal for the Study of Discourse"},{"key":"2023081619464958400_bib40","article-title":"Local textual inference: It\u2019s hard to circumscribe, but you know it when you see it\u2014and NLP needs it","author":"Manning","year":"2006"},{"key":"2023081619464958400_bib41","first-page":"3659","article-title":"Establishing annotation quality in multi-label annotations","volume-title":"Proceedings of the 29th International Conference on Computational Linguistics","author":"Marchal","year":"2022"},{"key":"2023081619464958400_bib42","doi-asserted-by":"publisher","first-page":"5783","DOI":"10.18653\/v1\/2020.emnlp-main.466","article-title":"Ambigqa: Answering ambiguous open-domain questions","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Min","year":"2020"},{"key":"2023081619464958400_bib43","doi-asserted-by":"publisher","first-page":"9131","DOI":"10.18653\/v1\/2020.emnlp-main.734","article-title":"What can we learn from collective human opinions on natural language inference data?","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Nie","year":"2020"},{"key":"2023081619464958400_bib44","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1162\/tacl_a_00185","article-title":"The benefits of a model of annotation","volume":"2","author":"Passonneau","year":"2014","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023081619464958400_bib45","doi-asserted-by":"publisher","first-page":"677","DOI":"10.1162\/tacl_a_00293","article-title":"Inherent disagreements in human textual inferences","volume":"7","author":"Pavlick","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023081619464958400_bib46","doi-asserted-by":"publisher","first-page":"9617","DOI":"10.1109\/ICCV.2019.00971","article-title":"Human uncertainty makes classification more robust","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Peterson","year":"2019"},{"key":"2023081619464958400_bib47","doi-asserted-by":"publisher","first-page":"507","DOI":"10.3115\/v1\/P14-2083","article-title":"Linguistically debatable or just plain wrong?","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Plank","year":"2014"},{"key":"2023081619464958400_bib48","doi-asserted-by":"publisher","first-page":"76","DOI":"10.3115\/1608829.1608840","article-title":"The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account","volume-title":"Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky","author":"Poesio","year":"2005"},{"issue":"2","key":"2023081619464958400_bib49","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1207\/s15326950dp4202_4","article-title":"Underspecification and anaphora: Theoretical issues and preliminary evidence","volume":"42","author":"Poesio","year":"2006","journal-title":"Discourse processes"},{"key":"2023081619464958400_bib50","doi-asserted-by":"publisher","first-page":"133","DOI":"10.18653\/v1\/2021.law-1.14","article-title":"On releasing annotator-level labels and information in datasets","volume-title":"Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop","author":"Prabhakaran","year":"2021"},{"key":"2023081619464958400_bib51","doi-asserted-by":"publisher","first-page":"2804","DOI":"10.18653\/v1\/2020.emnlp-main.224","article-title":"QADiscourse-Discourse Relations as QA Pairs: Representation, crowdsourcing and baselines","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Pyatkin","year":"2020"},{"key":"2023081619464958400_bib52","doi-asserted-by":"publisher","first-page":"784","DOI":"10.18653\/v1\/P18-2124","article-title":"Know what you don\u2019t know: Unanswerable questions for squad","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Rajpurkar","year":"2018"},{"key":"2023081619464958400_bib53","first-page":"1039","article-title":"Annotating discourse relations in spoken language: A comparison of the PDTB and CCR frameworks","volume-title":"Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916)","author":"Rehbein","year":"2016"},{"issue":"1","key":"2023081619464958400_bib54","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1162\/COLI_a_00182","article-title":"On the problem of theoretical terms in empirical computational linguistics","volume":"40","author":"Riezler","year":"2014","journal-title":"Computational Linguistics"},{"key":"2023081619464958400_bib55","doi-asserted-by":"publisher","first-page":"49","DOI":"10.18653\/v1\/W16-1707","article-title":"Filling in the blanks in understanding discourse adverbials: Consistency, conflict, and context-dependence in a crowdsourced elicitation task","volume-title":"Proceedings of the 10th Linguistic Annotation Workshop (LAW X)","author":"Rohde","year":"2016"},{"issue":"1","key":"2023081619464958400_bib56","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/01638539209544800","article-title":"Toward a taxonomy of coherence relations","volume":"15","author":"Sanders","year":"1992","journal-title":"Discourse Processes"},{"key":"2023081619464958400_bib57","doi-asserted-by":"publisher","first-page":"24","DOI":"10.18653\/v1\/W17-0803","article-title":"Crowdsourcing discourse interpretations: On the influence of context and the reliability of a connective insertion task","volume-title":"Proceedings of the 11th Linguistic Annotation Workshop (LAW)","author":"Scholman","year":"2017"},{"key":"2023081619464958400_bib58","article-title":"Discogem: A crowdsourced corpus of genre-mixed implicit discourse relations","volume-title":"Proceedings of the Thirteenth International Conference on Language Resources and Evaluation (LREC\u201922)","author":"Scholman","year":"2022"},{"key":"2023081619464958400_bib59","article-title":"Design choices in crowdsourcing discourse relation annotations: The effect of worker selection and training","volume-title":"Proceedings of the Thirteenth International Conference on Language Resources and Evaluation (LREC\u201922)","author":"Scholman","year":"2022"},{"key":"2023081619464958400_bib60","doi-asserted-by":"publisher","first-page":"188","DOI":"10.18653\/v1\/W19-0416","article-title":"Learning to explicitate connectives with Seq2Seq network for implicit discourse relation classification","volume-title":"Proceedings of the 13th International Conference on Computational Semantics - Long Papers","author":"Shi","year":"2019"},{"key":"2023081619464958400_bib61","doi-asserted-by":"crossref","first-page":"254","DOI":"10.3115\/1613715.1613751","article-title":"Cheap and fast\u2014but is it good? Evaluating non-expert annotations for natural language tasks","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Snow","year":"2008"},{"issue":"2","key":"2023081619464958400_bib62","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1515\/cllt.2010.009","article-title":"Coding coherence relations: Reliability and validity","volume":"6","author":"Spooren","year":"2010","journal-title":"Corpus Linguistics and Linguistic Theory"},{"key":"2023081619464958400_bib63","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1609\/hcomp.v8i1.7478","article-title":"A case for soft loss functions","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","author":"Uma","year":"2020"},{"key":"2023081619464958400_bib64","doi-asserted-by":"publisher","first-page":"1385","DOI":"10.1613\/jair.1.12752","article-title":"Learning from disagreement: A survey","volume":"72","author":"Uma","year":"2021","journal-title":"Journal of Artificial Intelligence Research"},{"key":"2023081619464958400_bib65","doi-asserted-by":"publisher","first-page":"138","DOI":"10.18653\/v1\/W16-5618","article-title":"Are you a racist or am I seeing things? Annotator influence on hate speech detection on Twitter","volume-title":"Proceedings of the First Workshop on NLP and Computational Social Science","author":"Waseem","year":"2016"},{"key":"2023081619464958400_bib66","doi-asserted-by":"publisher","first-page":"674","DOI":"10.3115\/1690219.1690240","article-title":"Genre distinctions for discourse in the Penn TreeBank","volume-title":"Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP","author":"Webber","year":"2009"},{"key":"2023081619464958400_bib67","article-title":"The Penn Discourse Treebank 3.0 annotation manual","author":"Webber","year":"2019"},{"key":"2023081619464958400_bib68","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2023081619464958400_bib69","doi-asserted-by":"publisher","first-page":"16","DOI":"10.18653\/v1\/W19-4003","article-title":"Crowdsourcing discourse relation annotations by a two-step connective insertion task","volume-title":"Proceedings of the 13th Linguistic Annotation Workshop","author":"Yung","year":"2019"},{"key":"2023081619464958400_bib70","first-page":"1","article-title":"Ted multilingual discourse bank (TED-MDB): A parallel corpus annotated in the PDTB style","author":"Zeyrek","year":"2019","journal-title":"Language Resources and Evaluation"},{"issue":"2","key":"2023081619464958400_bib71","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1007\/s10579-019-09445-9","article-title":"Ted multilingual discourse bank (TED-MDB): A parallel corpus annotated in the PDTB style","volume":"54","author":"Zeyrek","year":"2020","journal-title":"Language Resources and Evaluation"},{"key":"2023081619464958400_bib72","doi-asserted-by":"publisher","first-page":"7620","DOI":"10.18653\/v1\/2021.emnlp-main.601","article-title":"Learning with different amounts of annotation: From zero to many labels","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Zhang","year":"2021"},{"key":"2023081619464958400_bib73","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1007\/978-3-030-27947-9_20","article-title":"Explicit and implicit discourse relations in the Prague Discourse Treebank","volume-title":"Text, Speech, and Dialogue: 22nd International Conference, TSD 2019, Ljubljana, Slovenia, September 11\u201313, 2019, Proceedings 22","author":"Zik\u00e1nov\u00e1","year":"2019"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00586\/2154451\/tacl_a_00586.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00586\/2154451\/tacl_a_00586.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T19:47:24Z","timestamp":1692215244000},"score":1,"resource":{"primary":{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00586\/117215\/Design-Choices-for-Crowdsourcing-Implicit"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":73,"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1162\/tacl_a_00586","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}