{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:52:15Z","timestamp":1760241135179,"version":"build-2065373602"},"reference-count":27,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2019,12,6]],"date-time":"2019-12-06T00:00:00Z","timestamp":1575590400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The entropy rate h of a natural language quantifies the complexity underlying the language. While recent studies have used computational approaches to estimate this rate, their results rely fundamentally on the performance of the language model used for prediction. On the other hand, in 1951, Shannon conducted a cognitive experiment to estimate the rate without the use of any such artifact. Shannon\u2019s experiment, however, used only one subject, bringing into question the statistical validity of his value of     h = 1.3     bits per character for the English language entropy rate. In this study, we conducted Shannon\u2019s experiment on a much larger scale to reevaluate the entropy rate h via Amazon\u2019s Mechanical Turk, a crowd-sourcing service. The online subjects recruited through Mechanical Turk were each asked to guess the succeeding character after being given the preceding characters until obtaining the correct answer. We collected 172,954 character predictions and analyzed these predictions with a bootstrap technique. The analysis suggests that a large number of character predictions per context length, perhaps as many as     10 3    , would be necessary to obtain a convergent estimate of the entropy rate, and if fewer predictions are used, the resulting h value may be underestimated. Our final entropy estimate was     h \u2248 1.22     bits per character.<\/jats:p>","DOI":"10.3390\/e21121201","type":"journal-article","created":{"date-parts":[[2019,12,6]],"date-time":"2019-12-06T10:41:44Z","timestamp":1575628904000},"page":"1201","update-policy":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Entropy Rate Estimation for English via a Large Cognitive Experiment Using Mechanical Turk"],"prefix":"10.3390","volume":"21","author":[{"given":"Geng","family":"Ren","sequence":"first","affiliation":[{"name":"Sorbonne Universit\u00e9, \u00c9cole Polytechnique Universitaire, 75005 Paris, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shuntaro","family":"Takahashi","sequence":"additional","affiliation":[{"name":"Graduate School of Engineering, The University of Tokyo, Tokyo 113-8654, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0003-1752-3951","authenticated-orcid":false,"given":"Kumiko","family":"Tanaka-Ishii","sequence":"additional","affiliation":[{"name":"Research Center for Advanced Science and Technology, The University of Tokyo, Tokyo 153-8904, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,12,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1002\/j.1538-7305.1951.tb01366.x","article-title":"Prediction and entropy of printed English","volume":"30","author":"Shannon","year":"1951","journal-title":"Bell Syst. Tech. J."},{"key":"ref_2","first-page":"31","article-title":"An Estimate of an Upper Bound for the Entropy of English","volume":"18","author":"Brown","year":"1992","journal-title":"Comput. Linguist."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1063\/1.166191","article-title":"Entropy estimation of symbol sequences","volume":"6","author":"Grassberger","year":"1996","journal-title":"Chaos"},{"doi-asserted-by":"crossref","unstructured":"Takahira, R., Tanaka-Ishii, K., and D\u0119bowski, \u0141. (2016). Entropy Rate Estimates for Natural Language\u2014A New Extrapolation of Compressed Large-Scale Corpora. Entropy, 18.","key":"ref_4","DOI":"10.3390\/e18100364"},{"doi-asserted-by":"crossref","unstructured":"Takahashi, S., and Tanaka-Ishii, K. (2018). Cross Entropy of Neural Language Models at Infinity\u2014A New Bound of the Entropy Rate. Entropy, 20.","key":"ref_5","DOI":"10.3390\/e20110839"},{"unstructured":"Manning, C., and Schutze, H. (1999). Foundations of Statistical Natural Language Processing, MIT Press.","key":"ref_6"},{"unstructured":"Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., and Salakhutdinov, R. (August, January 28). Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.","key":"ref_7"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"650","DOI":"10.2307\/1418794","article-title":"Long-Range Constraints in the Statistical Structure of Printed English","volume":"68","author":"Burton","year":"1955","journal-title":"Am. J. Psychol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1016\/S0019-9958(68)90261-1","article-title":"A note on the entropy of partially-known languages","volume":"12","author":"Jamison","year":"1968","journal-title":"Inf. Control"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1109\/TIT.1978.1055912","article-title":"A Convergent Gambling Estimate of the Entropy of English","volume":"24","author":"Cover","year":"1978","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/S0020-0255(97)00074-1","article-title":"Entropy of English Text: Experiments with Humans and a Machine Learning System Based on Rough Sets","volume":"104","author":"Moradi","year":"1998","journal-title":"Inf. Sci."},{"unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, Wiley-Interscience.","key":"ref_12"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1002\/j.1538-7305.1956.tb03809.x","article-title":"A New Interpretation of Information Rate","volume":"35","author":"Kelly","year":"1956","journal-title":"Bell Syst. Tech. J."},{"key":"ref_14","first-page":"65","article-title":"Optimal gambling systems for favorable games","volume":"Volume 1","author":"Breiman","year":"1961","journal-title":"Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability"},{"doi-asserted-by":"crossref","unstructured":"Sorokin, A., and Forsyth, D. (2008, January 23\u201328). Utility data annotation with Amazon Mechanical Turk. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Anchorage, AK, USA.","key":"ref_15","DOI":"10.1109\/CVPRW.2008.4562953"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"unstructured":"Callison-Burch, C., and Dredze, M. (2010, January 6). Creating speech and language data with Amazon\u2019s Mechanical Turk. Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk, Los Angeles, CA, USA.","key":"ref_17"},{"doi-asserted-by":"crossref","unstructured":"Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016, January 1\u20134). SQuAD: 100,000+ Questions for Machine Comprehension of Text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.","key":"ref_18","DOI":"10.18653\/v1\/D16-1264"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3758\/s13428-011-0124-6","article-title":"Conducting behavioral research on Amazon\u2019s Mechanical Turk","volume":"44","author":"Mason","year":"2012","journal-title":"Behav. Res. Methods"},{"unstructured":"Urbano, J., Morato, M.M., and Mart\u00edn, D. (2010). Crowdsourcing Preference Judgments for Evaluation of Music Similarity Tasks. ACM SIGIR Workshop on Crowdsourcing for Search Evaluation, ACM.","key":"ref_20"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1053","DOI":"10.1016\/j.ipm.2012.01.004","article-title":"Using crowdsourcing for TREC relevance assessment","volume":"48","author":"Alonso","year":"2012","journal-title":"Inf. Process. Manag. Int. J."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1038\/nmeth.2855","article-title":"Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods","volume":"11","author":"Warby","year":"2014","journal-title":"Nat. Methods"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1515\/FREQ.1990.44.9-10.243","article-title":"Der bekannte Grenzwert der redundanzfreien Information in Texten-eine Fehlinterpretation der Shannonschen Experimente?","volume":"44","author":"Hilberg","year":"1990","journal-title":"Frequenz"},{"doi-asserted-by":"crossref","unstructured":"Genzel, D., and Charniak, E. (2002, January 7\u201312). Entropy Rate Constancy in Text. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA.","key":"ref_24","DOI":"10.3115\/1073083.1073117"},{"doi-asserted-by":"crossref","unstructured":"Levy, R., and Jaeger, T.F. (2006, January 4\u20137). Speakers optimize information density through information density through syntactic reduction. Proceedings of the 19th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.","key":"ref_25","DOI":"10.7551\/mitpress\/7503.003.0111"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1016\/S0019-9958(68)91123-6","article-title":"Distortion Theory for Sources with Abstract Alphabets and Memory","volume":"13","author":"Berger","year":"1968","journal-title":"Inf. Control"},{"key":"ref_27","first-page":"142","article-title":"Coding theorems for a discrete source with a fidelity criterion","volume":"7","author":"Shannon","year":"1959","journal-title":"Int. Conv. Rec."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/www.mdpi.com\/1099-4300\/21\/12\/1201\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:40:52Z","timestamp":1760190052000},"score":1,"resource":{"primary":{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/www.mdpi.com\/1099-4300\/21\/12\/1201"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,6]]},"references-count":27,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["e21121201"],"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.3390\/e21121201","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2019,12,6]]}}}