Congress text-mining-event
1 like977 views
The document discusses the importance of large-scale text mining in Canadian web archives for historians studying recent history. It highlights the challenges of working with different file types, such as metadata and content, and the capabilities of advanced analysis on powerful computing clusters. Case studies illustrate themes from different political parties over time, emphasizing the necessity of web archives for comprehensive historical research.
Congress text-mining-event
- 1. The Record of our Recent Past Large-Scale Text Mining in Canadian Web Archives Ian Milligan Assistant Professor
- 2. Why? The sheer amount of social, cultural, and political information generated every day presents new opportunities for historians.
- 3. Could one even study the 1990s and beyond without web archives?
- 4. No. Historians need to do this now, or we’re going to be left behind.
- 5. One Case Study • Archive-It Research Services/University of Toronto Libraries: “Canadian Political Parties and Political Interest Groups” collection • 2005 - 2015 • ARC/WARC/WAT Files
- 6. Problem One: Historians want content, but we can only locally work with metadata
- 8. Do we want metadata or content analysis?
- 9. Historians NEED content, but metadata can help us find and contextualize it
- 13. Metadata Extraction • Results @ http://ianmilligan.ca/2015/02/05/topic- modeling-web-archive-modularity-classes/
- 14. Metadata Extraction • Conservative themes (2014): economic development, family, immigration, legislation, women’s issues, senior issues, Ukrainians, constituency offices, some prominent (and not-so- prominent) MPs, and of course, our economic action plan. • Liberal themes (2014): Justin Trudeau (the new leader), cuts to social programs, child poverty, mental health, municipal issues, labour, workers, Stop the Cuts, and housing.
- 15. Metadata Extraction • Conservative themes (2006): education, university, but tons of information on Aboriginal issues; • Liberal themes (2006): community questions, electoral topics, universities, human rights, child care support.
- 16. As well as short stories..
- 24. 2005 Canadian Federal Election
- 25. WATs help us find the files we need to use - and to contextualize them
- 26. Problem Two: You can do amazing things with the content (WARCs), but you need a cluster.
- 28. WARC Analysis • 2005-2009: 244 GB of content; 2.9 GB of plain text • 10,606,822 websites • On a local powerful node (3 Ghz 8-Core Intel Xeon E5/64 GB RAM, data on SSD), about three to four hours per query • On a cluster, about ~10-20 minutes per query, depending on traffic
- 29. Large-Scale Text Analysis • With Hadoop about 15-20 minutes to extract all plain-text from any specified queries: i.e. all pages belonging to Green Party, Liberal Party, Conservative Party, Council of Canadians, etc. • Compared to “out of memory”/ go home for an extended weekend on a local node
- 30. Large-Scale Text Analysis • NER/LDA/Keyword Frequency broken down by scrape date: i.e. scrape carried out 2005-10, see change over time; • Downside: not everything is optimized for parallel environment; if not, it crawls (there goes a day) • Downside: scrape date != creation date, requiring temporal analysis
- 31. Trial Three: Culturomics for Web Archives?
- 33. Some code/walkthroughs/ sample data available at https://github.com/ ianmilligan1/WAHR
- 34. Thank you! Ian Milligan Assistant Professor https://uwaterloo.ca/web-archive-group/