Wikitech-l March 2025

34 participants
37 discussions

Several database changes happening in the coming months
by Amir Sarabadani 24 Apr '25

24 Apr '25

Hello, in the next coming months, these changes will happen in databases and the infrastructure. And it might affect you if you rely on them in your tools or queries. This list is ordered based on how soon the change will happen. We understand that updating your tools and systems can be time consuming, hence we are giving an advanced notice. I truly apologize for the inconvenience but many of these changes are needed to keep the site running smoothly. Image table redesign Around fourteen years after the creation of T28741 <https://phabricator.wikimedia.org/T28741>, we are implementing the changes described therein. Currently, every current version of an image has a row in the image table and if there are older versions of that file, those rows could be found in the oldimage table. These two tables (image and oldimage) will be dropped in around two months. The replacement will be two main tables: file and filerevision. Every file will have a row in the file table describing the name and the type. Every version of the file (current and old) will have a row in filerevision describing the file-specific information such as its size or the hash of the file, similar to the existing distinction between pages and revisions. Another improvement is that every file and file revision will get a unique auto increment id simplifying many operations and queries. You can check T28741 <https://phabricator.wikimedia.org/T28741> for more information. The new tables are already accessible in wikireplicas but the data hasn’t been fully migrated yet. Term store split out of wikidata’s database Wikidata’s database has been growing too fast and we need to move the term store (tables starting with wbt_) to a dedicated cluster to allow growth and improve wikidata’s performance by utilizing cache locality. The new section will be called x3 and you will be able to access it in wikireplicas but this also means you won’t be able to join these tables with the rest of wikidata’s database (such as page table) since they will be residing in two physically separate servers that also means most of your queries to wikidata’s database (and term store) will become faster. We are aiming for the switch to happen in three months’ time. You can follow the work in T351820 <https://phabricator.wikimedia.org/T351820>. Additionally, wb_type table will be dropped and the mapping will be hard-coded in the code instead. See gerrit:1110810 <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1110810> for more details. This helped us simplify a lot of Wikibase code (example <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1110720>). Categorylinks normalization Categorylinks is the next table in the series of links tables being normalized via the linktarget table (parent ticket <https://phabricator.wikimedia.org/T300222>, RFC <https://phabricator.wikimedia.org/T222224>). Similar to templatelinks and pagelinks tables, cl_to will be dropped and instead the new field cl_target_id will point to lt_id in the linktarget table. We will also drop the cl_collation field and replace it with cl_collation_id which will point to the collation_id field on the new table we are introducing called collation. We are aiming to get this fully done by the end of the next quarter (end of June 2025) but it depends on how fast the migration script can operate and that’s outside of our control. You can follow the work in T299951 <https://phabricator.wikimedia.org/T299951>.It’s worth noting that after this migration is done, we will start working on the imagelinks table. Thank you -- *Amir Sarabadani (he/him)* Staff Database Architect Wikimedia Foundation <https://wikimediafoundation.org/>

1 1

Excited to Contribute to Wikimedia Foundation
by nidha ahmed 17 Apr '25

17 Apr '25

Hi Sir/Madam, I hope you’re doing well. I am Nidha Ahmed, and a junior undergraduate student studying Artificial Intelligence and Data Science. I’m also going to join Google as an SWE Intern next summer. I’ve always been a huge admirer of the Wikimedia Foundation and its mission to make free educational content accessible to everyone. It’s incredible how your work empowers millions of learners around the globe, and I’m very eager to be part of this impact. I have experience with JavaScript, HTML, CSS, jQuery, ReactJS, and Python, and I’ve dabbled a bit in Android development as well. While I’m new to PHP and Ruby on Rails, I’m super excited to learn them! If you could share some resources or tips to get started with these technologies, it would be a huge help. A little about me: I’ve worked on projects like a movie recommendation system, a text summarization web app, and a geospatial analysis tool for public health. I’ve also contributed to open-source initiatives like designing real-time heatmaps for SoundScape. I’m looking to grow my skill set, hence I’m so drawn to Wikimedia. It would be helpful for me to get guidance on how to start contributing. I’m genuinely excited about the chance to learn from your community and support your amazing mission. I'm attaching my resume below for reference and proofs. Looking forward to hearing from you! Best regards, Nidha Ahmed Mohammad.

3 2

Database dumps
by Byrial Jensen 17 Apr '25

17 Apr '25

Until some weeks ago http://dumps.wikimedia.org/backup-index.html used to show 4 dumps in progress at the same time. That meant that new database dumps normally was available within about 3 weeks for all databases except for enwiki and maybe dewiki where the dump process due to size took longer time. However the 4 dumps processes at one time become 3 some weeks ago. And after massive failures at June 4, only one dump has been in progress at the same time. So at the current speed it will take several months to come thru all dumps. Is it possible to speed up the process again using several dump processes at the same time? Thank you, Byrial

3 2

User-Agent:
by Domas Mituzas 17 Apr '25

17 Apr '25

Hi! from now on specific per-bot/per-software/per-client User-Agent header is mandatory for contacting Wikimedia sites. Domas

19 61

API issue
by ovskmendov＠proton.me 17 Apr '25

17 Apr '25

Whenever I access the API, I get "Wikimedia Error: Our servers are currently under maintenance or experiencing a technical problem." Sometimes if I wait a bit, I'll get a few good requests, but it will shortly go back to the previous message. My code was working just fine a few days ago and I didn't change anything. Sent with [Proton Mail](https://proton.me/) secure email.

7 12

Acquiring list of templates including external links
by Takashi OTA 17 Apr '25

17 Apr '25

Hoi, This is an inquiry from my friend in academia, researching about Wikipedia. He would like to know whether there's a way to acquire a list of templates including external links. Here are some examples including external links. https://ja.wikipedia.org/wiki/Template:JOI/doc https://ja.wikipedia.org/wiki/Template:Twitter/doc Such links are stored in externallinks.sql.gz, in an expanded form. When you want to check increase/decrease of linked domains in chronological order through edit history, you have to check pages-meta-history1.xml etc. In a such case, traditional links and links by templates are mixed, Therefore, the latter ones (links by templates) should be expanded to traditional link forms. Sorry if what I am saying does not make sense. Thanks in advance, --Takashi Ota [[U:Takot]]

13 24

Talk to the Search Platform / Query Service Team—April 2, 2025
by Guillaume Lederrey 02 Apr '25

02 Apr '25

Hello all! The Search Platform Team usually holds an open meeting on the first Wednesday of each month. Come talk to us about anything related to Wikimedia search, Wikidata Query Service (WDQS), Wikimedia Commons Query Service (WCQS), etc.! Feel free to add your items to the Etherpad Agenda for the next meeting. Details for our next meeting: Date: Wednesday, April 2, 2025 Time: 15:00-16:00 UTC / 08:00 PST / 11:00 EST / 17:00 CET Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours Google Meet link: https://meet.google.com/vgj-bbeb-uyi Join by phone: https://tel.meet/vgj-bbeb-uyi?pin=8118110806927 Have fun and see you soon! Guillaume -- *Guillaume Lederrey* (he/him) Engineering Manager Wikimedia Foundation <https://wikimediafoundation.org/>

1 1

2025 Wikimedia Hackathon - registration is now open
by Deb Tankersley 01 Apr '25

01 Apr '25

*Registration for the 2025 Wikimedia Hackathon is now open! * *(with apologies for cross-posting)* The annual Wikimedia Hackathon <https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2025> is organized by the Wikimedia Foundation <https://wikimediafoundation.org/> in collaboration with Wikimedia User Group Turkey (WMTR) <https://meta.wikimedia.org/wiki/Wikimedia_Community_User_Group_Turkey> and will be taking place in Istanbul, Turkey, from May 2nd through May 4th, 2025 at the Renaissance Polat Istanbul Hotel <https://www.marriott.com/en-us/hotels/istrn-renaissance-polat-istanbul-hote…> . The Hackathon is an on-site, in-person event. Those people who cannot attend the Hackathon in-person will be able to follow some of the program content online, but should not register using the link provided below. More information will be provided closer to the event on how to participate remotely. Registration will be open until mid April 2025, or until we reach event capacity. The Hackathon event is free for all* to attend *and each person that attends the Hackathon is required to register. - Catering (lunch, dinner, snacks, and drinks) will be provided during the event. - Covid-related equipment will also be available upon request (e.g. self test kits and masks). - We’ll have discounted room codes for those that want to stay at the hotel venue, to be announced in the coming weeks. - Check if your passport is up to date and if you will need a visa (or an e-visa) to travel to Turkey, citizens of most countries can apply for an e-visa online <https://www.mfa.gov.tr/visa-information-for-foreigners.en.mfa>. Participants are expected to book their own travel arrangements and accommodation. However, the Wikimedia Foundation is offering to take care of travel and accommodation for a limited number of scholars and you will be able to apply for a scholarship during the registration process. Scholarship applicants - please note: - *Scholarship application and Wikimedia Foundation staff sponsorship will only be available to request until December 10th, 2024. * - Priority will be given to scholars who can mentor or support in areas of high demand and staff with ambitious or impactful projects. - Those that have requested a scholarship or sponsorship will be notified by mid to late December 2024 about the status of their application. - All scholarships provided will adhere to the WMF travel policy <https://foundation.wikimedia.org/wiki/Policy:Scholarship_travel_policy> guidelines. *Scholarship to this event will cover:* - Roundtrip airfare between your city of residence and Istanbul, Turkey - arriving on May 1, 2025 and departing on May 5, 2025. - Private accommodation at the conference hotel for up to 4 nights - Medical Insurance Coverage for any scholars traveling outside of their home country. (Scholars local to Turkey will need to use their regular insurance coverage.). - Airport transfers between the Istanbul Airport (IST) and the conference hotel. - All meals will be provided for the duration of the event (lunch, drinks, snacks, and dinner will be provided on event days; breakfast will be provided by the hotel for guests staying there). In addition, dinner will be provided on Thursday before the Hackathon officially kicks off. - Fees associated with obtaining a visa to travel to Turkey (visa application fees, and travel to in-person travel visa appointment (if needed)). - Citizens of most countries can apply for an e-visa online <https://www.mfa.gov.tr/visa-information-for-foreigners.en.mfa>. *Scholarship to this event will not cover:* - Per diem costs on travel days. - Ground transportation costs between your residence and home airport. - Insurance for your personal possessions. The Hackathon registration form runs on Pretix (a free, open-sourced third-party platform that helps organize and manage events) which may subject your data to additional terms <https://pretix.eu/about/en/privacy>. For more information on WMF’s privacy and data-handling for the Hackathon, please read the privacy statement <https://foundation.wikimedia.org/wiki/Legal:Wikimedia_Hackathon_Privacy_Sta…> . <https://pretix.eu/wikimedia/hackathon2025> If you encounter any issues while registering, or have questions about the registration process, feel free to contact the coordination team at hackathon(a)wikimedia.org. Cheers, Deb -- deb tankersley (she/her) developer outreach program management Wikimedia Foundation

1 1

EBNF grammar project status?
by Steve Bennett 01 Apr '25

01 Apr '25

What's the status of the project to create a grammar for Wikitext in EBNF? There are two pages: http://meta.wikimedia.org/wiki/Wikitext_Metasyntax http://www.mediawiki.org/wiki/Markup_spec Nothing seems to have happened since January this year. Also the comments on the latter page seem to indicate a lack of clear goal: is this just a fun project, is it to improve the existing parser, or is it to facilititate a new parser? It's obviously a lot of work, so it needs to be of clear benefit. Brion requested the grammar IIRC (and there's a comment to that effect at http://bugzilla.wikimedia.org/show_bug.cgi?id=7 ), so I'm wondering what became of it. Is there still a goal of replacing the parser? Or is there some alternative plan? Steve

26 217

Feedback Requested: SwaggerUI sandbox for MediaWiki REST APIs
by Halley Coplin 31 Mar '25

31 Mar '25

Hello, technical volunteer community! Earlier today we kicked off a limited release of a new SwaggerUI-based sandbox experience for MediaWiki REST APIs! In addition to offering funcitonally validated, interactive documentation, the sandbox will support translation (powered by translatewiki <https://translatewiki.net/>) for all endpoint and parameter descriptions, so that developers may read the documentation and download OpenAPI specs in their preferred language. The combination of interactive sandbox docs and OpenAPI specs were among the top requested items in the latest developer satisfaction survey <https://www.mediawiki.org/wiki/Developer_Satisfaction_Survey/2025/APIs#api-…>, so we are eager to hear what you all think about the new experience! The initial launch is available in the following Wikipedia language communities (selected based on the availability of translation coverage): - Arabic <https://ar.wikipedia.org/wiki/%D8%AE%D8%A7%D8%B5:%D9%85%D9%84%D8%B9%D8%A8_%…> (ar) - German <https://de.wikipedia.org/wiki/Spezial:RestSandbox> (de) - French <https://fr.wikipedia.org/wiki/Sp%C3%A9cial%3ARestSandbox> (fr) - Hebrew <https://he.wikipedia.org/wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:RestSandbox> (he) - Interlingua <https://ia.wikipedia.org/wiki/Special%3ARestSandbox> (ia) - Dutch <https://nl.wikipedia.org/wiki/Speciaal:RestSandbox> (nl) - Chinese <https://zh.wikipedia.org/wiki/Special:REST%E6%B2%99%E7%9B%92> (zh-hans, zh-hant) - English (testwiki <https://test.wikipedia.org/wiki/Special:RestSandbox>, Wiktionary <https://en.wiktionary.org/wiki/Special%3ARestSandbox>) If you are a member of any of these communities (or want to check it out in the testwiki), we would love your feedback <https://www.mediawiki.org/wiki/MediaWiki_Interfaces_Team/Feature_Feedback/R…>. On the linked feedback page, you can see additional information about this project, as well as track progress on issues that we are already aware of. In addition to verifying that everything is translated and working correctly, we would like to get general feedback on the experience. Please review and respond to the open ended prompts either here or on the REST Sandbox feedback <https://www.mediawiki.org/wiki/MediaWiki_Interfaces_Team/Feature_Feedback/R…> discussion page: 1. Do you believe that providing the documentation in your wiki community's preferred language will help developers better understand the content? 2. Are the translations accurate and appropriate in your language? 3. How disruptive or confusing are the mixed English headers and navigation items? 4. How do you intend to use the OpenAPI specs that are now available through the sandbox? 5. How will the sandbox change how you interact with the MediaWiki REST API? 6. Would you like to see similar experiences for other APIs offered by Wikimedia? 7. As an extension author, would you leverage this experience to provide interactive documentation for your extension's API? Following the initial limited release, we will collect feedback for approximately two weeks. Assuming we get enough feedback and do not have any major issues, we plan to make the same REST Sandbox available across all wiki projects before the end of April. Please let me know if you have any initial questions, comments, or concerns. Thanks, Halley *Halley Coplin* (she/her) Sr. Product Manager, MediaWiki Interfaces Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l March 2025