3
Yukon Qualification Exam
Part
At a Glance
Where is it?
Access the exam on your computer at https://www.raterhub.com/evaluation/
rater
Log in with the Gmail account
you provided to Appen
To view results, click on the
result title in the rating tool on
your computer, then navigate to
https://www.raterhub.com/url
on your mobile phone, and log
in with the same Gmail account
What's in it?
7 Needs Met tasks, each consisting of a query and multiple results
Determine user intent based on the query and location, review each result,
assign a Page Quality rating, then assign a Needs Met rating based on how
well the result meets the user intent
You must enter both PQ and NM ratings, but only NM will be scored
How do I pass it?
This is an open book exam, so keep the General Guidelines open
Quick Rating Guide for Needs Met:
Start at MM, then move the slider based on the following:
Is the result close enough to be helpful for a user in this location?
Yes, the result is close to the user's location: + 1 (MM+ to HM)
Yes, but closer results would be better: +05 . (MM+)
Location not important for this query: No change (MM)
Sort of, the result is near the user but too far to be convenient: - 1 (SM to SM+)
No, the result is nowhere near the intended location: FailsM (skip next question)
Does the result meet the query intent?
Yes, the result is highly relevant, accurate, up-to-date, highly credible, and gives the user
exactly what they are looking for and more: +2 (HM to FullyM*)
Yes, the result is highly relevant, accurate, up-to-date, and highly credible: +1 (MM+ to HM)
Yes, the result is relevant and meets the bare minimum of the query intent, but no more: + 0
to 05
. (MM to MM+)
Sort of, the result is relevant but slightly out-of-date or has minor inaccuracies: - 1 (SM to
SM+)
Not really, the result is on-topic but too broad, inaccurate, or outdated to be useful: - 2
(FailsM+ to SM)
No, the result isn't what the user is looking for and would satisfy no one: FailsM
*WSRBs are typically not rated above HM unless they satisfy a website intent or dual intent
Tips and reminders:
When unsure between two ratings, choose the lower rating
News or encyclopedia information must be accurate, up-to-date, and highly
credible
Medical or science information must align with well-established medical or
scientific consensus
Page Quality + Needs Met:
Useless results should always be rated FailsM, even if they have a high PQ rating;
useless is useless
On-topic, helpful results with low PQ should get a lower NM rating than on-topic,
helpful results with high PQ; page quality is part of helpfulness
The HM rating is appropriate for on-topic, helpful results with high PQ, or for on-
topic, very helpful results with medium PQ and other desirable characteristics
The HM rating is not appropriate for results with low PQ or any other undesirable
characteristics such as outdated or inaccurate information
SM is often appropriate for low quality but on-topic pages, but sometimes pages
have such low quality that they are useless for nearly all queries and should be
rated FailsM (e.g. gibberish pages, unless they are an exact match for a query with
clear website intent, in which case they should be rated FullyM)
Pages that lack a beneficial purpose should be rated Lowest PQ, regardless of
NM rating or page design (see Section 4.1)
Always rate the following as Fails to Meet:
Harmful to Self or Other Individuals (Section 4.2)
Harmful to Specified Groups (Section 4.3)
Harmfully Misleading Information (Section 4.4)
Untrustworthy (Section 4.5)
Spammy (Section 4.6)
Porn (Section 15.1)
Special Content Results Blocks (SCRB) vs. Web Search Result Blocks (WSRB):
SCRBs provide "at a glance" results without WSRBs include only a title, url, and brief
needing to go to a website. These should be snippet. These should be rated based solely on
rated based on block content and can be the landing page content (ignore the snippet)
rated as high as HM+ or FullyM so long as they and are typically not rated above HM unless
meet all requirements for the rating. they satisfy a website or dual intent.
