Creating Search Quality Algorithms - Richard Lawrence - BrightonSEO.pdf
- 1. How to create your own search quality evaluation algorithms Richard Lawrence Sanity.io @richlawre
- 2. @richlawre ● Principal SEO at Sanity Who the hell is this guy anyway?
- 3. Who the hell is this guy anyway? @richlawre ● Sanity is a headless CMS and more!
- 4. @richlawre ● Doing a Data Science degree in my spare time Who the hell is this guy anyway?
- 6. The ‘helpful content update’ might have been a bit of a damp squib… @richlawre
- 7. …but Google is always working towards ranking helpful content more highly @richlawre
- 8. So wouldn’t it be great to know if your content is helping your audience - at scale? @richlawre
- 9. The search rater guidelines hold the key @richlawre 167 page document that says what good looks like!
- 10. Google says it doesn’t directly use the ratings in its ranking algorithms “We use responses from Raters to evaluate changes, but they don’t directly impact how our search results are ranked.” bit.ly/ratings-answer @richlawre
- 11. But it will use the rated content to help find features of what ‘good’ looks like @richlawre
- 12. Similar methods have been used for years in various areas - like counterfeit notes @richlawre
- 13. Features are found that best separate authentic and counterfeit notes Distance between edge & watermark Width of shaded area Counterfeit Authentic @richlawre
- 14. Features for high vs. low quality content will likely be more complex @richlawre
- 15. Bing confirmed this is how it works in 2019 bit.ly/bing-confirmation @richlawre
- 16. With 90% of its algorithms being ML based @richlawre bit.ly/bing-features
- 17. Plus it revealed its process @richlawre bit.ly/bing-process
- 18. So how can we harness this as an industry? @richlawre
- 19. We can try to create our own! @richlawre
- 20. 1. Label the content 2. Create a ‘Needs Met’ algorithm 3. Create a ‘Page Quality’ algorithm What we need to do @richlawre
- 22. Get a representative sample of searches 448 million search queries bit.ly/448-million @richlawre
- 23. Here’s how to play around with the file @richlawre bit.ly/large-file
- 24. Then gather the top 20 rankings for each sample query Likely available feature of your favourite rank tracking software @richlawre
- 25. Use some search raters to rate the content Collect labels Choose provider Create guidelines Must not be identical to Google’s… Needs Met & Page Quality 2 search raters with 3rd called in for disagreements @richlawre
- 26. Creating a Needs Met algorithm @richlawre
- 27. This measures fulfilling search intent Features will mainly be relating to relevance and structure @richlawre
- 28. GPT language models are perfect for this The open source option @richlawre
- 29. GPT-3 became cheaper in September too @richlawre
- 30. We need to create a pattern for GPT-J to learn Content: <h1>Compare car insurance quotes</h1> <p>It's quick and easy to compare car insurance and find cheaper cover – we just need a few details about you and your vehicle.</p> Target query: car insurance Needs Met rating: Good @richlawre
- 31. It will then rate new content Content: <h1>Car insurance</h1> <p>From theft to write-offs and even lost keys, you'll be covered with us. Here's what you'll like about our comprehensive cover </p> Target query: car insurance Needs Met rating: ????? @richlawre
- 32. We need to scrape content from each page to give to the language model - with the rating @richlawre
- 33. Then use this info to train GPT-J @richlawre bit.ly/finetune-gptj
- 34. You can also use existing services @richlawre NLP Cloud Forefront.ai
- 35. NLP Cloud also became cheaper! @richlawre
- 36. Validate performance with a test set @richlawre
- 37. Judge performance with a Confusion Matrix @richlawre Correct Wrong Correct Wrong True positive False negative False positive True negative Actual Prediction
- 38. Few shot learning can help improve performance @richlawre Prompt Example 1 Rating: Excellent Example 2 Rating: Poor Example 3 Rating: ???? GPT-J Good
- 39. As can explaining to the model what it needs to do! @richlawre Consider the content to rate. Rate it according how well it fits the search query.
- 40. We’ve done this for you within Sanity Studio @richlawre
- 41. And lots of other great features @richlawre
- 42. Contact us for more info about the beta for these features: bit.ly/sanity-beta @richlawre
- 43. This isn’t perfect of course - though still very useful @richlawre ● Only text content ● Useful indication only ● Great at scale
- 44. Creating a Page Quality algorithm @richlawre
- 45. This is much more difficult! @richlawre
- 46. It measures how well a page achieves its purpose @richlawre This is about quality of content, independent of search queries
- 47. So features can relate to a large number of areas! @richlawre ‘Main Content’ vs ‘Supplementary Content’ Website background information Amount of Main Content Position of Main Content Depth of ‘about’ info Wikipedia presence
- 48. And you have to work out how to measure them @richlawre Amount of Main Content Length of Main Content area Number of words in Main Content
- 49. It becomes a huge multivariate challenge @richlawre Page Length of MC area ‘About us’ word count Clicks to ‘About us’ Page 1 17cm 500 2 Page 2 20cm 300 1 Page 3 15cm 1000 2 Page 4 25cm 750 3
- 50. Then we need to find features that best separate the groups Number of words in ‘About’ section Length of ‘Main Content’ area High quality Low quality @richlawre
- 51. But with a large number of features! @richlawre
- 52. This can be explored with a number of potential models @richlawre Linear Discriminant Analysis
- 53. @richlawre This can be explored with a number of potential models Random Forest
- 54. @richlawre This can be explored with a number of potential models Neural Network
- 55. This is a huge challenge! @richlawre
- 57. How to measure them? @richlawre
- 59. The work is ongoing here! @richlawre
- 61. Google likely uses its raters to gather labelled data on content quality @richlawre
- 62. It will then likely use that to find features of ‘good’ and ‘bad’ content @richlawre
- 63. And creates algorithms to distinguish between the two @richlawre
- 64. You can do the same! @richlawre
- 65. Get your own labelled content and create your own scoring algorithms @richlawre
- 66. We have created a ‘Needs Met’ score within Sanity Studio @richlawre
- 67. So that you can get an indication of content calibre directly in your publishing workflow @richlawre
- 68. Contact us to get more info about the beta here: bit.ly/sanity-beta @richlawre
- 69. Richard Lawrence Principal at Sanity.io @richlawre @richlawre