Related content using embeddings

[simonw]

I'm writing a TIL about this as I go.

[simonw]

The first time this ran successfully took this long:

If I got the code right it should be a lot faster on subsequent runs, since it will only calculate similarities for stuff related to the latest added entries.

[simonw]

https://til.simonwillison.net/tils now has the new tables:

https://til.simonwillison.net/tils/similarities?_facet=id looks good.

[simonw]

Here's the new SQL query: https://til.simonwillison.net/tils?sql=select%0D%0A++til.topic%2C+til.slug%2C+til.title%2C+til.created%0D%0Afrom+til%0D%0Ajoin+similarities+on+til.path+%3D+similarities.other_id%0D%0Awhere+similarities.id+%3D+%27python_pyproject.md%27%0D%0Aorder+by+similarities.score+desc+limit+10

select
  til.topic, til.slug, til.title, til.created
from til
join similarities on til.path = similarities.other_id
where similarities.id = 'python_pyproject.md'
order by similarities.score desc limit 10

[simonw]

Didn't quite work:

Skipped 446 rows that already existed
[{"id": "svg_dynamic-line-chart.md", "other_id": "observable-plot_wider-tooltip-areas.md", "score": 0.7923009914460658}]
error: Could not access 'HEAD~10'
Error: Must specify entries or --all

[simonw]

OK, the second time that ran it took 8s, which is as hoped for:

[simonw]

Here's the TIL with the full write-up: https://til.simonwillison.net/llms/openai-embeddings-related-content

[simonw]

The related content for the new article looks good:

Related

• sqlite Related content with SQLite FTS and a Datasette template function - 2022-07-31
• python Calculating embeddings with gtr-t5-large in Python - 2023-01-31
• datasette Crawling Datasette with Datasette - 2022-02-27
• sqlite Copy tables between SQLite databases - 2023-04-03
• mastodon Export a Mastodon timeline to SQLite - 2022-11-04
• datasette Scraping Reddit and writing data to the Datasette write API - 2023-03-13
• sqlite Comparing two training datasets using sqlite-utils - 2023-05-23
• shot-scraper Social media cards generated with shot-scraper - 2023-04-29
• sphinx Adding Sphinx autodoc to a project, and configuring Read The Docs to build it - 2021-08-10
• sqlite Replicating SQLite with rqlite - 2020-12-28