Scribd
Upload
20 views4 pages

Semantic Search in AI Documents

The document outlines a Python program that utilizes the SentenceTransformer library to perform semantic search on a set of AI-related documents. It includes functions to embed documents and find the most similar documents to a given query using cosine similarity. An example query is provided to demonstrate the functionality of the program.

Uploaded by

venkat Mohan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
20 views4 pages

Semantic Search in AI Documents

The document outlines a Python program that utilizes the SentenceTransformer library to perform semantic search on a set of AI-related documents. It includes functions to embed documents and find the most similar documents to a given query using cosine similarity. An example query is provided to demonstrate the functionality of the program.

Uploaded by

venkat Mohan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

PROGRAM:

!pip install sentence-transformers scikit-learn

from sentence_transformers import SentenceTransformer

from [Link] import cosine_similarity

import numpy as np

# Load the pre-trained model

model = SentenceTransformer('all-MiniLM-L6-v2')

# Example AI lab documents (you can replace these with your own data)

documents = [

"Machine learning models can be trained to recognize patterns in data.",

"AI research focuses on building intelligent systems that mimic human behavior.",

"The field of artificial intelligence has seen major advances in recent years.",

"Deep learning involves training neural networks on large datasets."

# Function to embed a list of documents

def embed_documents(documents):

return [Link](documents)

# Embedding the documents

document_embeddings = embed_documents(documents)

# Function to perform semantic search given a query

def semantic_search(query, document_embeddings, top_k=3):

# Embed the query

query_embedding = [Link]([query]

# Compute cosine similarities between the query and all document embeddings

cosine_similarities = cosine_similarity(query_embedding, document_embeddings)

# Get indices of the top_k most similar documents

top_indices = cosine_similarities[0].argsort()[-top_k:][::-1]
# Return the top_k most similar documents

return [(documents[i], cosine_similarities[0][i]) for i in top_indices]

# Example query

query = "What are the recent advancements in AI?"

# Perform semantic search

top_documents = semantic_search(query, document_embeddings)

# Print the results

print("Top similar documents for the query:", query)

for doc, score in top_documents:

print(f"Score: {score:.4f}, Document: {doc}")

OUTPUT:

You might also like