Hybrid Retrieval with FAISS & BM25

Introduction to Document Retrieval Challenges

Document retrieval is pivotal in various applications, especially in Retrieval-Augmented Generation (RAG) systems. The challenge lies in swiftly providing contextually relevant documents that align with user queries. Traditional methods often struggle due to their inherent limitations, leading to less than optimal results.

Understanding FAISS and BM25

FAISS, which stands for Facebook AI Similarity Search, excels in performing fast vector similarity searches. However, while this approach is capable of identifying semantically similar documents, it may not always pinpoint the most relevant content for specific queries. On the other hand, the BM25 algorithm focuses on keyword-based searches, providing relevance scores based on term frequency and document length. Each method has distinct advantages, making it evident that a hybrid retrieval system can enhance performance significantly.

The Advantages of a Hybrid Approach

Integrating both FAISS and BM25 creates a powerful hybrid retrieval system. This combination allows users to reap the benefits of both vector and keyword searches. By leveraging the strengths of each method, you can achieve higher accuracy in document retrieval. The improved relevance of results means users will find what they're looking for quicker and more effectively.

Building Your Hybrid Retrieval System

To create your hybrid retrieval system, we’ll take you step-by-step through the essential processes, starting with the loading of documents and creating embeddings. Below, we outline how to index documents, perform searches using FAISS, and compute the BM25 scores.

Loading and Indexing Documents

We'll begin by loading PDF documents and preparing them for processing. Here’s how you can do it.

Loading PDFs and Creating Embeddings

import PyPDF2
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

# Load PDFs
def load_pdf(file_path):
 with open(file_path, 'rb') as file:
 reader = PyPDF2.PdfReader(file)
 return '\n'.join(page.extract_text() for page in reader.pages)

# Create document embeddings
vectorizer = TfidfVectorizer() # You may replace this with any embedding model
documents = ['doc1.pdf', 'doc2.pdf'] # List your PDF files here
corpus = [load_pdf(doc) for doc in documents]
embeddings = vectorizer.fit_transform(corpus).toarray()

Performing FAISS Searches

Next, we will set up FAISS for quick vector searches. We’ll retrieve the embeddings and search based on the user query.

FAISS Search Implementation

import faiss

# Build index
dimension = embeddings.shape[1]
index = faiss.IndexFlatL2(dimension)
index.add(np.array(embeddings).astype('float32'))

# Performing search
def faiss_search(query, k=5):
 query_embedding = vectorizer.transform([query]).toarray().astype('float32')
 distances, indices = index.search(query_embedding, k)
 return indices

Computing BM25 Scores

After obtaining preliminary results from FAISS, the next step involves determining the BM25 scores for improved ranking.

BM25 Implementation

from rank_bm25 import BM25Okapi

# Computing BM25 scores
bm25 = BM25Okapi(corpus)
def bm25_search(query):
 scores = bm25.get_scores(query.split())
 return np.argsort(scores)[::-1][:5] # Return top 5 results

Merging Results from FAISS and BM25

Finally, we merge the results from both FAISS and BM25 to provide the user with the most relevant documents.

Merging Results

def hybrid_search(query):
 faiss_results = faiss_search(query)
 bm25_results = bm25_search(query)
 merged_results = list(set(faiss_results) | set(bm25_results)) # Combine results
 return merged_results[:5] # Returning top results

Conclusion and Next Steps

Ready to elevate your document retrieval strategy? Embrace the hybrid approach with ProsperaSoft and unlock a new level of efficiency in handling user data.

Just get in touch with us and we can discuss how ProsperaSoft can contribute in your success