Vector DB War - Which One Should You Pick?🎯
By transforming raw data into vector embeddings, you unlock a wealth of potential.
These embeddings capture the essence of your data beyond the conventional structured variables, and form the bedrock of today’s AI applications.
If you want to build a production-ready AI application, assembling your vector retrieval stack is essential. One of the key components for this stack is your vector search and retrieval database.
Vector Database is the bridge between the latent, abstract mathematical representations of your data and their real-world applications. Your Vector Management stores, structures, and prepares your vector data for various machine learning tasks. Using Vector Search, you can perform efficient and relevant data retrieval from extensive data repositories.
Today, we're presenting the Vector DB Comparison hosted by Superlink, offering significant convenience for selecting the optimal option from a plethora of choices.
To access the full table👇
How to read the table?
Search Capabilities
Filters: Vector search can be combined with other search logic for performing more advanced queries. This is for metadata filtering within Vector Search. You can refine results based on additional contextual information and enhance precision in search queries.
Hybrid Search: Hybrid search is an advanced retrieval technique that enhances search accuracy and relevance by integrating multiple search algorithms. Typically it combines traditional keyword-based search with the contextual understanding offered by vector semantic search.
This is built-in blending of the 2 lists (keyword & vector search results) being done natively within the DB using something like Reciprocal Rank Fusion.
Facets: Faceted navigation is used for self-directed drill-down filtering on query results in a search app, where your application offers form controls for scoping search to groups of documents (for example, categories or brands).
Still hard to understand?
Imagine you're shopping online, and you have a long list of items to choose from. Faceted navigation is like having handy filters on the side that help you quickly narrow down your options based on specific categories or brands.
Instead of scrolling through everything, these filters act as shortcuts. If you're looking for shoes, you might use the "brand" filter to see only Adidas or Nike. It's a way to guide your search and focus on what matters to you, making the process much more user-friendly and efficient (see below example).
Geo Search: This is built-in geo-spatial index (like latitude and longitude) for location queries.
Multi-vector: This is the ability to create and index multiple vectors for a given document.
Sparse: This enables you to bring your own sparse vectors (e.g. TF-IDF) and have them be indexed and available for retrieval in the database.
BM25: This is built-in tokenization, indexing, and scoring for BM25, which is a sparse method.
Full-Text: A full-text search finds words or phrases within larger texts. You can search within a specific text field or across all text fields.This is built-in inverted index for terms, capable of boolean queries.
Models
Text Model: This is the ability to plug in commonly-used embedding models for text vectorization (sentence-transformer/HuggingFace feature extraction models etc.)
Image Model: This is the ability to plug in image embedding models for image vectorization (e.g. CLIP).
Struct Model: This is the ability to plug in models to embed structured data like user clicks and graphs (node2vec).
APIs
RAG: This calls a LLM model internally for a full-stack RAG solution.
LangChain: This is built-in integration with LangChain.
LlamaIndex: This is built-in integration with LlamaIndex.
RecSys: This is built-in support for recommender system.
Ops
Pricing: Yeah this is important too!
Vector Dims: Max number of vector dimensions.
Document Size: Max size of a stored document.
Multi-Tenant: This is the ability to store multiple isolated indexes within one database instances.
Managed: This means that a managed cloud offering is available.
In-process: This is the ability to run embedded within the application process, which leads to a simpler deployment and management in situations when the whole workload fits into one machine (see eg. Embedded Weaviate).
Disk Index: The vector search index can be held on disk while in use, as oppose to holding it in RAM, which makes the management of large indices more affordable.
Ephemeral Index: Ephemeral index supports without a server.
Sharding: The search index is sharded across multiple machines.
That’s it for today!
🛠️✨ Happy practicing and happy building! 🚀🌟
Thanks for reading our newsletter. You can follow us here: Angelina Linkedin or Twitter and Mehdi Linkedin or Twitter.
Source of images/quotes:
Blog post by Epsilla. Hybrid Search
Blog post by Microsoft. Add faceted navigation to a search app
Blog post by Superlinked. Building Blocks