Weaviate

An Open Source Neural Search Engine

9 min readApr 17, 2022

In today’s digitalized world, search engines have become one of the most powerful tools on the internet and an essential part of our daily lives. By consolidating and organizing the wealth of information available online, search engines like Google, Yahoo, or Bing help billions of online users find the content they need at a rapid pace. Almost 30 percent of global web traffic is generated via online search usage, showing the vital role these platforms play in directing and navigating user flows to different websites. As search engines rank search results in order of relevancy, meaning that the most valuable links for users’ queries are displayed prominently on the results page, high rankings have become one of the top digital priorities for companies worldwide.

Summary of today’s article

What is keyword-based searching
What is Neural Searching
Intro to Weaviate; Vector Search Engine
Weaviate Architecture
Weaviate Modules and Custom Modules
Use cases of Weaviate

What is Keyword-based searching

The most straightforward way of searching is keyword-based searching. What it does is match the query with the keywords which already have and output the best result. But this approach is fragile as there are multiple ways of describing the same data object. Another drawback of this approach is the polysemy which means that the same word can have multiple meanings.

Back in 2015 tech giant, Google launched RankBrain which combines AI to bring a revolution in keyword-based search.

Before RankBrain, the process was something like this.

After Introducing RankBrain

So what RankBrain does,

Depending on the keyword, RankBrain will increase or decrease the importance of backlinks, content refreshes, domain authority etc.
Then, it looks at how Google searchers interact with the new search results. If users like the new algorithm better, it stays. If not, RankBrain rolls back the old algorithm.

But since data is growing exponentially and people wanted to search through voice, images, videos and interact with voice assistants like Siri, Cortana, and Alexa, engineers wanted to apply AI and Deep Learning techniques in order to achieve these use cases. That’s where the neural search comes into the picture.

What is Neural Search

Neural Search is a new approach to retrieving information using neural networks. It uses advanced techniques as below to make the search more accurate and scalable.

Use of word embeddings to capture semantic properties of words
Focus on end-to-end learning of higher-level tasks (e.g., question answering, text summarization)

Applications of Neural Search are,

Semantic search
Search between data types
Search with multiple data types

Typical Process of a neural search engine

Index Flow — the AI model stores datasets in the form of searchable indexes with the help of an encoder neural network

2. Query Flow — the user inputs his query in the form of a document. The AI then uses neural networks to identify the query and then comes up with a list of results based on a similarity score within the input embeddings by using a matching algorithm.

How does Weaviate come into the picture?

Weaviate

Weaviate is a vector search engine and vector database. Weaviate uses machine learning to vectorize and store data and find answers to natural language queries. With Weaviate you can also bring your custom ML models to production scale.

Additionally, Weaviate is a low-latency vector search engine with out-of-the-box support for different media types (text, images, etc.). It offers Semantic Search, Question-Answer Extraction, Classification, Customizable Models (PyTorch/TensorFlow/Keras), and more. Built from scratch in Go, Weaviate stores both objects and vectors, allowing for combining vector search with structured filtering and the fault-tolerance of a cloud-native database, all accessible through GraphQL, REST, and various programming languages clients. (Intro from the documentation)

80% of the data is unstructured. Hence it is difficult to search and classify data. So adding context and meaning to the data is important. Weaviate focuses on this issue and builds a production-grade, cloud-native, AI-based semantic search engine (or vector search engine | neural search engine).

I’ll explain the drawback of the traditional search engine by using an example in the documentation.

{ “data”: “The Eiffel Tower is a wrought iron lattice tower on the  Champ de Mars in Paris.” }

Storing this data in a traditional search engine might leverage inverted indices to index the data. This means that to retrieve the data; you need to search for “Eiffel Tower” or “wrought iron lattice”, etc. to find it. But what if you have vast amounts of data and you want the document about the Eiffel Tower but you search for: “landmarks in France”? Traditional search engines can’t help you there and this is where vector search engines show their superiority.

Weaviate uses vector indexing mechanisms at its core to represent the data. The vectorization modules (e.g., the NLP module) vectorizes the data object in a vector space where the data object sits near the search query. This means that Weaviate can’t make a 100% match, but a very high one to show you the results.

So what is vectorization?

It is also called word embedding, vector mapping, feature learning
Maps the data to vectors of real numbers.
Data Stores in a high dimensional vector space
This captures the semantics and relationships between data
Data is automatically stored with its meaning
Enables automatic classification

Vector Space

each data object has a corresponding vector and position in the vector space.
Here data is represented in a 300 long array. That means vector space is 300 dimensional.

High-level Architecture of Weaviate

Refer to this for more details regarding the architecture. You could find information related to,

How weaviate store data
How an inverted index, a vector index, and an object store interact with each other
Learn how to combine an HNSW with an inverted index
ways to scale weaviate
planning the resources
Weaviate Modules
Role of REST and GraphQL APIs, etc.

Weaviate Schema

In order to store data and retrieve it, weaviate uses a data schema that defines the blueprint of how data is stored. It consists of classes and properties definitions and how they are related to each other. Due to Weaviate’s vectorization modules, the classes and properties you add will be automatically placed in context. read more
Once weaviate instance is started( Weaviate instance running with the text2vec-contextionary module ), using a weaviate client we can create a schema.
After that data can be imported to schema through a weaviate client.
Then using the GraphQL query, we can retrieve the data.

Next, I ll walk through the weaviate modules which enable additional functionalities to the vector native database.

Modules

Weaviate is completely modularized. The functionality of the vector native database can be enhanced by these modules. There are mainly two types of modules.

Dense Retrievers — These modules vectorize the data. Which means transforming the data into vectors. (ex: text2vec-contextionary, text2vec-transformers, text2vec-openai, multi2vec-clip )
- I encourage you to go through text2vec-contextionary to understand how the real world data entities are stored in a vector space
Reader or Generator modules — These modules add additional functionality. (ex: question answering module, text summarization module). A Reader module takes the set of relevant documents that are retrieved by the Retriever module and extracts a piece of relevant information per document. A Generator module would, on the other hand, use language generation to generate an answer from the given document

Typically a (vectorizer) module consists of two parts:

Module code for Weaviate, written in Go, hooks into specific lifecycles and provides various capabilities (like controlling the API function) to integrate the module into the regular flow in Weaviate.
- Tells the Weaviate GraphQL API that the module provides a specific nearText method.
- Validates specific configuration and schema settings and makes them available to the APIs.
- Tells Weaviate how to obtain a vector (e.g. a word or image embedding) when one is necessary (by sending an HTTP request to a third-party service, in this case, the Python application around the inference model)
Inference service is typically a containerized application that wraps an ML model with a module-specific API which is consumed by the module code executed in Weaviate.
- Provides a service that can do model inference.

Creating a custom module

Additionally, weaviate allows developers to create custom modules.
If the developer is comfortable with Golang, vector search engines, and with the design of the module, he can start creating the module.
In order to design a proper module schema, the developer should have an idea about the inference model that he would use and accordingly have to add additional arguments to the module design.
Example design related to Token classification module

{
  Get {
    Article {
      summary
      _additional {
 tokens (
   properties: ["summary"],
          limit: 2
          certainty: 0.7
 ) {
          property
          entity
          certainty
          word
          startPosition   
          endPosition   
        }
      }
    }
  }
}

The result will look like below

{
    "data": {
        "Get": {
            "Article": [
                {
                    "summary": "Shuttered cinemas led them to frantically sell their films to streamers, who were all too ready with outstretched arms and wide-open chequebooks to help bolster their booming business. Watching the splashy animated adventure The Mitchells vs the Machines on Netflix just weeks after Godzilla vs Kong jumpstarted the global box office, one can imagine that some wound-licking might soon be taking place over at Sony, where it was originally envisioned as a big-budget tentpole before being offloaded during the pandemic. It also carries with it a considerable budget, a big bet that belongs on the big screen, instead of launching online after a token one week theatrical window. Sony might be kicking themselves in the immediate future but the millions who will get to see the film on Netflix and the enthusiasm that will surely follow is a sign that its animation arm is doing something special. In the ongoing studios v streamers war, we can call this one a tie.",
                    "_additional": {
                        "tokens": [
                          {
                              "property": "summary",
                              "entity": "MISC",
                              "certainty": 0.9894614815711975,
                              "word": "The Mitchells vs the Machines",
                              "startPosition": 225,
                              "endPosition": 254
                          },
                          {
                              "property": "summary",
                              "entity": "ORG",
                              "certainty": 0.7529033422470093,
                              "word": "Netflix",
                              "startPosition": 258,
                              "endPosition": 265
                          }
                      ]
                  }
                }
            ]
        }
    }
}

Additional properties which have been applied to the module design are based on the result outputs from the inference model. Here BERT-NER has been used as the inference model. This model returns a result as below.

[
  {
    "entity_group": "PER",
    "score": 0.9964176416397095,
    "word": "Clara",
    "start": 11,
    "end": 16
  },
  {
    "entity_group": "LOC",
    "score": 0.9961979985237122,
    "word": "Berkeley",
    "start": 31,
    "end": 39
  },
  {
    "entity_group": "LOC",
    "score": 0.9990196228027344,
    "word": "California",
    "start": 41,
    "end": 51
  }
]

Hope this makes sense with the design. In the design, inside the additional property, we can see 5 extra arguments as below.

{
          entity
          certainty
          word
          start
          end
        }

which is related to the result from the inference model.

{
    "entity_group": "PER",
    "score": 0.9985478520393372, // certainty
    "word": "Sarah",
    "start": 11,
    "end": 16
  }

Refer to the below links for more details related to building a custom module.
- GitHub
- Custome Module

Use cases of weaviate

Weaviate is an industry-agnostic solution. So weaviate can be applied to all industries based on their use cases. Below you can see some of those use cases.