MongoDB: Setting Up Advanced Text Search Without ElasticSearch

If you use MongoDB and have decided to implement text search in your application, you have surely come to realize the limitations of its "text" index.

Indeed, MongoDB gives its users the ability to perform text search on an index declared as such, with a few specificities:

• The ability to take accents into account or not
• The ability to configure it to search for simple word variations "rein" => ("rein", "reins","reine")
• The ability to omit stop-words (le, la, un, des, ...)

But when it comes to doing a partial search for a word, the results returned are sometimes too restrictive, let me explain:

Let's assume city names like ["Tours","Joué les tours", "Tournan en brie", "Vautour saint-martin"], with a simple MongoDB native text index and a search on the text "tour", the only city returned will be "Tours".

However, if I want my users to be able to find all the cities above, a simple text index is no longer enough.

Of course, I can already hear some of you shouting at me to use ElasticSearch, which is true, but for small-scale projects, I find the tool and its setup slightly "overkill".

That's why I'm going to present a solution based on MongoDB (with Mongoose), which won't scale as well as an ElasticSearch instance but works very well on reasonable data volumes!

Beware, this method only works for single-word searches for now!

The solution

Creating the new index

Elastic Search bases its indexes on what are called ngrams, i.e. multiple substrings of a string of characters. For example, the word "Tours" broken down into ngrams becomes "t to tou tour tours o ou our ours u ur urs r rs s", that is, all the possible unique combinations of adjacent letters.

So this is the first step, writing an algorithm to break words down into ngrams. Here we'll take as input a list of strings containing all the data searchable by the user, and it will return a single string containing all the possible ngrams separated by spaces.

Words.compute_ngrams = function (data) {
    let str_array = data;
    
    //We ensure ngrams unicity using a "set" object
    let ngrams = {};
    
    //Making sure the data is an array
    if (!Array.isArray(data)) {
        str_array = [data];
    }

    str_array.forEach(function (str) {
        if (str != null) {
        	//For each data, we make sure it's a string and split it by words
            (str+"").split(' ').forEach(function (word) {
                if (word && word != "") {
                    word = word.toLowerCase();
                    //Starting from the first letter, we add all ngrams of all lengths
                    for(let l_start=0; l_start <= word.length; l_start++){
                        for (let l_end=1; l_end <= word.length; l_end++) {
                            ngram = word.substr(l_start,l_end)
                            ngrams[ngram] = 1;
                            
                        }
                    }
                }
            });
        }
    });

    return Object.keys(ngrams).join(" ");
}

Now that we're able to transform all the desired data into a single string containing all the ngrams, we're going to need an index in which to store it.

let Schema = new mongoose.Schema({
    //...
    text_search: { type: String, required: true, default: ""}
});

Schema.index({ text_search: "text"});

To do this we'll use a classic MongoDB text index so that it can treat each ngram in the string as a full-fledged word.

Do you see the trick? MongoDB can only handle whole words? So we literally "chew" the work for it by giving it already broken-down words that it will treat as whole words!

Then all we need to do is regenerate the ngrams for the desired data—here city name, postal code and region—every time an object is created or modified, and save it to the database.

data.text_search = Words.compute_ngrams([
    data.city_name,
    data.post_code,
    data.region
]);

Performing a search

The text search used is MongoDB's, which is case- and accent-insensitive by default.

let results = await Schema.find({
    $text: {
        $search: '"'+search+'"', $language: "none"
    }
});

However, it is necessary to specify $language: "none" to prevent MongoDB from removing all stopwords, since many of our ngrams are 1- to 3-character strings, which are often treated as stopwords.

We also wrap the query in double quotes to prevent MongoDB from combining ngrams from several different words.

And there you have it, job done—you can now perform partial string searches, at the beginning, end, or middle of a word! Note that the size of the index will vary greatly depending on the number of documents and the amount of data you use to generate your ngrams!

But it works!