Database: Differences between replication and fragmentation (sharding)

If you've ever dived into database documentation (whether relational or non-relational), you've surely already come across these two terms: "replication" and "sharding"

In French, "sharding" would be translated as "fragmentation," but what does that really mean?

What is it for?

The main reason for choosing these data storage strategies is to implement "horizontal scaling."

For most computer systems, it will be cheaper to invest in two small machines with performance P1 than in one big machine twice as powerful P2.

It's the same problem for databases: when your application grows to the point that the system starts to slow down, it's time to stop making the machine bigger (vertical scaling) and to start splitting the load across multiple machines (horizontal scaling).

Method #1: Replication

This strategy simply consists of having a certain number of machines work together (let's say 3 for the example), promoting one to the rank of primary and the others to the rank of secondaries.

Note that it's still common to come across the terms "master" and "slaves" to designate these two roles.

When new data is saved on the primary server, the latter will propagate this new information to the other machines so that they "replicate" the new data schema.

Replication is therefore simply an intelligent copy of the data.

The advantage is having several different (but synchronized) machines from which your application can read data, which allows it to support more read load.

However, the drawback is having to go through the primary server to write or modify data; the writing process itself doesn't become more efficient, but in general the load on your system mainly comes from the number of concurrent reads.

Moreover, if the main server goes down, you'll have to wait for the secondaries to elect a new primary before being able to write again, which can take some time.

Method #2: Sharding

Here there is no hierarchical relationship between the machines, they are all on equal footing and each holds part of the data.

Let's take a directory as an example: machine 1 would hold names from A to I, machine 2 from J to R, and machine 3 from S to Z.

Here we improve performance in both reading AND writing since each machine is responsible for its own data. Be careful though, the sharding needs to be well thought out because if all the most popular data ends up on the same machine, it risks bearing more load than the others.

But the real weak point of this approach is that if one of the machines goes down, an entire portion of the data becomes inaccessible, even for reading!

Method 3: Hybrid

The third method, the most costly but also the most robust, consists of first sharding your data, then replicating each shard.

Potentially, this also means having 9 machines in total (in our example), which makes for a much more complex architecture!

Conclusion

To choose the best solution for your project, you need to know what matters most to you: availability, consistency, or partition tolerance, knowing that you can only choose two of these options.

This principle of distributed databases is called the CAP theorem, which I'll let you discover if the topic interests you!