RAM-Based Data Model

Background

Writing an reading from disk has proven to be too slow for any reasonable load given our need to process continuous and concurrent data streams. Our backend architecture uses a Ram-based data model. We utilize distributed memory based on memcached. We have found that placing data in memcached and subsequently writing data to disk asynchronously is a viable solution.

Writing data to disk sequentially is much faster than writing a large number of small files in random order.  The same is true for reads.  Random access to data on disk simply does not scale to large data models.  The amount of data collected for a viable server monitor is fairly large, moreover the application monitor data is much larger.

Our requirements for a persistence solution include:

• Replication – It’s not feasible to do traditional backups when the data is quite large.
• Fault Tolerance – When a server in the storage cluster goes down we should not lose data or access to the cluster.
• Dynamic Expansion – We want the ability to add servers to a storage cluster without needing to restart.

Research Results

We have looked at numerous technologies for a persistence solution. We defined a set of tests that are meant to more or less model our production data models. In simple terms we write 1000 100KB files from 10 threads, read them back and validate the data. This is followed by 100,000 file writes of 100KB from 100 threads and read/validate. We timed the 1000 file tests. The purpose of the 100,000 files test is a basic load validation, less a timing exercise. Here are the results:

                                    Writes                     Reads

1. NFS                  115-200 secs       8 secs
2. MogileFS          125-130 secs       52-55 secs
3. MongoDB         40-45 secs           40 secs
4. Lustre              70-75 secs           14 secs
5. Membase         74-76 secs           50 secs
6. Cassandra        50-80 secs           50-80 secs

Things We Learned

We built an async write capability using memcached, a queue server, a write process and NFS. We started by looking at distributed file systems as an NFS replacement. We discovered that technologies like Membase and Cassandra combine a RAM based data model with async writes to a backing store that uses distributed storage. As we gained experience with Membase and Cassandra we realized that we didn’t necessarily need to build our own version of RAM storage with async backing store on a distributed file system.

We found that NFS breaks down under load. We discovered that the client pauses for a few seconds under heavy write load. There is evidence, much of it anecdotal, that NFS has issues under heavy load. We wanted to get first hand experience.

It’s clear from the data that NFS and Lustre reads are significantly faster than any other technology. We found that because these are kernel mode file systems, as opposed to user mode file systems, such as those using FUSE, for example, they exhibit much better read performance. This is due to the use of the file buffer cache managed by the kernel. This also made it clear that our tests were not at all representative of random access. Therefore, the use of the kernel’s buffer cache produced dramatic differences. We decided not to make an attempt to create realistic random file access.

In terms of a distributed file system it is clear to us that Lustre is far and away more reliable and scalable. Administration of a Lustre deployment is no more difficult than others and is far more predictable.

Conclusions

While Lustre is almost certainly the more stable and scalable technology, we have a few issues with deploying it in production; we need kernel modules on all clients and we need not only kernel modules but also a patched kernel on each server in the storage cluster. Moreover, we need to provide a RAM based solution with async backing store in front of Lustre.

We are moving forward to further evaluate the use of Membase or Cassandra. Both have a RAM based architecture and a viable distributed storage model. We will perform tests of replication, failover and dynamic expansion.

Stay tuned, we’ll fill you in on the results and our final decision.