For the last few months I was involved with very interesting project – ElasticInbox. The goal was to build highly available email store without a single point of failure which can run on commodity hardware and scale linearly. Initially we were planning to load ~30M messages.
In past few years there’s been very interesting development in the area of data storage technologies driven by the cloud adoption – all sorts of NoSQL databases, Blob stores, etc. After researching various data storage technologies we’ve chosen Cassandra and OpenStack Object Store for our implementation.
Despite that, we designed ElasticInbox to be independent of the particular storage technology. For example, we used jclouds to provide blob store abstraction. In future, we may add HBase driver in addition to Cassandra.