Apache Cassandra

Apache Cassandra is software that presents a database table without the database. There is a fashion for replacing database software with software that does part of what database software does and claiming a speed improvement from the result. Apache Cassandra is one of the options and presents a single database table with some of the overheads of a database replaced by your code.

There are several alternatives to databases that are effectively indexed tables sitting in isolation. You have to write your own code to do what databases do. On rare occasions this manual approach is faster. On most occasions tuning your database will give you a better speed increase faster than rewriting your code. On some occasions no amount of rewriting your code and tables manually will match the performance of a database with hundreds of person years invested in development.

Apache Cassandra has a lot of people time tied up in development and usage but explained successes are rare. There are cases where people claim better performance from using Apache Cassandra but cannot explain what they changes or why it works. A real problem is the lack of a comparison to the same time spent tuning their original database or, in the case of MySQL, using a different database engine.

Two billion rows

I read a brag page about how Apache Cassandra handled a large amount of data on one table compared to the previous software used by the bragger. The Apache Cassandra specifications, or one of their case studies, said the table can handle two billion rows. A visitor made the comment that the two billion limit refers to columns, not rows. The introductory articles at the Cassandra Web site mention two billion columns and unlimited rows.

Two billion is the maximum positive value of a 32 bit signed integer. MySQL can use unsigned 32 bit integers as an index and a 32 bit unsigned integer can store four billion. Over the many years I have works on databases, there were several occasions when one of my applications hit four billion rows and I restructured a table or the table processing to reduce the number of rows. Two billion is not a big number. Twenty billion is not a big number.
One hundred billion is a big number for rows in a database table based on comments by people about their biggest tables.

In some occasions people are using PostgreSQL table partitioning to spread large tables over several servers. Looking through the Cassandra white papers, Cassandra users are doing the equivalent of partitioning PostgreSQL tables. There is a lot of flexibility in existing databases for handling large numbers of rows without switching to Cassandra.

Latest update

I am writing about Apache Cassandra 0.7 from cassandra.apache.org. Prior to 0.7, Apache Cassandra, like many other no database tables, bragged about super performance and the lack of database indexes to slow things down. Apache Cassandra 0.7 brags about finally having full indexes. I think that makes Apache Cassandra 0.7 the same as a normal database.

The slowest database tables are slow because they have flexible indexes, row level locking, and data schemas. The schemas let you change the layout of a table row without taking your Web site offline. Row level locking lets you update part of a database table without stopping access to other parts of the table. Indexes give you faster selections at the expense of slower updates. Exactly the same things happen when you make the same facilities available for the table based software including Apache Cassandra.

Proven?

Cassandra is in use at Digg, Facebook, Twitter, Reddit, Rackspace, Cloudkick, Cisco, SimpleGeo, Ooyala, OpenX, and more companies that have large, active data sets. The largest production cluster has over 100 TB of data in over 150 machines.

Sounds impressive. All those big sites using Cassandra but how do they actually use them? Look through the articles at wiki.apache.org/cassandra/ArticlesAndPresentations.

Consistency

A common thread when talking about Cassandra is consistency. Several case studies and blogs contain lines similar to the following from Digg. The customer was happy to give up consistency in return for performance. Hey, you can get better performance out of relational databases by giving up consistency. Why switch to something else?
Since it was already necessary to abandon data normalization and consistency to make these approaches work, we felt comfortable looking at more exotic, non-relational data stores.

Why is consistency important? When can you throw it out? Consistency is vital in a shopping site when recording stock quantities and money paid, a mistake costs you money and drives away a customer. Consistency is not important when displaying user ratings of products. When there are a lot of users rating a product, it does not matter if the current rating is 4.5, you can display the rating of 4 from 10 minutes ago. Ratings can be calculated using an irregular process running in the background and can distribute the updated ratings slowly across servers. You can achieve exactly that type of split processing while staying with MySQL and PostgreSQL. You would not switch to Cassandra just for that type of change. You would choose Cassandra only when you want to combine inconsistency with some other features of Cassandra.

Data normalisation

Cassandra is often used to replace normalised data with unnormalised data to improve retrieval speed. Look again at the line from Digg. They abandoned data normalisation.
Since it was already necessary to abandon data normalization and consistency to make these approaches work, we felt comfortable looking at more exotic, non-relational data stores.

Data normalisation gives you accurate data. Denormalisation produces inaccurate data. A common approach with any database is to create original data normalised then create copies of the data denormalised for faster retrieval. You do not need Cassandra just for this feature. The example mentioned in the Digg blog connects user 1 to users 2, 3, and 4 through table A then connects user 2, 3, and 4 to other users through table B. Cassandra is effectively used to preselect the list and store the list for future use. You can do that with any database. MySQL MyISAM tables are a good choice because they have low read overheads. Cassandra may offer an advantage when they prebuilt lists are very large.

Everything in memory

Cassandra users often mention they store everything in memory. Several database products perform better when they have lots of memory. The database use of memory often depends on initial settings. Looking at the switch to Cassandra mentioned in some blogs and white papers, they did the equivalent to increasing the server memory by a huge amount, altering the database settings to use all the extra memory, then monitoring the result in fine detail for a long time to refine the settings. You can make MySQL and some other databases ten or more times faster with the same approach. This is something worth doing before rewriting your application to use new database software.

Alternatives

Digg switched to Cassandra after looking at several alternatives:
After considering HBase, Hypertable, Cassandra, Tokyo Cabinet/Tyrant, Voldemort, and Dynomite, we settled on Cassandra.

Conclusion

Apache Cassandra offers the same trade off as the alternatives. You get less function than a database and you write your own code to perform the work previously performed by a database. Database tuning will initially give you bigger gains than replacing database code with your code. In some rare instances your code might be more efficiant than the database code and worth the massive investment you need to replace database tables with discrete tables of the Apache Cassandra type.