Archive for December, 2008

Hard Loading – something to avoid.

Friday, December 19th, 2008

Last week I got a question about sharding using our Spockproxy.  The question was how can I create a query for the proxy so it effectively runs:

/*in shard 1*/
SELECT * FROM table_a WHERE f_key IN (a, b, c);


/*in shard 2*/
SELECT * FROM table_a WHERE f_key IN (d, e, f);

By design our proxy will not do this.  The whole point is to hide the sharding from the application.  Given a query it will either send the same query to all the shards and combine the results or only send that query to one shard when it can figure out that the results(s) can only come from one shard (because you specified the shard key in the where clause).

I did figure out a way it could be done using views but would this ever be desirable?  

Like “Hard Coding” where values are built into the code of your application I’ll call this technique “Hard Loading” where an attribute’s value (other than the shard key) is reflected by the specific shard the data is found in.  At first glance this might seem efficient – but like hard coding I recommend against it.  Simply growing your system to include a new shard will break things.  Besides rearranging your data you’ll have to examine all of your queries – which ones assume data lives in a particular shard?  How can you fix them?  You find yourself in a position where because adding a shard is so difficult you are actually praying that you do not grow.  You never want to be in that position.

Slides from Proxy talk

Monday, December 8th, 2008

I’ve reposted the slides from my Spockproxy talk (Spockproxy is a Sharding only version of the MySQL proxy).  Since I’ve have to move this web site it’s been some effort to move all of the files into their new homes.

These slides are in a variety of formats and have loads of great information if you’re considering a sharding solution – even if that is not Spockproxy.  Of course once you see how easy it is you’ll put it on your short list.

The slides are available at  If you want to hear the talk you’ll have to invite me.


Sunday, December 7th, 2008

Spockproxy is a custom version of the MySQL Proxy that is optimized (dedicated) to range based sharding.  This is not done with Lua so it is perhaps less flexible but if you are trying to do sharding it is one of the easiest solutions to set up – and will require minimal, if any, changes to your application code.

Is is available as an open source down load at Sourceforge.