Comments on: High performance replacement of the MySQL Memory storage engine with NDB

By: Yves Trudeau

Yves Trudeau — Mon, 16 Mar 2009 14:57:17 +0000

Johan, I obviously missed your post… Mine is indeed very similar.

By: Johan Andersson

Johan Andersson — Mon, 16 Mar 2009 14:41:20 +0000

matt, sure you have replicas. With Cluster you always have (unless you have set NoOfReplicas=2). We have implemented this with quite a few customers using Cluster for session management (but then we have also stored a “shard catalog/index” on Cluster as well):

http://johanandersson.blogspot.com/2008/12/cluster-session-management-tips-and.html

BR
johan

By: Mark Callaghan

Mark Callaghan — Sat, 14 Mar 2009 03:52:37 +0000

My preference is for the query cache and the HEAP engine to be fixed to scale on SMP. As you mention, there is a lot of value to having the data in the database. Any data subject to audit controls (SOX, HIPPA, PCI) requires access control and keeping it in a database table or the query cache provides that. Also, if the data is kept in the query cache or a HEAP table, then you don’t have the overhead of managing separate memcached processes.

I don’t think too much work is required to fix both the HEAP engine and the query cache. But I have not heard of anyone doing this. Someone? Anyone? From the outside looking in MySQL seems to be investing much more energy in memcached than in fixing these problems.

By: matt

matt — Sat, 14 Mar 2009 01:45:21 +0000

Yves correct me if I am wrong, but you can still have replicas built in here as well. This would provide some fault tolerance for this solution. So even if a data node goes done your data/database is still up.

Mark I agree memcached is sooooo much simplier. But their is a lot of memory database love out their, not 100% sure why. But I have some customers who refuse to use memcached.

Additionally this solution does offer a few other features not found in memcached ( security, the ability to search on multiple keys, the ability to join, etc. )… so their maybe a place for this.

By: Mark Callaghan

Mark Callaghan — Sat, 14 Mar 2009 00:41:31 +0000

I am sure that performance with NDB is awesome. But memcached is much simpler. And for this use case, most of the awesome-ness of NDB is not being used but you still get the complexity.

By: Yves Trudeau

Yves Trudeau — Fri, 13 Mar 2009 20:57:00 +0000

The problem with NDB cluster is that a global checkpoint GCP is needed for a create/alter table. Normally, there is a GCP at each 2 seconds, so a creating and dropping a temporary table can take up to 4 seconds. See http://www.bigdbahead.com/?p=204 for another way of handling efficiently temporary files.

By: Antony Curtis

Antony Curtis — Fri, 13 Mar 2009 20:38:29 +0000

This is interesting….

Perhaps the NDB team can create a new handler class for as a replacement to Memory so that MySQL could use it for all it’s temporary tables?

By: Yves Trudeau

Yves Trudeau — Fri, 13 Mar 2009 19:16:24 +0000

Roland: Unfortunately, many recent NDB features are not documented. I learned about this one from Johan.

Mark: It depends some people don’t want to add another moving parts. NDB can be a full database, not only be use for web session. Other tables can be persistent on disk. Performance wise, NDB, at least with the NDB API, is almost on par with memcached.

By: Mark Callaghan

Mark Callaghan — Fri, 13 Mar 2009 19:07:13 +0000

So you think people will be better off running NDB rather than memcached for this?

By: Roland Bouman

Roland Bouman — Fri, 13 Mar 2009 18:51:49 +0000

Nice tip!

never heard of this variable…I should have…is it possible it has been documented only recently?

kind regards,

Roland