Introducing MiniWheatFS

It is well known that MySQL can use a lot of temporary files and that, using a ramdisk (tmpfs on Linux) can improve performance drastically as Matt has shown here. The problem with tmpfs is its limited size. You need just one query creating a huge temporary file that happens once per day to screw up the tmpfs scenario. I have done some fuse programming in my previous job and I think it is possible to use fuse to mix a tmpfs filesystem with a regular filesystem, the regular filesystem being used only when the tmpfs is full. Fuse has a lower performance level than a regular filesystem but I think the possibility to use tmpfs the vast majority of the queries needing a temporary file will overcome the performance limitation of fuse.

That is the explanation for the MiniWheatFS name, the frosted side is the tmpfs and the healthy side is the regular filesystem. There are many fuse implementations available and I am currently debating (for now with myself) which one is the most appropriate starting point. The main candidates I have are unionfs-fuse and virtualfs-c++. The project is hosted on Launchpad at https://launchpad.net/miniwheatfs but I haven’t pushed any code yet.

If somebody is interested to give a hand in the project, just let me know, I think it can be very interesting and useful.

About Yves Trudeau

I work as a senior consultant in the MySQL professional services team at Sun. My main areas of expertise are DRBD/Heartbeat and NDB Cluster. I am also involved in the WaffleGrid project.
This entry was posted in linux, mysql, performance, yves. Bookmark the permalink.

3 Responses to Introducing MiniWheatFS

  1. This is interesting point – tmpfs itself is backed up by swap (you can specify amount of RAM to use) – what is the benefit of using traditional file system instead of allocating high amount of swap space say 100GB and letting tmpfs to use it.

    Though I’m not sure how efficient tmpfs is in such condition and I’d look forward to comparison of tmpfs to with fuse to such configuration

  2. Yves Trudeau says:

    The use of swap to extend tmpfs is interesting and will worth to be compared. I do suspect the performance to be lower and swapping could cause other performance impacts on MySQL. Anyway, it will probably be one of my first comparison benchmark.

  3. Pingback: Big DBA Head! - Database Brain Power! » The early end of the MiniWheatFS project for tmpfs with MySQL!!!