Intel X-25e and Mysql Part 1b: Don’t let your Drive Over Eat!

The plan was only to do two quick posts on RAID Performance on the X-25e, but this was compelling enough to post on it’s own.  So in part I Mark Callaghan asked hey what gives with  the SLC Intel’s single drive random write performance,  It’s  lower then the MLC drive.   To be completely honest with you I had overlooked it, after all I was focusing on RAID performance.  This was  my mistake because this is actually caused by one of the Achilles heals of most flash on the market today, crappy performance when you fill more of the drive.  I don’t really know what the official title for it is but I will call it “Drive Overeating”.

Let me try and put this simply:  a quick trick most vendors use to push better random write #’s and help wear leveling is to not fully erase blocks and use free space on the drive as a place where they can basically “write random blocks sequentially”.  They can then free the deleted blocks later on.  When the drive becomes overly full ( Overeats), performance starts  to degrade as blocks must be fully erased.   So as you get closer to capacity drive speeds will slow.   I had heard that intel reserves some flash that is not accessible to help boost performance ( for example 32GB is available, but an additional XGB is hidden for these operations ) but I have not seen this in print anywhere.  If someone has a link on this I would appreciate it.

Anyways, what you want to see consistent #’s from various size datasets in fileio when using flash.  We maybe adjusting the size, but the threads + the # of requests is going to stay the same…   so theoretically a 32thread test with 2 million requests on a 20GB should produce similar #’s to a 40GB test with the same setup  ( unlike regular disk we do not need to move around to find the random block ).  Now their is always special circumstances, i.e.  cache, external processes, etc that can influence the numbers slightly, but the goal is consistancy.  Take a look at the earlier Violin SSD benchmarks the 100% writes ( 0% on the graph ) 20Gb is very close to the 360GB in terms of performance.   Now lets take a closer look at the X-25e:

Req ps
Run #1 – 12G 3887.52
Run #2 – 12G 3837.52
Run #3 – 12G 3894.7
Run #1 – 20G 2943.68
Run #2 – 20G 2749.56
Run #3 – 20G 2745.14
Run #1 – 28G 2149.56
Run #2 – 28G 1966.99
Run #3 – 28G 1956.95

You can see here that running the fileio test with a 12G dataset actually is nearly 2x faster then running the same test over 28G.   Further let’s see what happens if you run a 12G test directly after a 28G test finishes:

Req ps
12G – After a 28G 1960.67
12G – After a 12G 3894.7

Once again we double the performance of the test. So heavy write loads are going to suffer a substantial performance dip. This is going to also slowly degrade over time as the background process struggles to keep up.  I put several 28G tests and ran them back to back to try and see if their was some point where I stabilized:

Even after 12 test’s we are still dropping, although its a much slower drop then when we started out.  Bottom line here,  I think a optimal performance best  practice for using these drives is to reserve some amount of space when creating the partition to prevent the Overeating effect from hampering performance.   Depending on your wrokload this maybe as high as 50% of the total drive, which then makes you really recalculate the $/gb & $/io costs.

This entry was posted in benchmark, hardware, Matt, mysql. Bookmark the permalink.

6 Responses to Intel X-25e and Mysql Part 1b: Don’t let your Drive Over Eat!

  1. matt says:

    FYI, I am also going to run 10-20 12G tests, because its also possible and likely that “saving” space to boost performance will only stave off the over eating effect.

  2. Great post Matt. One thing I’ve always been a skeptic when I see marketing comparisons of SSD to Hard Drives on a $/GB basis.

    It would be great to see those figures again on a 50% fill factor ;) I wonder if consumer demand will make the benchmarks/metrics used changed.

  3. Storage guys are used to sacrificing 30% of space to reduce seek latency and keep data on the outside of the disk, but 50% is a bit obscene.

  4. Pingback: x-25e, 25% reduction in random writes… » Big DBA Head!

  5. famzah says:

    This is a known issue:
    http://www.pcper.com/article.php?aid=669&type=expert&pid=1

    Intel released a new firmware:
    http://support.intel.com/support/ssdc/index_update.htm

    Which according to the author of the above article resolves the problem:
    http://www.pcper.com/article.php?aid=691

    • matt says:

      Famzah,

      Different drive different issue. The fragmentation issue is an issue on the MLC drives not the SLC drives. The x-25e is different from the x-25m. With the fragmentation issue your fragmentation does not simply go away when you let the drive sit idle for a minute. This has to do with the amount you put on the drive and the background process not keeping up.