Waffle Grid: Testing Waffle With Dolphin Interconnects

Let’s grab your attention with a shameless marketing ploy:  How would you like up to a 14X performance boost in your Mysql database performance? Now your hooked who wouldn’t want a 14x boost?  Well we got that using Dolphin interconnects and Waffle Grid! Read on!

As you know over the last few weeks the good folks over at Dolphin Interconnect Solutions have lent me use of a couple of servers with their interconnects installed ( They also sent me a couple of cards to test with, but I ended up not being able to secure the hardware to put these through their paces at my home office ) .  I ran into several challenges in the testing, and we also found out a lot about Waffle Grid and where we have some open bugs ( its good to test Waffle at the fringe of performance and see what breaks).

The oddest thing I ran into here was the way using the Dolphin cards dramatically changed the characteristics of the tests I was running.  As mentioned I was seeing 3x the number of transactions but it was also pushing 3-4x less memcached traffic then the slower 1gbe tests were doing.  Well I resolved this, basically it was boosting the performance of the Network between dbt2 & the mysql client, which is were the biggest boost came from.  I am putting together a separate post on this.   Having resolved that I was able to adjust the test to boost only the traffic between the mysqld and memcached.  The results in dbt2 were the 1gbe connection scored an almost identical score to the dolphin test.

Here is the DBT2 chart over a 200 minute test between the two:

As you can see their is little discernible from the Dolphin & the 1gbe in these tests.   This is a far cry from the tests that were showing 3x performance improvement.  So the question really is why are they the Same?  Well a little digging shows that our disk was not able to keep up with flushing pages out of buffer pool, so here the extra speed offered by the dolphin cards was negated by a disk bottleneck.  This is a good lesson for those who are thinking about deploying Waffle Grid, Waffle is not going to get rid of, or directly reduce write bottlenecks.  By eliminating reads from disk however it may indirectly help write performance, but Waffle was not built to directly address disk write performance.

So In order to flex the Dolphin cards, I really need to get rid of writes on these boxes.  So I turned to sysbench.  I loaded a table with 100 million records and tested with the OLTP read-only test at 3 different transaction levels.  The first test warmed the cache by completing 500K transactions. I then ran a test @ 500K transactions again, and then a test with 1 million transactions.  I wanted the cache warm for each of these tests so I did not restart.  Here are the results:

The warm up was fairly consistent between the two interconnects which is to be expected as the first 500K transactions will be impacted heavily by disk access.  The second 500K transactions look great.  The dolphin cards boosted the transactions per second from about 900 to around 1440, which is a significant boost.  Over 1 million transactions the 1gbe falls from 900 TPS to about 780 while the dolphin enabled tests get better jumping from 1440 to around 1500 TPS.  This makes sense, as each memcached get on the Dolphin cards is on average 800 micro seconds faster then the 1bge connection.  As more transactions come in the difference between the two tests cumulative total time will widen.

For those who want to know how much better this test was over non-waffle (sysbench tps), a reran the same test later:

No Waffle 1Gbe Dolphin
117 tps 799 tps 1487 tps

The latency from the Dolphin cards is awesome compared to 1gbe latency.  The latency for the dolphin cards under load  is 3-4x  faster then the 1gbe latency.  Generally we are never above 300 microseconds, while the 1Gbe cards tend to be around 1100 microseconds.  This can really add up.

For those looking for the ultimate high performance MySQL/Waffle Grid deployment using high speed interconnects is going to be key in order to acheive the highest possible performance boost.  That being said, The Dolphin interconnects perform as advertised and certainly boost the network performance substantially.  These cards will make a worthy addition to any high end Waffle Grid deployment.

This entry was posted in HA, hardware, mysql, performance, Waffle Grid. Bookmark the permalink.

One Response to Waffle Grid: Testing Waffle With Dolphin Interconnects

  1. Pingback: Big DBA Head! - Database Brain Power!