Darn Ec2 Latency!


———
MEMCACHED
———
Memcached puts    141075
Memcached hits    122696
Memcached misses  18329
Memcached Prefix:  29298
Memcached Get Total Lat 12222 (us)
Memcached Get Recent Lat 10925 (us)
Memcached Miss Total Lat 8490 (us)
Memcached Miss Recent Lat 5740 (us)
Memcached Set Total Lat 142 (us)
Memcached Set Recent Lat 65 (us)

12ms?  Really, I mean come on…. what are we using 10baseT?

at least it appears somewhat consistent.  Consistently bad, but consistent… and if you can not survive the odd spike to 35ms, what good are you.

This entry was posted in mysql, performance, rant, Waffle Grid. Bookmark the permalink.

5 Responses to Darn Ec2 Latency!

  1. TCN says:

    I understand network latency and that there are periods where there can be uncontrolled performance defects.

    However, in order to make the statement you’ve blogged. I believe that further stats are required. You simply do not provide enough information about circumstance and environment in your post to disregard this as “bad” other than human configuration (or lack there of).

    Come on. Break it out dewd.

  2. matt says:

    Latency was generated using waffle 0.4, running dbt2 & sysbench, with a maximum of 16 threads with innodb conncurrency set to 4… so their should have been no more then 4 active gets at any 1 point. These machine were “small” ec2 instances. I was talking to memcached over the private interface. Tests ran over a 6 or 7 hour period of time. Latency in this case does include the memcached “lookup” time. All data being sent is 16K ish in size.

    The exact same test, with the same parms and config on dedicated hardware yeild latency numbers in the 4ms range for 100Mb/s, 1.2 ms range for 1gbe, and 300us for infini.

  3. Kevin Burton says:

    Hey,

    Are you using get() or getMulti() ? You need to be running on getMulti since it can batch up 1k gets() into a single getMulti() and hence use one frame to reduce latency.

    For large result sets, this might seriously boost your performance.

    Of course, you might already be using getMulti :)

  4. matt says:

    Unfortunately we have to use get in the function we are using ( it retrieves 1 block from disk generally ). We are looking at multi-get in the read-ahead but that’s a small % of the reads comparatively. I was thinking about trying to move the memcached code up a function, but that has some risks… still pondering though.

  5. Pingback: Big DBA Head! - Database Brain Power! » WaffleGrid new features