e1000 InterruptThrottleRate

The e1000 Nic is very common and can be found in many servers.  During my last NBD cluster engagement, on a set of 5 Dell 1950 servers equipped with a quad e1000 Nic, we were trying to tune cluster performance and one of the thing that was disapointing was that the system time, to processed interrupts from the Nics, was always low.  Since neither the ndbd processes or the mysqld processes were using a significant amount of CPU, we were wondering where the bottleneck was.

Then, Michel Donais, from the client, read the Linux driver documentation and bingo!!! Those e1000 Nics throttle interrupts by default at 8000/s.  8000… it is a big limiting factor….   design to prevent DOS attacks.  We removed the throttling and… reach more than 30,000 interrupt/s and guess what, the number of transaction per second scaled with it along with the ndbd and mysqld cpu usage.  Interesting isn’t it? Then I remembered another recent gig on Sun T2000 using e1000 Nics…

This engagement was a very frustrating one,  nothing was busy and the performance level was microscopic.  In despair, I even tried to switch from GB ethernet to 100 MB ethernet with no change!!!  This pointed out to a very slow processing of network packets.  There was a lot of similitude between the Dell servers and the T2000s, same type of unexplainable saturation, just a different level.  From the experience of last week, I did some research and found that the InterruptThrottleRate is activated on T2000 and… set by default to 550.  Is it possible that many performance issues of NDB cluster on the T2000 be caused by that?  The e1000 is fairly common with Sun servers, I don’t know if the other common Nics have a similar feature.

For more information:  http://docs.sun.com/app/docs/doc/819-6990/ggtvv?a=view

About Yves Trudeau

I work as a senior consultant in the MySQL professional services team at Sun. My main areas of expertise are DRBD/Heartbeat and NDB Cluster. I am also involved in the WaffleGrid project.
This entry was posted in linux, mysql, NDB Cluster, yves. Bookmark the permalink.

5 Responses to e1000 InterruptThrottleRate

  1. Stoner says:

    For those interested, here’s the Intel page that documents the e1000 driver settings.

    http://support.intel.com/support/network/sb/CS-009209.htm

  2. Antony T Curtis says:

    It would be interesting to try out ndb on FreeBSD which has a mixed mode interrupt+polling to protect against interrupt livelock (what the throttle attempts to mitigate) yet is supposedly able to get great throughput on 10gE.
    Could be worth checking out, although I recall that polled mode is not enabled in the stock kernel – needs a kernel+modules rebuild.

  3. Michel says:

    To keep people updated, I went even better with this.

    I rebuilt the igb driver, using the IOATDMA system. I also added the CONFIG_IGB_RX_MQ and the CONFIG_IGB_SEPARATE_TX_HANDLER system.

    Then, (at the time), we removed IRQBalance, as the tool tend to balance IRQs according to themselves, not to the CPU affinity of applications, and now I went one up and applied one different CPU for every network card’s interrupt. One for RX, one for TX. When doing heavy system usage, I get up to 10-12% individual CPU usage in hardware interrupts, and up to 90% CPU usage in software interrupts, both in TX and in RX.

    I’m able to completely spam the system in pings (up to 60K/second) while keeping latency at bay (0,035-0,050ms ping time)

    Otherwise if not doing that, the system will aggregate the interrupts together, which is nice CPU-wise, but will account for a larger delay (up to 0,25-0,5ms during heavy usage).

    Now if someone could tell me how to edit the CMOS registers of a PowerEdge 1950 to enable Intel’s DCA (Direct Cache Access), I would be happy. Right now it’s not enabled, and 1 hour discussion with a Dell technician ended up in me stopping the call (ending the suffering might be a better term), since it was clear the technician had no way to correct this, and he couldn’t hang up without me having my technical problem resolved, so at the end it was long interminable pauses, and he mainly took the last half-hour to try to stomp me to see if it was a selling point, if the MB, CPU, Netcard accepted it, and so on.

  4. Michel says:

    One last update on this topic… I finally was able to access Intel DCA on my computer. That’s a totally odd ball:
    - The CPU (E5240) does support DCA although Intel marketing info doesn’t say so.
    - The Motherboard and BIOS (Intel-based) supports DCA, although the BIOS doesn’t say so. In fact, there’s no mention of DCA anywhere.
    - The network card (Intel e1000-based) supports DCA, by compiling a new driver version.
    - The “CPUInfo” doesn’t show DCA in its features, although I know it’s supported.

    By forcing (not only coercing, I mean thumping the byte directly inside the CPU configuration until it internally bleeds) the DCA active using some MSR tool, I was able to activate DCA on the BIOS, meaning the driver was able to use it.

    Stability wasn’t affected, and interrupt rate seems to at least be par with what I had before the hack. I have not made any benchmarks these days so I cannot tell what’s the system’s reaction. But from my standpoint, it cannot be worse than before.

    That’s it for this topic for me :) Next!

  5. Yves Trudeau says:

    Wow Michel, great job!