Dell MD3200i + VMware ESX hanging issues

The Dell MD3200i SAN is a cheap(ish) effective storage device certified for VMware ESX.
I had no issues using it with a single ESX host, but when trying to use it with more than one host I was seeing hangs of the ESX servers, loss of connectivity from hosts to the VirtualCenter and strange logs saying that it couldn't communicate with the LUN.

Mar 24 19:56:11 virt08 vobd: Mar 24 19:56:11.295: 797333253337us: [esx.problem.vmfs.heartbeat.timedout] 4eb93a45-ea456b83-4c0a-0010189da888 disk03-1.

Mar 24 19:57:36 virt08 vobd: Mar 24 19:57:36.699: 797418651679us: [esx.problem.vmfs.heartbeat.recovered] 4eb93a45-ea456b83-4c0a-0010189da888 disk03-1.

After much going back and forth with Dell ensuring that I had the network configured with their latest recommendations (they keep changing it), they took al look at the stats on the network card of the MD3200i


========================================================================
                          TCP PORT STATISTICS - PORT 0                
========================================================================
  TCP Received Segments          0xf6f28bd
  TCP Tx Segments                0xd773310
  TCP Rx Segments In Error       0x0
  TCP Tx Byte Count              0x1a167dd174
  TCP Rx Byte Count              0x2683bbc3
  TCP reTx Timer Expired Count   0x584
  TCP Rx Dup ACK Count           0xa7890
  TCP RX ACK Count               0xee2915
  TCP Rx Delayed Ack Count       0x5a4256
  TCP Tx Ack Count               0x65098dc
  TCP Rx Seg Out Of Order Count  0x3242c
  TCP Rx Window Probe Count      0x0
  TCP Rx Window Update Count     0x49c


Here they saw high numbers of ACK issues (approximately 10% of all network transactions).

This prompted them to ask that I set up the IO queue depth and a delayed ACK workaround from VMware.

This immediately fixed my issue!

To change IO queue depth

esxcli nmp roundrobin setconfig --type "iops" --iops 1 --device <device UID>

Delayed ACK Workaround

Disabling Delayed ACK in ESX/ESXi 4.x and ESXi 5.0.x

1. Log in to the vSphere Client and select the host.
2. Navigate to the Configuration tab.
3. Select Storage Adapters.
4. Select the iSCSI vmhba to be modified.
5. Click Properties.
6. Modify the delayed ACK setting, using the option that best matches your site's needs:
   • Modify the delayed ACK setting on a discovery address (recommended):
     1. On a discovery address, select the Dynamic Discovery tab.
     2. Select the Server Address tab.
     3. Click Settings.
     4. Click Advanced.
   • Modify the delayed ACK setting on a specific target:
     1. Static Discovery tab.
     2. Select the target.
     3. Click Settings.
     4. Click Advanced.
   • Modify the delayed ACK setting globally:
     1. Select the General tab.
     2. Click Advanced.
7. In the Advanced Settings dialog box, scroll down to the delayed ACK setting.
8. Uncheck Inherit From parent.
9. Uncheck DelayedAck.
10. Reboot the host

This change should be safe to apply to all MD3xxx SANs, so you could just do the global change.

You should now have a fully working system

I hope this page helps you fix your problems, please click on an advert to show your appreciation.

Latitude E Series secondary hard disk

If you have a Dell Latitude E Series laptop you'll probably be very happy with the device (especially if you get the backlit keyboard which allows you to pretend to switch between typing and watching the TV like I do).

As an IT professional though, you may well have come across the problem of re-imaging these systems, because you need somewhere to store your Ghost or True Image files. Sure you can store them on the network, but a secondary storage device is so much quicker (and if your company is anything like mine, the first you hear about a new joiner is when someone stands at your desk and introduces themselves).

...and so to the Latitude E.

Dell can supply a Media Module to give you a second SATA hard disk, but it's external only and only allows you to use a 1.8" disk. This means that the maximum storage space is compromised and the performance is very poor.

But we do have another option...

Lenovo make a module for their laptops called the ThinkPad Serial ATA Hard Drive Bay Adapter II (Part Number 43R1980) which replaces the removable drive bay with a 2.5" HDD.

See those two plastic plugs on the back? You don't need 'em. Chop them off with a sharp knife...

 

Then fit in your hard disk... in this case a very nice 500Gb one...

... and slap it in the drive bay.

 

It might be a bit tight, but it does the job and doesn't look wrong if you leave it in there all the time.