Dell MD3200i + VMware ESX hanging issues

The Dell MD3200i SAN is a cheap(ish) effective storage device certified for VMware ESX.
I had no issues using it with a single ESX host, but when trying to use it with more than one host I was seeing hangs of the ESX servers, loss of connectivity from hosts to the VirtualCenter and strange logs saying that it couldn't communicate with the LUN.

Mar 24 19:56:11 virt08 vobd: Mar 24 19:56:11.295: 797333253337us: [esx.problem.vmfs.heartbeat.timedout] 4eb93a45-ea456b83-4c0a-0010189da888 disk03-1.

Mar 24 19:57:36 virt08 vobd: Mar 24 19:57:36.699: 797418651679us: [esx.problem.vmfs.heartbeat.recovered] 4eb93a45-ea456b83-4c0a-0010189da888 disk03-1.

After much going back and forth with Dell ensuring that I had the network configured with their latest recommendations (they keep changing it), they took al look at the stats on the network card of the MD3200i


========================================================================
                          TCP PORT STATISTICS - PORT 0                
========================================================================
  TCP Received Segments          0xf6f28bd
  TCP Tx Segments                0xd773310
  TCP Rx Segments In Error       0x0
  TCP Tx Byte Count              0x1a167dd174
  TCP Rx Byte Count              0x2683bbc3
  TCP reTx Timer Expired Count   0x584
  TCP Rx Dup ACK Count           0xa7890
  TCP RX ACK Count               0xee2915
  TCP Rx Delayed Ack Count       0x5a4256
  TCP Tx Ack Count               0x65098dc
  TCP Rx Seg Out Of Order Count  0x3242c
  TCP Rx Window Probe Count      0x0
  TCP Rx Window Update Count     0x49c


Here they saw high numbers of ACK issues (approximately 10% of all network transactions).

This prompted them to ask that I set up the IO queue depth and a delayed ACK workaround from VMware.

This immediately fixed my issue!

To change IO queue depth

esxcli nmp roundrobin setconfig --type "iops" --iops 1 --device <device UID>

Delayed ACK Workaround

Disabling Delayed ACK in ESX/ESXi 4.x and ESXi 5.0.x

1. Log in to the vSphere Client and select the host.
2. Navigate to the Configuration tab.
3. Select Storage Adapters.
4. Select the iSCSI vmhba to be modified.
5. Click Properties.
6. Modify the delayed ACK setting, using the option that best matches your site's needs:
   • Modify the delayed ACK setting on a discovery address (recommended):
     1. On a discovery address, select the Dynamic Discovery tab.
     2. Select the Server Address tab.
     3. Click Settings.
     4. Click Advanced.
   • Modify the delayed ACK setting on a specific target:
     1. Static Discovery tab.
     2. Select the target.
     3. Click Settings.
     4. Click Advanced.
   • Modify the delayed ACK setting globally:
     1. Select the General tab.
     2. Click Advanced.
7. In the Advanced Settings dialog box, scroll down to the delayed ACK setting.
8. Uncheck Inherit From parent.
9. Uncheck DelayedAck.
10. Reboot the host

This change should be safe to apply to all MD3xxx SANs, so you could just do the global change.

You should now have a fully working system

I hope this page helps you fix your problems, please click on an advert to show your appreciation.

Can't open Discovery Search Mailbox

While I was playing about with Discovery Search, I had the issue that I couldn't open up the Discovery Search Mailbox with either OWA or Outlook.

I suspected it was a permissions issue, but I was already a member of the Discovery Management group and when I looked at the Full Access Permissions in Exchange Management Console I could see that group listed.

After much scratching of my head, I decided to dive into ADSIedit and have a look at that Mailbox.

I found the Mailbox under Default Naming Context > DC=Domain,DC=Name > CN=Users, CN=DiscoverySearchMailbox{blahblahblah}

and when I took a look, I saw that the Discovery Management group wasn't listed. I just added it back in and gave it full control and the problem was solved!

I hope this tip proves useful please click on an advert to show your appreciation.

Exchange 2010 Litigation Hold - Now What Do I Do?

It's easy to put a mailbox on Exchange Litigation Hold... but how do you get the information back out?

To view what's in the mailbox, you need to do a Discovery Search.

First you need to ensure you have the Discovery Management role assigned to your mailbox.

Add-RoleGroupMember -Identity "Discovery Management" -Member jsmith

Now you have this, you can perform a Discovery Search through the ECP webpage https://mailserver/ecp

Click on Mail Control - Discovery

Then create a new Mailbox Search, you can choose what type of items, which mailboxes and what dates. You can do an Estimate Search which is handy for showing what the results will be, and then choose a "Discovery Search Mailbox" to output the results to.

Once the search completes, you can open the "Discovery Search Mailbox" using Outlook or OWA (Click on your name and type Discovery Search Mailbox).

I hope you find this article helpful, please click on an advert to show your appreciation

Adding .NET 3.5 to Windows 2008 R2

Windows 2008R2 doesn;t install .NET Framework 3.5 automatically, yet when you download it from the MS website, it won't install (you get the error You must use the Role Management Tool to install or configure Microsoft .NET Framework 3.5 SP1).

What you need to do is install it using PowerShell in a command prompt (running as an Administrator):

powershell.exe -noexit -ImportSystemModules  Add-WindowsFeature net-framework 

Easy!

Finding who's using your bandwidth on Cisco ASA

Until recently the Cisco ASA had a Top Talkers facility which shows you who is using up your bandwidth.

Now that's no longer available, one quick(ish) and easy(ish) way is to:

Connect to the ASA with Putty and capture the output of "show conn" to a CSV file.

Then edit the file with a text editor (I suggest Notepad++) and replace " bytes " with ""

Save the file, then open it up in Excel and you can sort by size and work out who's using what...

Find email addresses in Exchange 2010

The quickest and easiest way to work out who (or what) is using an email address is to open up a PowerShell command window and run: 


get-recipient -results unlimited | where {$_.emailaddresses -match "address@company.com"} | select name,emailaddresses,recipienttype


for full details on the result, try 


get-recipient -results unlimited | where {$_.emailaddresses -match " address@company.com "} | fl

thumbs.db can't be deleted because it's in use

If you are like me, you keep all your media on a network drive.

In what it thinks is an aid to productivity, Windows 7 (and Vista) will go to these network drives and automatically create a hidden file called thumbs.db which contains a thumbnail image of every media (video or photo) file. Naturally this takes time if you have hundreds of images or some large videos, and I often try to delete the directory before the thumbs.db have finished being created... and get an annoying error message saying that the file can't be deleted as it's still opened by another application.

You can turn off the generation of this file by going:

Start > Run > Gpedit.msc

User Configuration > Administrative Templates > Windows Components > Windows Explorer > Turn off caching of thumbnail pictures in hidden thumbs.db> Enabled

I hope this proves helpful... if it does, please click on an ad to show your appreciation.