High Read/Write latency with RDM Disks (MSCS) on 5.1
When you upgrade to 5.1, the default path selection policy for active/active arrays is now Round Robin (RR). In 5.0, It would be Fixed PSP. The reason I wanted to stress on this is that MSCS cluster does not support using RR. And when you use it you will see significant performance degradation. The VM seems to be running fine but slow, event when creating a folder, you may have to wait. One other good indicators is the significant increase in the backup times after an upgrade.
You can check out performance graphs and counters. In my case, there was no dropped commands but upto 5000ms latency on read and writes, and when I run esxtop, it was clear that there was something wrong as DAVG was around 50 and KAVG was around 5000. You can also change default PSP if you need to.
If you want to change the defaults:
The screenshot below does show normal values for the same device after changing the PSP for RDM disk to Most recently used (MRU).
DAVG/cmd: the average response time in milliseconds per command being sent to the device, threshold 25
KAVG/cmd: the amount of time the command spends in the VMkernel