Wired Life in the Roanoke Valley

Title: First, Break All the Rules
Author(s): Marcus Buckingham and Curt Coffman
ISBN: 0-684-85286-1
Published: 1999, The Gallup Organization (Simon & Schuster)
Reviewed: March 12, 2010

This book is a discussion of research performed by the Gallup Organization. Specifically, it is the product of two “mammoth research studies” that were run during the last 25 years (from 1999). The surveys asked the questions “What do the most talented employees need from their workplace?” and “How do the world’s greatest managers find, focus, and keep talented employees?”

The authors found that the strength of a workplace can be determined by the answers to 12 questions which “measure the core elements needed to attract, focus, and keep the most talented employees.” The twelve questions can be broken into five areas of focus (described using the analogy of a mountain climb):

Base Camp: “What do I get?”

1. Do I know what is expected of me at work?
2. Do I have the materials and equipment I need to do my work right?

Camp 1: “What do I give?”

3. At work, do I have the opportunity to do what I do best every day?
4. In the last seven days, have I received recognition or praise for doing good work?
5. Does my supervisor, or someone at work, seem to care about me as a person?
6. Is there someone at work who encourages my development?

Camp 2: “Do I belong here?”

7. At work, do my opinions seem to count?
8. Does the mission/purpose of my company make me feel my job is important?
9. Are my co-workers committed to doing quality work?
10. Do I have a best friend at work?

Camp 3: “How can we all grow?”

11. In the last six months, has someone at work talked to me about my progress?
12. This last year, have I had opportunities at work to learn and grow?

The analogy of a mountain climb is appropriate because the lower questions must be met before one can work on the upper ones. The authors state, “the key to building a strong, vibrant workplace lies in meeting employees’ needs at Base Camp and Camp 1. This is where you should focus your time and energy.” Management training in the 90s focused on Camp 2 and Camp 3 while missing significant lapses in Base Camp and Camp 1. Managerial success is, in the opinion shared in this book, found in questions #1-6.

The remainder of the book focuses on what managers can do to ensure success (defined as employees strongly agreeing (5 on a scale of 1-5) with questions #1 through #6. To create this environment, managers must: select based on talent, define the right outcomes, focus on strengths, and find the right fit.

Takeaway Questions:

1. Am I focusing on the process or the outcomes?
2. Am I focusing on excellence or failure?
3. Are my reports in the right role?
4. Am I trying to be a catalyst or a controller?
5. With whom do I spend most of my time?

Tech Ed 2010 is done. Another week is checked off the calendar. I have “Dismissed All” of the office alerts I escaped this week while at Tech Ed. The sessions are over. The bags are packed. Where to we stand?

1. Lots of great sessions on Security, Hyper-V, Storage, Exchange, and SCOM.
2. Lots of things to implement on my return.
3. Lots of business cards handed out (seriously when else are you going to use them).
4. Lots of walking done.
5. Lots of tweets sent to #teched.
6. Lots of good food in NOLA.
7. Lots of appreciation for the folks at Tech Ed and In Depth.
8. Lots to look forward to in Atlanta, GA (Register now and save $500).

Wish List for Next Year

1. A return of IT Pro product releases.
2. A return of Tech Ed Connect.
3. A return of Steve Riley.
4. A return of Marcus Murray.
5. A return of good snacks.

One of the many things I learned at this year’s Tech ED is Microsoft’s preference for and improvements to iSCSI in Hyper-V. Since we are a Fiber Channel shop we haven’t given a lot of thought to running with iSCSI. However, the WSV315 (Clustering) and WSV16 (Storage) sessions really hammered home Microsoft’s current commitment to this technology as well as their commitment to Hyper-V performance.

WSV316 Summary (Presented by Robert Larson and David Lef) [Minus the MSIT Stuff]

1. In Windows Server 2008 R2 iSCSI management is greatly improved.
2. There are some serious considerations to make for iSCSI performance in Hyper-V:
1. General network best practices still apply.
2. Use jumbo frames with higher IO request sizes.
3. Use dedicated NICs or vLANs.
4. Unbind unneeded services from your iSCSI nics (File sharing, DNS, etc.).
3. Server workloads and applications should determine your Cluster Shared Volume configuration
1. Virtual config should resemble an equivalent physical config.
2. If a separate LUN is needed for OS, Data, and Logs than the Hyper-V server should have three separate LUNs (one for OS, data, and logs).
3. You can avoid a lot of disk contention this way.
4. Ask yourself (and your SAN vendor): How many IOPS can we handle? How many CSVs do I need to spread the IOPS out?
5. Hyper-V Backups, Parent, Guest, or Both?
1. Backup the Parent if it is not Windows or if you want to protect the entire machine.
2. Backup the Guest for application specific considerations (SQL, Exchange).
6. Improve Performance
1. Use active/active multipathing load balance policies (round robin, least queue depth) vs. failover mode.
2. Follow the vendor’s guidelines for timer settings.
3. Should you run your vendor’s host utilities to optimize?
4. Pay attention to spindle count for the workload
5. Turn off TCP Chimney for NICs used for iSCSI

WSV315 Summary (Presented by Steven Ekren) [Minus the MSIT Stuff]

1. Host Clustering (Clustering your Hyper-V nodes)
1. Clustering of Physical Machines
2. Allows for VM mobility
3. Allows for VM survivability
4. Allows for OS health monitoring
5. Allows for load distribution
6. Great for Hyper-V
2. Guest Clustering (clustering your VMs)
1. Runs in the VMs
2. Allows guest patching with no downtime
3. Uses shared storage and iSCSI is the only option at this time (#fail)
4. Allows for application mobility
5. Allows for application health monitoring
3. Host and Guest Clustering
1. Best of both worlds
4. Tips and Tricks for Hyper-V Clustering
1. Heartbeat Settings
1. SameSubnetThreshold & SameSubnetDelay – Configure in guest cluster to increase tolerance for live migration. Change from 1000ms to 5000ms and the loss of Heartbeat to 15.
2. This is important since the live migration can cause slow response.
2. Keeping VMs Off the Same Node
1. Use “AntiAffinityClass NAME”
2. Configure it on the host cluster
3. KB296799
4. It tries to avoid placing two clustered VMs on the same Hyper-V node
3. Low Priority VMs
1. Change their autostart setting
2. Configure this on the host cluster
3. Set it to NO if you want the VM to remain off after migrating
4. Ensuring Startup
1. Persistent Mode
2. Configure this on the host cluster
3. When enabled, and the cluster starts, VMs will attempt to place themselves on the node where they were last manually moved

Suffering from what must be a massive inferiority complex Microsoft used the VIR 315 session to set the record straight about Hyper-V R2 performance. Citing comparisons performed by Virtual Reality Check and others Kenon Owens stated that in Hyper-V on Windows Server 2008 R2 the performance between VMWare’s ESX and Hyper-V was almost identical. This is nothing new and basically boiled down to about 25 minutes of full-blown marketing talk. Given that we already run Hyper-V R2 this was not really necessary. Just a heads-up for future sessions – VMWare bashing is really not necessary in a session on Hyper-V tuning (hint: we already run it).

After the initial marketing bombardment the real meat of the session began. Given by Clint Huffman, the performance analysis section of the presentation was superb.

Here are Clint’s 21 steps to better Hyper-V performance:

1. Know your server before virtualizing it. Leveraging some of Mark Russinovich’s work, Clint stated that CPU (\Processor(*)\% Processor Time > 75%), Memory (\Memory\Available MBytes < 100MBs), Disk (\LogicalDisk(*)\Avg Disk sec/Read|Write > 15ms), and Network (\Network Interface(*)\Output Queue Length > 2) were all metrics that, generally speaking, indicated poor server performance.

2. Use Hyper-V Performance Counters. You will need to run the Hyper-V specific metrics like \Hyper-V Hypervisor Logical Processor (*)\% Total Run Time, \Hyper-V Hypervisor Virtual Processor(*), and \%Guest Run Time. Avoid using the task manager and % Processor Time (as they are not helpful in Hyper-V environments).

3. Assess your processor ratios.

4. Install the Hyper-V integration Components.

5. Use Windows Server 2008 (R2 Preferably) VMs.

6. Use SLAT Processors.

7. Close your manager sessions when not in use as they consume resources.

8. Unless you are running RemoteFX, use the basic video drivers.

9. Avoid running services on the root partition.

10. Keep an eye on your Disk Response Times. They are still important.

11. If you can afford it, use SSDs.

12. Avoid snapshots unless you really need them.

13. Use Pass-Through Disks with SCSI.

14. Use Fixed-Sized VHDs.

15. Disabled removable media as it is polled periodically.

16. Leave at least 512MB of RAM for your root partition. 1GB or more is recommended.

17. Allocate enough RAM for your guests. Check \Memory\Available MBytes better than 10% is good.

18. Avoid network congestion. Measure and add NICs as needed.

19. Avoid legacy network adapters.

20. Use NIC features in R2 (VMQ).

21. Use PAL. Clint also showed us the PAL 2.0 Beta 1 tool. This tool can be used to effectively parse performance data.

This might have been the most practical session for me at TE2010. Clint did a great job of giving us some great tips to help us improve our Hyper-V environments.