Wired Life in the Roanoke Valley

In August I wrote about the beginning of our Data Center makeover. We have since completed the project and I wanted to provide a follow-up on what had been accomplished. Here are the details:

1. We cleaned up abandoned and incorrect wiring. This seems far easier than it turned out to be. When we started to reposition our server racks we were able to pull up some raised floor panels that I am sure hadn’t been moved in years. When I started pulling up the abandoned cables it became clear that I would just have to cut them. I made sure all the power was disconnected and then started hacking away. Here is a picture of some of the cables that came out of the floor.
2. We replaced our 20 year old UPS. We really learned a lesson the hard way here. Physical infrastructure really is something you need to stay ahead of the curve with. When the old 10kw UPS was installed sometime in the early 1980s it was more than what was needed at the time (a good thing). Unfortunately, once it was installed no one thought about it again. Over time the excess capacity was used up and additional, rack-based UPS, were purchased and installed. We had no exit strategy. Therefore, when we needed to address power again we were unable to do it easily or in a manner that used anything we currently had installed. It became a rip and replace with about 12 hours of down time. Uncool. Thankfully, we are on the other side of it now and we have a new 52kw MGE unit installed that provides all the capacity we need for today plus an additional 50%.
3. We replaced our wall panel based power circuits. Initially we planned to use wall mounted circuit panels to provide our UPS power. However, before we made the UPS purchase, our vendor suggested that we try a rack-based PDU. Wow, I am glad he made that suggestion. Instead of having to call the electrician each time we wanted to move a circuit we only have to call them once – to install the PDU whip. After the whip is installed we can move it, as needed, within the room.
4. We replaced our air conditioning system. This was the most expensive part of the project. About 8 years ago we had two 5 ton units installed in our data  center. Just like the UPS it was more than enough capacity at the time. Unfortunately, we didn’t monitor our cooling needs either and soon we had eaten up our redundancy. Any anomaly with the AC units lead to downtime for our servers because we could not cool them. After many discussions we decided to replace the two five ton units with three 12 ton units. We figured that this would provide for virtual and rack-dense growth as well as provide redundancy for our full UPS load of 52kw.
5. We moved our server racks. Our data center provided a lot of open and available space for the department. Until this summer’s work we even had staff located in the room (a bad idea). After moving the stored PCs, printers, and laptops out and relocating the staff, we had plenty of room to move our racks. Before the summer the racks were wedged into one corner of the room. This was a serious issue when it came to air flow. Some of the servers were expelling hot air into a wall which caused the whole corner to heat up. Moving the servers allowed us to address the air flow, wiring, and power issues. Not we have a row of full racks that look and function better than before.
6. We added appropriate storage. I realize that this is a minor thing but for us it is great. One of the issues I have with storing cardboard boxes in the server room is the particulate matter they generate. Additionally, they are always in the way and generally unsightly. We decided that we would use the wall space in the data center to place metal cabinets. These cabinets allow our network team to store supplies and parts in a way that does not interfere with the operation of the servers or the maintenance of their environment.

DNS is a fickle beast. Microsoft DNS is even more so. Back in March of this year, something happened and we managed to scavenge away all of our DNS records. This includes SRV and A records. It was a great day – everything stopped working and I got to learn a lot about repopulating DNS records forcibly. For example, to re-register your DCs in DNS try this:

nltest [/server:ComputerName.MyDomain.COM] /dsregdns

Did I mention that it was great? We managed in one day to blow our DNS away and then turn off all scavenging. We thought that turning it off would keep us from killing ourselves until we could figure out what was going on. It seemed like a good idea at the time..

Anyway, wouldn’t you know it, we never went back to turn scavenging back on. The whole once bitten  twice shy deal. Only recently, when one IP address could be connected to four or five PC names did we understand why scavenging is so important. Here is what we found when we went to fix our DNS:

1. Our PCs and Servers were not automatically updating their dynamic DNS entries because our DNS zone did not have scavenging enabled. We had turned it off to ensure that records were not scavenged. We didn’t know that we could have done that by just turning scavenging off for the server. In order for scavenging to run it has to be enabled on the SERVER and the ZONE as described in this excerpt from http://support.microsoft.com/kb/296116/it.

Note The aging and scavenging can be performed on a per-server, per-zone, or per-record basis, according to an algorithm.

When a DNS dynamic update client registers its record for the first time, the timestamp is set from the original update, and the timestamp never updates as long as scavenging is not enabled. We recommend that you first enable Aging and Scavenging on the zone level, allowing the current clients to update their timestamp. The time period should be equal to or greater than the No-Refresh interval that is configured on the zone. As soon as this time period has passed, Aging and Scavenging can be implemented on the server level to allow scavenging on the records. Scavenging does not occur unless it is enabled at the server level and the zone level.

2. We learned that Microsoft really isn’t up-front about DNS. We had to track down information using Google. We found a few helpful articles, here they are:

DNS Scavenging
Exporting DNS Records with a Timestamp

3. Beware of clusters. Somehow, back in May, we must have manually re-created the Cluster DNS entry for Exchange 2007. This allowed it to work but messed up the permissions on the record. It is essential that the account that runs the cluster service has permission, in DNS security, to update the record. Microsoft talks about this configuration in KB871111.

4. Remember to take things slow when re-enabling scavenging. It is a necessary part of DNS infrastructure, especially with DHCP.

Each year I go solo with fantasy football. Maybe I just don’t know enough people or enough people that aren’t already in a league, but I am always playing people I don’t know. Maybe I will head over to ESPN again this year and pray for better luck. I usually make it about 4 weeks before I get destroyed to the point it isn’t fun any longer. Let’s hope I make it to week five this year.

I have to admit, I am still sore about Microsoft’s change to Prometric from Pearson VUE and the lack of a single-test upgrade path for the MCITP:Enterprise cert. Thankfully Microsoft just sent me a very cathartic survey about the MCP program. I didn’t pull any punches and I urge you to do the same. Be honest.