In August I wrote about the beginning of our Data Center makeover. We have since completed the project and I wanted to provide a follow-up on what had been accomplished. Here are the details:
- We cleaned up abandoned and incorrect wiring. This seems far easier than it turned out to be. When we started to reposition our server racks we were able
to pull up some raised floor panels that I am sure hadn’t been moved in years. When I started pulling up the abandoned cables it became clear that I would just have to cut them. I made sure all the power was disconnected and then started hacking away. Here is a picture of some of the cables that came out of the floor. - We replaced our 20 year old UPS. We really learned a lesson the hard way here. Physical infrastructure really is something you need to stay ahead of the curve with. When the old 10kw UPS was installed sometime in the early 1980s it was more than what was needed at the time (a good thing). Unfortunately, once it was installed no one thought about it again. Over time the excess capacity was used up and additional, rack-based UPS, were purchased and installed. We had no exit strategy. Therefore, when we needed to address power again we were unable to do it easily or in a manner that used anything we currently had installed. It became a rip and replace with about 12 hours of down time. Uncool. Thankfully, we are on the other side of it now and we have a new 52kw MGE unit installed that provides all the capacity we need for today plus an additional 50%.
- We replaced our wall panel based power circuits. Initially we planned to use wall mounted circuit panels to provide our UPS power. However, before we made the UPS purchase, our vendor suggested that we try a rack-based PDU. Wow, I am glad he made that suggestion. Instead of having to call the electrician each time we wanted to move a circuit we only have to call them once – to install the PDU whip. After the whip is installed we can move it, as needed, within the room.
- We replaced our air conditioning system. This was the most expensive part of the project. About 8 years ago we had two 5 ton units installed in our data
center. Just like the UPS it was more than enough capacity at the time. Unfortunately, we didn’t monitor our cooling needs either and soon we had eaten up our redundancy. Any anomaly with the AC units lead to downtime for our servers because we could not cool them. After many discussions we decided to replace the two five ton units with three 12 ton units. We figured that this would provide for virtual and rack-dense growth as well as provide redundancy for our full UPS load of 52kw. - We moved our server racks. Our data center provided a lot of open and available space for the department. Until this summer’s work we even had staff located in the room (a bad idea). After
moving the stored PCs, printers, and laptops out and relocating the staff, we had plenty of room to move our racks. Before the summer the racks were wedged into one corner of the room. This was a serious issue when it came to air flow. Some of the servers were expelling hot air into a wall which caused the whole corner to heat up. Moving the servers allowed us to address the air flow, wiring, and power issues. Not we have a row of full racks that look and function better than before. - We added appropriate storage. I realize that this is a minor thing but for us it is great. One of the issues I have with storing cardboard boxes in the server room is the particulate matter they generate. Additionally, they are always in the way and generally unsightly. We decided that we would use the wall space in the data center to place metal cabinets. These cabinets allow our network team to store supplies and parts in a way that does not interfere with the operation of the servers or the maintenance of their environment.











DNS is a fickle beast. Microsoft DNS is even more so. Back in March of this year, something happened and we managed to scavenge away all of our DNS records. This includes SRV and A records. It was a great day – everything stopped working and I got to learn a lot about repopulating DNS records forcibly. For example, to re-register your DCs in DNS try this:
Entries (RSS)