Categories: Insurance, Data Centers, Healthcare, Decontamination, Preservation & Stabilization, Preventive Reconditioning
Following a fire, water, or other type of incident, the contamination deposited on equipment can adversely affect electrical, electronic and mechanical assemblies. Sometimes the contaminants are visible, such as dark soot, while others blend in with existing debris, making themselves known over time in the form of rust.
Computer Server Inspection
In many cases, the first thing you may notice following a fire is soot covering a surface. Soot consists mostly of carbon, which can be corrosive and conductive depending on the composition of the consumed matter. As an example, chlorine accounts for a significant portion of polyvinyl chloride (PVC) mass. Once consumed, even small concentrations of PVC byproduct are highly reactive. This reactivity is the cause for corrosion/rust.
It's vital not to power on equipment that exhibits loss related contaminants. Prematurely powering on equipment with conductive matter can cause electrical short circuits, damaging equipment that may not have sustained functional damage as a result of the loss.
Keep in mind the distinct difference between a surface that is contaminated and equipment that sustained physical or functional damage. There is a tendency to describe both as simply “damaged”. Newly manufactured circuit boards are not damaged because they have residue from the soldering process. The manufacturer simply cleans the boards. Following a loss, the boards are decontaminated using the same processes employed by the manufacturer.
There could also be damage as a result of heat or physical impact. Depending on the extent of damage, repairs may cost as much as the whole server. This does not mean that if a portion of the server experienced heat damage, replacement should be the default. A detailed repair versus replacement cost analysis should be performed.
There are quite a few instances whereby procurement of a new server, which may be the primary control or “brains” of a much larger and more expensive machine, would trigger upgrades to other systems. As an example, a large $1.5M stamping press utilized to shape or cut metal utilizes a computer as the backbone of its operation (the brain). That computer communicates with the press’s controller and operator interface panel. Replacement of the computer with a newer version could cause a communication problem between the controller, operator interface panel, and the new “brain”. Some view this scenario as an opportunity to advocate for a new press. If indeed the computer must be replaced, the cost-effective solution may be to upgrade the controller and operator interface panel. When faced with this scenario, a quote was provided for $750K to perform the upgrade. If the computer costs $1,500 to replace, and the cost to restore the original is estimated at $2,500, restoration is still far more cost effective considering the $750K upgrade that would follow the replacement.
Upgrading controllers, in many cases, is also measured in months. This would cause the equipment owners’ clients to look for the same service elsewhere. Regardless of insurance proceeds that may pay for business income loss, losing clients could permanently hinder ongoing operations.
Even if there is no outward damage, it is important to follow these steps:
- Avoid powering on equipment before it has been properly inspected by a professional decontamination expert.
- If there is an uninterruptible power supply (UPS) protecting the server, it should be powered off.
- Dehumidifiers should be introduced to control unstable environments. Maintaining a relative humidity below 40 percent will help slow the rate of surface deterioration when exposed to corrosive contaminants.
Recovery experts will assess the contamination and quantify scientifically where it truly settled. It could very well be that only a small area within a data center was affected. Equipment owners tend to advise that the whole place was equally impacted by the loss, when in reality, smoke migrates based on airflow within the structure. Restoration professionals can also deploy a vapor barrier that will help slow the rate of corrosion. Therefore, recovery may start with preservation and stabilization.
Decontamination Is Key
When vetting recovery experts, it is important to find a company which quantifies the need and the success of the decontamination empirically. The sky does not fall within every loss the same way. As stated above, sections of a data center could be completely unaffected and ready to resume production. In order to instill the necessary confidence that the equipment will not prematurely fail, the restoration expert must be able to harvest samples and process them in a lab in an effort to compare the results to the appropriate industry standard.
There are many good general cleaning companies. Hospitals use such services for non-technical surfaces (floors, cabinets, windows). Many hospitals have in-house biomedical departments to perform technical preventive maintenance cleaning and system calibration. Hospitals retain technical decontamination specialists to address circuit board level particulate removal, which is beyond the scope of general cleaners and biomedical personnel. The same is true for the hospital server computers and network gear. The key is to find an entity that specializes in technical equipment. Doing so also results is favorable outcomes when warranties and services contracts are considered.
Let’s bring in a real-life example.
A large university data center experienced a fire within a power distribution panel. While the fire was contained within the panel, smoke permeated into the data center. Approximately 1,200 servers, network switches, and memory storage arrays had to be decontaminated. The university wanted assurances from their equipment manufacturers that post decontamination warranties and service contracts would not be revoked. Before any work began, AREPA obtained buy-in from the various manufacturers, so they, too, would feel confident when testing the equipment post decontamination. Replacement of the equipment was estimated at $40M. AREPA completed the recovery for $891K and in a fraction of the time it would have taken to replace all the affected items.
Decontamination can successfully remove a variety of contaminants, ranging from benign tornado debris to acidic smoke and fire extinguishing effluent. When properly restored, equipment will operate as it did pre-loss with likely extended life expectancy. The reason this can be said is because companies rarely power off equipment that is contracted to operate around the clock. The data center housing your favorite social media platform does not take a week off for preventive work even when proper redundancy is in place. Operational and environmental debris accumulates over time and causes servers to operate at higher temperatures than designed. Professional decontamination will remove everything off the circuitry, allowing heat to properly dissipate as required for optimal operating parameters. Doing so extends the life of the equipment.
Professional decontamination followed by testing and repair, will restore servers and related networking equipment faster, for less money, and with fewer long-term problems than less invasive cleaning and repair techniques.
If you experienced an incident that resulted in contamination on servers or other equipment, AREPA can help. Contact us to find out how we can quantify scientifically the true extent of exposure, and present a proper plan to decontaminate and restore your hardware in an effort to minimize business income loss.
ABOUT THE AUTHOR
By: Amir Rubin
Amir Rubin is an electrical engineer with 18 years’ experience performing damage assessment to high-tech electronic, electrical, and mechanical equipment that was impacted as a result of fire, water or other contamination event. As Senior Technical Consultant within Envista Forensics for over 10 years, Mr. Rubin conducted hundreds of accurate and in-depth equipment inspections to determine extent of damage, repair/restoration, and potential cost effective solutions to mitigate business interruption. Mr. Rubin has developed procedures for proper decontamination, testing, and repair, which includes working directly with the equipment owners, their technical representatives, and the manufacturers to successfully resolve equipment loss claims.