Inside an Unexpected Service Equipment Meltdown

It’s 10 a.m. on a weekday. The lights in your building start flickering and then suddenly go out. You smell smoke and immediately call the fire department. After the firemen put out the fire in your service equipment room, you ask yourself, “What just happened?”

The clues offered in this gallery of photos reveal the root cause of this catastrophic electrical equipment failure and the resulting fire that ensued. Each image helps reinforce the need to properly inspect, operate, and maintain the equipment under your control.


Electrical Service Meltdown 1

Ground Zero

Upon initial investigation, you see the 400A main breaker (on left) is fried. The service conductors are melted right through the steel breaker enclosure and front cover. All of the service conductor insulation is melted away, and only bare and burned 500kcmil copper remains. As you look at the charred ¾-in. plywood behind the main breaker you realize how lucky you were that this incident happened during normal business hours so you could quickly react to the fire.


Electrical Service Meltdown 2

Damage Could Have Been Much Worse

The ¾-in. plywood caught on fire from the intense heat produced by the short‐circuited internal breaker components. Additional heat was generated by the phase‐to-ground and phase‐to‐phase faults that ensued after the conductor insulation melted. With no conductor insulation, the copper conductors pressed against the enclosure top and front cover. Note the other wood framing members in this electrical room. Black soot from the fire can be seen on the 2x4 framing members and plywood ceiling above the steel enclosure.


Electrical Service Meltdown 3

Heat Transfer

The intense heat in the breaker enclosure transferred to the adjacent panelboard enclosure. You can see the scorched paint on the left interior wall of this enclosure. An unused red conductor in the panelboard was in contact with the side wall and melted on the steel. If you look closely, you can see the mark left on the enclosure after the conductor with melted insulation was pulled away.


Electrical Service Meltdown 4

Melted Cable Insulation in Adjacent Switch Panel

Upon further investigation, you examine the 500kcmil conductors in the through‐the‐wall pipe nipple that feed a 400A switch, which serves as a second service entrance for the building. The service conductor insulation in the nipple is also melted together. So how did all of this happen?


Electrical Service Meltdown 5

Pole-Top Source of Power

We start our investigation by reviewing the service and service equipment layout. The building is fed by a 208Y/120V, 3-phase pole‐mounted utility transformer arrangement. Each transformer has its own pole‐mounted fuse for each of the three utility phases. The service lateral consists of three 500kcmil phase conductors and a reduced‐size neutral. Most of the service equipment is 65 years old.


Electrical Service Meltdown 6

Ground Fault Current Flowed Through This Main Breaker Panel

The service conductors enter the bottom of this enclosure (lower left) and are connected to the three terminals at the top of the switch. A second set of phase conductors is tapped from these same terminals and feeds the line‐side of the back‐to‐back 400A fused switch on the back side of this wall. The intense heat from the shorted and grounded phase conductors transferred enough heat to the phase conductors in the through‐the-wall pipe nipple to melt all of the insulation together in the nipple (see previous slide). Although no fault current was actually flowing in the tapped phase conductors in the nipple, ground fault current was flowing through the steel enclosures as it tried to find a circuit path back to the electric utility source through the bonded neutral.


Electrical Service Meltdown 7

Energized Bare Conductor Burns through Top of Steel Enclosure

When the line‐side service conductors overheated from the short circuit within the main breaker, the conductor insulation melted. The residual bending stress in the conductors forced the conductors tight against the top of the enclosure and the front cover. The C‐phase conductor ground faulted to the enclosure top while the B‐phase conductor ground faulted to the enclosure front cover and melted completely open. When both conductors contacted the enclosure steel, a phase‐to‐ground‐to‐phase fault was established.


Electrical Service Meltdown 8

Failed Arc Chute Plates Lead to Phase-to-Phase Faults

It is clear in this photo that one or more of the arc chute plates fell across the movable contact of the B‐phase. The B-phase arc chute steel plate is severely arc‐damaged from making contact with the A phase. It looks like the cross‐connection occurred through the notch in the insulated housing. This notch is where the operating shaft connects all three movable contacts to the breaker trip mechanism. The A‐phase movable contact assembly is severely melted away. A similar shorting occurred between the B-phase and C-phase.


Electrical Service Meltdown 9

Severe Arc Flash Damage on C-Phase Contacts

This is a close up of the C‐phase movable contact for the breaker. The left main contact appears to be missing its precious metal contact pad. The spring for this same contact appears to be broken. It is unclear if the contact pad unsoldered itself due to the intense heat created by the arcing. It is possible that the missing contact pad and broken spring pre‐existed and were contributory to initiating the breaker meltdown. The triangular contact is the C‐phase arcing contact. It “makes” before the main contacts close and “breaks” after the main contacts open. The arcing that occurs during opening and closing of the breaker is designed to occur on the sacrificial arcing contacts and not the main contacts of each phase.


Electrical Service Meltdown 10

Molten Metal Drips Down from Above

Molten copper and brass dripped down within the breaker housing. The solidified metals are seen just above the three thermal‐magnetic trip assemblies. The arc chute plates shorted the breaker internally near the main contacts. All the fault current flowed upstream of these current sensing elements. The breaker was not able to sense the overcurrent conditions from the shorting. The only remaining overcurrent protective devices for this service were located on the electric utility pole. The utility’s fuses provide primary fuse protection for its three pole‐mounted transformers. These fuses allowed the maximum transformer capacity to feed into this fault before they melted opened.


Electrical Service Meltdown 11

Load Side of Breaker Enclosure Suffers Less Damage

The bottom area of the breaker enclosure was away from the zone of intense heating. The intense heating occurred near the breaker contacts and the grounded phase conductors at the top of the enclosure. You can see that the conductor insulation is intact on the breaker load‐side conductors. The grounded‐conductor insulation in the forefront is intact (and not melted) until it passes to the left side of the breaker housing where arcing occurred within the breaker housing.


Electrical Service Meltdown 12

Two for the Price of One is Not Always a Good Deal

Here’s a close up shot of the line‐side terminals on the failed main breaker. You can see on the left-most terminal, some of the conductor strands have been cut to allow them to fit in the terminal clamp hole. It is not likely that this terminal was designed or rated for two 500kcmil conductors. If it was rated for two conductors, there would be no need to cut the conductor strands to make them fit. You can see severe discoloration of the copper due to the overcurrent and overheating.


Electrical Service Meltdown 13

Improper Use of 400A Switch

This photo shows the lower section of the 400A switch that is tapped from the load side of the 400A main breaker that melted. This switch is not a service disconnect and is not classified as service equipment. It is a feeder switch. The neutral of this feeder switch should not be bonded to the steel enclosure. Since this switch neutral was incorrectly bonded to the enclosure, when ground‐fault current flowed on the main switch enclosure, it also formed a parallel fault‐current path through the feeder supply conduit, switch enclosure, back to the improperly bonded neutral conductor, and then back to the utility source. The fault current flowing in the enclosure heated the steel up enough to melt the sealing compound in the screw holes of the black insulation board. The melted, dripping sealant is visible in several places in the photo.


P3 strives to bring you quality relevant industry related news.

See the origial article at:


Continue reading
  1193 Hits
1193 Hits

Fifteen Best Practices for a Successful Data Center Migration


Data center migrations are often complex and risky. These best practices will help I&O leaders invest the appropriate amount of time and money into planning, execution and testing in order to protect the business and maximize the chances of a successful data center migration.


Key Challenges

  • Colocation, merger and acquisition (M&A) activity, outdated facilities, consolidation initiatives and new approaches to sourcing data center capacity often create the need to move IT equipment from one data center to another.
  • Most migrations experience cost overruns and performance degradations due to improper planning or the lack of dedicated teams and resources.
  • Existing staff members often have little experience in planning, organizing and actually moving equipment from one data center to another.


I&O leaders planning data center migrations as part of an infrastructure delivery strategy:

  • Use outside expertise with a proven methodology, if you do not have the internal knowledge and skills to effectively plan, manage and execute the data center migration project.
  • Design and execute a strong communication plan at all phases of the project to eliminate rumors. Engage all affected constituencies often and with complete information.
  • Develop risk mitigation, migration and fallback plans by using disaster recovery (DR) procedures for test runs.
  • Continuously document the process throughout the project to develop lessons learned and reference materials for postmigration evaluations and closure.


This research examines best practices in data center migration. It is not a full examination of all migration methodologies. Rather, it is a list of best practices that are intended to help I&O leaders succeed during a migration process.

Data center migration is not merely about establishing an infrastructure and moving workloads from Point A to Point B. It's often a complex and risky project, where the right process and expertise are crucial. Just to illustrate, advanced planning is paramount to a successful migration. Nevertheless, if the organization does not have the right internal resources to plan properly, it must combine those internal resources with external ones in order to effectively plan, manage and execute the migration project. I&O leaders must prevent the migration project from adversely affecting the business. They must ensure that applications and services continue to function normally — with minimal downtime and with no degradation in performance.

To help I&O leaders achieve this goal, this research focuses on 15 best practices for data center migration (see Figure 1).

Figure 1. Fifteen Best Practices for Data Center Migration

Data Center Migration
Research image courtesy of Gartner, Inc.
Source: Gartner (March 2017)


Most successful data center migration projects share similar practices, such as expertise, preparation, management, execution, communication and business alignment. The purpose of this research is not to provide an extensive examination of each one of these practices. Rather, it is to provide a pragmatic list of best practices. This list was derived from observations of numerous Gartner clients involved in data center migration projects over the last six years.

Based on our guidance in "Data Center Migrations — Five Steps to Success," we've organized these 15 best practices into the five major steps shown in Figure 1: initiation, risk assessment, planning, execution and closure. However, organizations may vary the order of these best practices, depending on the circumstances of their migration. They can also apply these best practices to multiple environments (for example, on-premises, colocation and/or cloud).

The following sections describe the 15 best practices that should be considered in a data center migration project.

1 — Skills and Expertise

Having the right skills and expertise is crucial. Because data center migration is not an everyday activity, existing staff members often have little experience in planning, organizing and actually moving workloads from one data center to another. I&O leaders must assess whether internal resources are available and capable of effectively planning, managing and executing the migration. If internal resources are lacking, we recommend using outside expertise.

Note: A key reason why migration projects fail is that I&O teams often think of migration primarily as an equipment-moving project. However, the majority of the work — as well as the majority of the risk — lies in developing a workload migration plan. The equipment is the easy part — workload placement, dependencies, business impact and risk are the hard parts.

2 — Project Team

Major or complex migration needs a dedicated leader. That leader should establish a team with representatives from all affected constituencies. The leader also should have the authority to allocate resources and to direct people (see "Data Center Migrations — Lessons That Will Save Time and Money" ).

Team members should not only be experts in their domains, but also be willing to learn about adjacent disciplines, because surfacing all the interdependencies will be critical. The more elaborate or expansive the migration, the greater the likelihood that team members will need to be relieved of their day-to-day responsibilities to work on the migration project full time. Be realistic about the experience of these staff members, and be prepared to augment their skills with outside help as needed. For longer projects, be aware that replacement planning will likely be required: As team members move to other roles during the project, their skills will need to be replaced.

3 — Preparation

A successful data center migration depends on previous preparation and advanced planning.

A committee including both IT and business representatives should be established to adequately account for all required resources, including:

  • Migration costs (see "Three Alignments to Achieve Before Using Data Center Consolidation to Optimize Costs" )
  • Existing contracts for software, hardware, services (including telecom), maintenance, DR, facilities and other items

Preparation should also involve the following tasks:

  • Creating a detailed inventory of both equipment and applications
  • Performing a business impact analysis (BIA) review (see "Use Business Impact Analysis to Enable Effective Business Continuity and Disaster Recovery Programs" )

These fundamental tasks should entail a detailed evaluation or audit of exactly what needs to be moved, when and how.

Rehearsal is also an essential part of the preparation process. To ensure all units understand the process, organizations should first define the rehearsal on paper, iterating as necessary. Then, the organization should schedule multiple migration rehearsals during the project to validate assumptions and to determine critical information like migration time and resource availability.

4 — Simplification

The less there is to move, the easier the migration. Therefore, simplify, minimize, virtualize, consolidate and eliminate as much as possible before starting the migration process.

5 — Interdependencies

Part of the risk assessment stage, often called the "discovery phase," should include a detailed assessment of the interdependencies among applications and IT equipment (servers, storage and networking). This assessment will help define the viability and details of a phased migration. We recommend using several tools, including configuration management database (CMDB) tools and others, during the risk assessment stage.

6 — Communication

Executing an effective communication plan during all phases of the migration project is paramount to eliminating rumors and false information sources. Communication should engage all affected constituencies, and clearly assign roles and responsibilities.

Publicizing the migration to both the internal IT staff and the affected business units is also critical to avoiding surprises. HR can be of great assistance — particularly if personnel will be impacted. Publicize milestones, failures and successes. Prepare a detailed employee and management communication plan, and provide updates regularly (preferably through a continuously updated online portal).

7 — Planning

Data center migrations typically occur in stages. The number of stages often varies according to factors like data center size, service risk levels, budget and time constraints. Most organizations use a variation of the following multistage approach:

  • First, the organization migrates low-risk groups, with the assumption that something in the process will probably need to be corrected.
  • Once the overall process is solid and the staff is well-trained in dealing with contingencies, the organization migrates high-risk groups.

8 — Contingency Plan

Problems will arise during the migration. The challenge is to identify these problems ahead of time and formulate appropriate risk mitigations. The key to success is a good, solid preparation phase. For example, a detailed inventory of equipment and network links is typically the foundation of contingency plans. In addition, interim equipment and backup systems should be included in the contingency plans wherever necessary.

9 — Premigration Tests

Performance improves with practice. Before migrating the equipment, run a complete set of tests to establish a "baseline" of infrastructure and application operability, functionality and performance.

10 — Migration

To avoid unexpected errors, impose a "change freeze" period starting before the migration and ending after the migration.

During the migration, be prepared to face common issues, such as network connectivity problems, incorrect credentials (username and password) and lack of validation/testing. These issues may have a cascading effect on other workload moves, especially if an escalation path has not been properly determined. Thus, such issues may elongate the freeze period and negatively impact the project.

11 — Testing

For all testing, adopt a risk-based approach that incorporates the BIA results and input from business subject matter experts (SMEs). For example, when performing application-level tests, ensure that application owners/business units participate in the validation effort and sign-off process.

Do not forget to include failover scenarios in your testing plan.

12 — Postmigration Tests

Use the same test cases executed during the premigration phase, and compare the postmigration results with the baseline results. Differences in results may indicate that new problems have arisen during the migration process. Address all issues discovered.

After a successful migration, have special support resources on hand for a few days. Pay extra attention to differences in online transaction processing performance, peak performance and batch-processing performance. Finally, double check that all services were fully tested in terms of functionality, resiliency and performance.

13 — Audit

A postmove review and audit of the migration project is recommended. The review should include an evaluation of the following:

  • The project process
  • Conformity with the initial business plan and design specifications
  • Conformity with the project schedule
  • Feedback from project members and stakeholders

This review will provide valuable insight regarding critical success factors, lessons learned and knowledge that can be syndicated for other projects within your enterprise. Lessons learned could make future projects more effective.

14 — Closure

Execute the closure properly to ensure no hidden costs or expenses will continue to impact IT in the future. For example, take the following steps:

  • Close out service contracts on older IT equipment
  • Close out software contracts no longer in use
  • Degauss leftover storage assets (including copiers)
  • Cancel vendor contracts for building management, maintenance services and supply delivery services

15 — Updates

Finally, processes, procedures and documentation — including DR plans, compliance tests and audit certifications — should be updated once the data center migration is complete. In addition, validate whether all support systems (for example, CMDB) were properly updated. The new data center will likely have a different setup, which may also require training sessions for the operational teams. Lastly, recognition and marketing are also important. Recognize the efforts of all parties, and promote the successful migration with the business entities.

Acronym Key and Glossary Terms

BIA:  business impact analysis
CMDB:  configuration management database
DR:  disaster recovery
SME:  subject matter expert


The best practices identified in this research were derived from observations of numerous Gartner clients involved in data center migration initiatives. This data was collected from January 2011 through January 2017. The clients involved spanned a wide variety of industries and multiple geographies.

P3 strives to bring you quality relevant industry related news.

See the origial article at:

Continue reading
  1554 Hits
1554 Hits

Google and Cisco Signed the Papers

...and Now Starts the Heavy Lifting

Google Cisco

What to look out for when the latest marriage in the land of hybrid cloud starts bearing fruit.

There is probably no vendor closer than Cisco to the kind of ubiquity VMware enjoys in enterprise data centers. Zooming out from the technical details of Cisco’s recently announced cloud partnership with Google, this is a key thing to understand.

Recently, the leadership at Amazon Web Services and Google Cloud Platform started taking steps that demonstrate an understanding that the road to the traditional enterprise market lies through companies whose products are already in enterprise data centers. Their top rivals – Microsoft, IBM, and Oracle – are such companies, and the most obvious way to deal with that reality is to partner with other enterprise stalwarts, such as Cisco and VMware.

Hence, since this past August you can spin up VMware servers in AWS that reportedly look and act like they’re on the same network as your on-premises VMware environment (result of a partnership between Amazon and VMware announced a year earlier), and sometime next year you’ll be able to run a Google cloud software stack on a Cisco HyperFlex system inside your data center that will make GCP an extension of your on-premises IT (or vice versa).

“This type of announcement gets them [Google] a tremendous amount of enterprise attention,” Stephen Elliot, program VP for management software and DevOps at IDC Research, said in an interview with Data Center Knowledge. “These types of announcements are a recognition that the companies that are going to win in the future are going to be those that really understand the legacy challenges” but present a roadmap for transitioning that legacy to any cloud environment.

A Kubernetes World

Another thing Google and Cisco’s partnership does is provide new enterprise distribution opportunities for Kubernetes, the Google-born open source project that’s quickly becoming the dominant platform for managing and orchestrating Linux containers, he added. Built to mimic the way Google deploys and runs software across its global data center network, Kubernetes will likely become core to the way most developers and IT operations staff work in the future.

In Google and Cisco’s vision, Kubernetes is how software deployed on-premises will run the same way cloud-native software runs in the cloud. In a different partnership, Google, VMware, and Pivotal are busy adopting Kubernetes for VMware – another path to the enterprise data center for the open source platform; both Amazon and Microsoft recently joined the Cloud Native Computing Foundation, the Linux Foundation group that now administers Kubernetes; and Docker, the company that did more than any other to popularize use of application containers, is integrating Kubernetes with its flagship enterprise product.

Also born at Google, also open source, and also part of the future hybrid cloud stack by Cisco and GCP is Istio, whose alpha release Google, IBM, and Lyft launched in May. An enabling technology for container-based systems, it is a way to combine micro-services that run in containers into applications without altering their code and to manage and secure them in a consistent manner.

Another part of the stack is Apigee, the API management platform Google acquired last year. This is key to unlocking the value of hybrid cloud. Through APIs, services running in Google’s cloud will be able to access and use data stored on legacy enterprise systems in company data centers.

Big Questions

We’re witnessing early stages of what Elliot described as a “massive workload migration decade,” and both technology vendors and their customers are going through the thought process necessary to build the enterprise technology platforms of the future. As enterprises go through the process, the vendors have to be prepared to help them migrate to those platforms.

There are tens if not hundreds of billions of dollars of “technical debt,” or investment in existing enterprise data centers out there. “There’s also CIOs recognizing that different workloads are going to be on different types of architectures,” Elliot said.

Helping CIOs get to a point where their teams are using modern application architectures while leveraging their existing tech investments is key. A partnership like Google and Cisco’s can be really interesting for large enterprise accounts, but it will depend on the way the partners will handle the integration.

Since the partnership revolves around open source technology, there’s also the question of how much the vendors expect to rely on the open source communities to enable this integration, and how much individual enterprise end users are willing to invest in open source development efforts themselves, he pointed out.

The fundamental question overall is how complete of a package enterprises can expect to see once the solution hits the market. What level of integration between Kubernetes, Istio, Apigee and their own systems should they expect, how much security will be baked in, and what level of support they will get?

We’ll start seeing answers to those questions next year, but they are questions all technology vendors who are hoping to survive in the enterprise market should be working to answer. “It’s going to be a multi-cloud world, and it’s going to be pretty complex,” Elliot said. “If you’re not in this game, making these kinds of announcements, you’re on the edge of a cliff.”

P3 strives to bring you quality relevant industry related news.

See the origial article at:


Continue reading
  1043 Hits
1043 Hits

How to Avoid Common Mistakes in Data Center Planning and Design

The new, free eBook from Schneider Electric, “A Practical Guide to Data Center Planning and Design” will walk you through the data center planning process, including design and site selection. It also includes some best practices and success stories based on real customer implementations.

Download the eBook now to get some valuable tips that’ll help ensure your next data center is a resounding success – and doesn’t fall victim to any of those common mistakes.

Download the eBook

Continue reading
  1044 Hits
1044 Hits

Power Quality Tips

Here are some common signs that you might be having problems with power quality and steps to take to begin addressing some of the issues.

How do you know if you have power quality problems?

The answer is you either conduct a battery of specific tests in the course of a power quality analysis, or you produce reports from your power quality monitor. Fully executing either approach requires expertise not typically present in plant engineering.

It might be better to ask how you know to suspect power quality problems. Here are some common signs, but note that their absence does not mean your power quality is necessarily what it should be:

• A high rate of motor bearing failures and/or motor winding failures.

• The mechanics find pitted bearings in mechanical drives and other equipment under their purvey.

• Circuit board replacement is a normal occurrence. Same for PLC module replacement.

• Lights flicker. Lights don’t seem bright enough. Lighting system component replacement is a normal occurrence.

• Neutral conductors appear discolored.

• Nuisance breaker trips occur, but their source is rarely, if ever, identified.

• Insulation resistance (IR) testing shows cable failure at an abnormal rate. Note that a qualified electrical testing firm can tell you from experience whether the rate is abnormal.

If you have any of these symptoms, suspect one or more power quality problems. You may be able to identify some issues with a little sleuthing and some basic measurements, but you’ll probably need a qualified firm to do a complete workup.

If you suspect power quality problems that are due to equipment failures and other symptoms, how do you respond?

It is unlikely you have the required expertise in house, and you probably have to go through a process before you can bring in a firm with that expertise.

While that process is in progress, what can you do to start addressing at least some of these issues?

• Perform voltage measurements on all feeders, then on all branch circuits. Measure line to ground and line to line, RMS. You’re looking for low voltage, high voltage, and voltage imbalance. The sheer scope of the work may require hiring an electrical services firm. While awaiting approval, take the measurements for your critical equipment.

• Inspect for grounding and bonding errors. If you see a ground rod on the load side, that’s a red flag that something is wrong. This rod serves no electrical purpose, and is probably substituting for proper bonding.

• Check all transformers (except auto-transformers) for proper grounding; the National Electrical Code (NEC) considers them to be separately derived sources.

One way to turbocharge this process is to start keeping a spreadsheet of the problems as they occur if they seem related to power quality.

If you note key information, you can sort in a way that will enable you to conduct a Pareto analysis. This will, for one thing, reveal patterns that can lead to quicker resolution.

Include these fields:

• Whether the supply is a branch circuit or feeder.
• Nominal voltage.
• Building or area where load is situated.
• Affiliated production line, if applicable.
• Type of equipment served (use standardized codes, such as 1 for production motor with drive, 2 for production motor without drive, 3 for lights, 4 for HVAC, 5 for computers, etc.).

P3 strives to bring you quality relevant industry related news.

See the origial article at: &

Continue reading
  1011 Hits
1011 Hits