Results 1 to 3 of 3

Thread: power outage affecting core routers and switches

  1. #1
    Join Date
    Feb 2003
    Posts
    3,667

    power outage affecting core routers and switches

    We are still performing our root cause analysis, but wanted to provide you with an update. At approximately 3AM EDT our core routers and switches stop functioning. We traced it down to a faulty UPS taking down 1/2 a cabinet including our core routing switches. At the moment we have bypassed the switch and are ensuring the effected servers are back up. Routing/switching has already been re-established.

    More as we find out.

  2. #2
    Join Date
    Feb 2003
    Posts
    3,667
    This is what caused the UPS failure:

    smaller.jpg

    In case you're wondering, that's a 30A internal UPS power cable and breaker that failed (it's behind the battery).

    We are talking to the manufacturer as this is not a normal defect.

  3. #3
    Join Date
    Feb 2003
    Posts
    3,667
    1) The UPS failure took down one of our core switches providing service to several cabinets; we will be changing the topology and adding additional redundancy so one subsection power failure won't affect anything beyond that specific section (i.e. the 1/2 cabinet serviced by that power outlet).

    2) We are still talking to the manufacturer, they agreed tor replace the unit, BUT, we may not do this. Why? Previously our power feeds were generator backed only and UPS backed up at the cabinet level; if there's a power outage, then it takes a few split seconds before the generators kick in, the UPS solves that by covering that period. However, now all our power is generator backed up and UPS backed up before it gets to the cabinet, so the use of the UPS in the cabinet may no longer make sense as it introduces a single point of failure (and we want to avoid these) and there's no added benefit of having two UPS points (before the cabinet and on the cabinet).

    3) We want to urge everyone to move to our new network sooner than later as it adds more redundancy by design. There's no cost difference between the "regular" and "premium" network.

    More to follow once I have it.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

-->