ictCanada - Route to Kitchener DC – Incident details

All systems operational

Route to Kitchener DC

Resolved
Major outage
Started about 2 months agoLasted 1 day

Affected

Data Center - Ontario

Major outage from 1:50 AM to 11:32 AM, Degraded performance from 1:50 AM to 11:32 AM, Major outage from 11:32 AM to 2:33 PM, Degraded performance from 11:32 AM to 2:33 PM, Major outage from 2:33 PM to 4:06 PM, Degraded performance from 2:33 PM to 4:06 PM, Major outage from 4:06 PM to 9:14 PM, Degraded performance from 4:06 PM to 9:14 PM, Major outage from 9:14 PM to 11:37 PM, Degraded performance from 9:14 PM to 11:37 PM, Major outage from 11:37 PM to 1:35 AM, Degraded performance from 11:37 PM to 1:35 AM, Major outage from 1:35 AM to 7:45 AM, Degraded performance from 1:35 AM to 7:45 AM

DNS / Name Servers

Major outage from 1:50 AM to 7:45 AM

IP Addressing

Major outage from 1:50 AM to 7:45 AM

Routes

Major outage from 1:50 AM to 7:45 AM

VPS Platform

Major outage from 1:50 AM to 7:45 AM

Storage Buckets

Major outage from 1:50 AM to 7:45 AM

Updates
  • Resolved
    Resolved

    This incident has now been considered resolved.

  • Update
    Update

    Current status: This is a complex and time-consuming repair as the accident damaged some large fibre cables and infrastructure. Technical teams have been working around the clock to complete this repair.

    Fibre Crews & Technicians: on-site

    Estimated Time To Restore: N/A

  • Update
    Update

    Expected resolution: 7 p.m.- 7:30 p.m. (delayed)

    Current status: This is a complex and time-consuming repair as the accident damaged some large fibre cables and infrastructure. Rogers and Zayo technical teams have been working around the clock to complete this repair.

    Estimated Time To Restore: N/A

    Next update: 9:30 PM ET or sooner should a major change in status occur.

  • Update
    Update

    2024-10-02 5:12:00 PM EDT - Additional comments

    Teams continue to work diligently on repairs at both identified sites. At the first site, north of Dixie, the 288-count cable has been fully repaired, and the work on the 24-count cable is complete. The 72-count cable is currently in progress.

    Meanwhile, at the second site, east on Orenda, the 288-count cable for the east side has been prepared and is in the truck, while the 288-count cable for the west side is being lashed and secured toward the west. The 288-count cable must be fully completed before work on the 120-count and 60-count cables can commence. The estimated time for splicing to begin is between 7:00 PM and 7:30 PM.

    Fibre Crews & Technician ETA: on-site

    Estimated Time To Restore: N/A

    Next update: 7:30 PM ET or sooner should a major change in status occur.

  • Update
    Update

    2024-10-02 12:04:00 EDT - Additional comments

    Expected resolution: Late afternoon


    Current status: This is a complex and time-consuming repair as an accident damaged some large fibre cables and infrastructure. Rogers and Zayo technical teams have been working around the clock to complete this repair.

  • Update
    Update

    2024-10-02 10:32:00 EDT - Additional comments


    Rogers fibre crew continues working on prepping the cables for splicing. On the north side, the cable has now been pulled and lashed, and is now in the truck to be prepped for splicing. 

    The additional crews have arrived on-site to provide additional support however, due to the number of cables impacted, the preparations are taking longer. Concurrently, Several Rogers Support teams are on a conference bridge along with working on a restoration plan to ensure the repair efforts are progressing. 

    Technician ETA: on-site

    Estimated Time To Restore: N/A 

    Next update: 12:00 PM ET or sooner should a major change in status occur.

  • Identified
    Identified

    Rogers and Zayo fiber damaged. Crew is replacing the cable. ETR: None.

    Teams have identified multiple impacted fiber cables. Crews are currently prepping the existing 288 cable on both sides, with the strand work halfway complete. In addition, they are addressing the traffic light issues.


    Once this is done, the fiber crew will proceed with a new 432 cable, prepping and starting the splicing process. A new crew is now working on the east side of Orenda Rd. If available, an additional contractor will be called in after 7 AM to assist with the work on the east side.

    Fibre Crews &Technician ETA: on-site
    Estimated Time To Restore: N/A
    Next update: 10:30 AM ET or sooner should a major change in status occur

  • Investigating
    Investigating
    We are currently investigating this incident.