Resolved -
After a period of monitoring we can confirm that all services were restored and have been stable.
May 9, 00:06 BST
Monitoring -
Pelion and Telus have applied mitigation measures to restore Telus subscriber traffic.
We will continue to monitor for stability.
May 8, 19:05 BST
Update -
Pelion support and engineering teams remain on a bridge call with Telus and we have observed some restoration of Telus subscriber traffic recently, most significantly since approx. 18:00 BST (12:00 CT).
We are continuing to work with the mobile network to improve stability and to mitigate any further issues.
We will provide a further update shortly.
May 8, 18:18 BST
Update -
Pelion support and engineering teams are continuing to troubleshoot issues at our Austin data centre.
The previous issues with the providers fiber break has been resolved and most services have been restored, however at this time we are still experiencing 1 way data transit on Telus subscribers. We are currently engaging with the mobile network .
We will provide a further update shortly.
May 8, 16:45 BST
Update -
Pelion Support & engineering teams continue to expedite further mitigations.
Pelion teams are engaged on calls with our data centre partner, and with Telus to address the issues.
Currently radius is being successfully services in our secondary sites, however there is still some impact to Telus subscriber traffic.
We will provide a further update shortly.
May 8, 14:47 BST
Update -
Status:
Incident Update
Pelion Support & engineering teams continue to work on bringing up the new resilient link to try to mitigate the problem further.
The Pelion US platform is available for customers to use.
Single sited Austin IPsec VPNs will still be impacted.
Telus traffic is still being served from our Miami datacentre. Any customers who are still experiencing issues with Telus connectivity and do not have an IPsec VPN to Austin should attempt power cycles/reboots in the first instance.
The original issue with our DC provider is still ongoing, and the fibre provider is attempting to restore service but no expected time to resolution is known for the original issue.
We apologise for any inconvenience and we will provide further updates as they become available.
May 8, 12:17 BST
Update -
Status:
Incident Update
Pelion engineering teams have accelerated a planned upgrade and have failed traffic to a new resilent link.
This has restored service to the Pelion US platform.
Pelion engineering teams are working on the cross site communications between Austin and Miami, so this may still be impacting Single sited IPsec VPNs as Telus traffic is currently in Miami.
The original issue with our DC provider is still ongoing but we are working on further mitigating the problem.
We apologise for any inconvenience and we will provide further updates as they become available.
May 8, 09:24 BST
Update -
Status:
Incident Update
The Fibre provider has identified there is a Fibre break.
The source of this has been identified and repairs are beginning.
Unfortunately, no expected time to resolution is currently known.
We are continuing to work with our provider on this issue and will provide further updates when they are known.
We apologise for any inconvenience.
May 8, 08:20 BST
Update -
Status:
Incident Update
The DC provider's field engineer has determined there is an issue with the fibre connections in Austin.
The DC providers fibre vendor is currently investigating the issue.
We apologise for any inconvenience.
May 8, 06:02 BST
Update -
Status:
Incident Update
The DC provider has dispatched a field engineer to site.
The DC provider are continuing to investigate the issue.
We will provide further updates as soon as more information is available.
We apologise for any inconvenience.
May 8, 03:23 BST
Identified -
Status:
Incident Update:
Austin outbound internet is currently impacted.
Pelion has engaged our datacentre provider who are currently investigating the issue on their end.
Telus traffic has failed over and is being served by our Miami DC.
Single sited Austin IPsec VPNs & the Pelion US platform will still be impacted.
We are continuing to work with our DC provider on resolving this issue.
We apologise for any inconvenience.
May 8, 02:23 BST
Investigating -
Status: Investigating Incident
Service Impacted:
Austin DC
Priority:
High
Pelion Environment Area(s) Affected:
US
Summary:
Pelion Support teams are currently investigating issues with the Austin DC which may have an impact on your service.
We have identified from monitoring that we are seeing connection issues/outbound internet issue with the Austin DC.
This will be impacting the PCM US platform, single sited Austin IPsec VPN and Telus connectivity.
Update:
Pelion support teams are currently investigating the issues.
We will provide an update shortly and we apologise for any inconvenience.
May 8, 00:41 BST