Incident 3277 - Service Interruption: Wireless Network Degradation

Incident

Incident 3277 - Service Interruption: Wireless Network Degradation

Status: Closed
Start:
End:
Duration: 4 days 1 hour 17 minutes
Affected Components:
Device Communications Wireless Network Communications PMG GSM Network - (Mexico and Canada Only) GPS Location Breadcrumbs PMG LTE Network 1-minute Breadcrumbs PCG/Duo LTE Network Device Installations Satellite Communications Mobile Originated PCG/Duo (Outbound messaging from the device to the back office) Mobile Terminated PCG/Duo (Inbound messaging from the back office to the device) Mobile Originated PMG (Outbound messaging from the device to the back office) Mobile Terminated PMG (Inbound messaging from the back office to the device) Mobile Originated Trimble Gateway (TG) (Outbound messaging from the device to the back office) Mobile Terminated Trimble Gateway (TG) (Inbound messaging from the back office to the device)
Update

We are currently investigating an issue impacting the degradation of our wireless network. We have opened a ticket with our wireless network provider, and they are currently investigating the issue. At the moment, this problem seems to only be affecting our Canadian customers. We will provide more information as soon as more details become available.



Product Impact

Wireless Network Communications (Canada Only) 

Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 

Investigating

Incident Update: 

Our carrier partner has confirmed an outage only affecting 3G (GSM) Roaming services in Canada. They are engaged and applying remediation efforts. There is currently no ETR (Estimated Time of Resolution) of when services will be restored. An update will be provided as more information becomes available. 



Product Impact:

Wireless Network Communications (Canada Only) 

Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 

Identified

Incident Update:

Our carrier partner has identified the source of wireless network degradation in Canada. They are aggressively troubleshooting the issue with no Estimated Time of Resolution. An update will be provided as more information becomes available.


Product Impact

Wireless Network Communications (Canada Only) 

Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 


Identified

Incident Update: Identified


Our wireless network carrier partner continues to troubleshoot the issue impacting device communications. Additional escalation measures have been taken to resolve this. There is no Estimated Time to Resolution. Further updates will be provided as more information becomes available.

Incident Summary: At the moment, there is an ongoing incident affecting our network partners' ability to roam across multiple networks. They are currently investigating a network issue that is causing failures for some subscribers trying to use 3G (GSM) services. The impact is limited to 3G data. These 3G GSM services are only hosted in Canada and Mexico and were sunset in the United States in Q4 of 2022.


Product Impact

Wireless Network Communications (Canada Only) 

Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 


Identified

Incident Update: Identified

There are no new updates at this time. Our wireless network carrier partner continues to troubleshoot the issue impacting device communications, and additional escalation measures have been taken to resolve this. Currently, there is no Estimated Time to Resolution. We will provide further updates as more information becomes available.

Incident Summary: At the moment, there is an ongoing incident affecting our network partners' ability to roam across multiple networks. They are currently investigating a network issue that is causing failures for some subscribers trying to use 3G (GSM) services. The impact is limited to 3G data. These 3G GSM services are only hosted in Canada and Mexico and were sunset in the United States in Q4 of 2022.


Product Impact

Wireless Network Communications (Canada Only) 

Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 

Identified

Incident Update: Identified

There are no new updates at this time. Our wireless network carrier partner continues to troubleshoot the issue impacting device communications, and additional escalation measures have been taken to resolve this. Currently, there is no Estimated Time to Resolution. We will provide further updates as more information becomes available.



Product Impact

Wireless Network Communications (Canada Only) 

Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 

Identified

Incident Update: Identified

There are no new updates at this time. Our wireless network carrier partner continues to troubleshoot the issue impacting device communications, and additional escalation measures have been taken to resolve this. Currently, there is no Estimated Time to Resolution. We will provide further updates as more information becomes available.



Product Impact

Wireless Network Communications (Canada Only) 

Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 

Investigating

Incident Update: Identified

There are no new updates at this time. We are currently experiencing an incident affecting the roaming capabilities of our wireless network partners. The issue is not with our carrier network provider but with their roaming partner. A provider that facilitates roaming connectivity for operators has reported the problem. They are investigating a network issue that is causing failures for some subscribers attempting to use 3G GSM roaming services. This impact is limited to 3G roaming data. Currently, there is no Estimated Time to Resolution. We will provide further updates as more information becomes available.


Product Impact

Wireless Network Communications (Canada Only) 


Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 



Product Impact

Wireless Network Communications (Canada Only ) 

Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 

Investigating

Incident Update: Identified

Since midnight on June 27, 2024, our carrier partner has been aggressively troubleshooting an issue affecting their roaming partner's 3G GSM network in Canada. During this process, the 4G LTE network experience has also been degraded. We are collaborating with our carrier partner to understand the full extent of this degradation and its impact. We will provide updates as more details become available. We sincerely apologize for the inconvenience.



Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 



Product Impact

Wireless Network Communications - 4G LTE Network

Wireless Network Communications -3G GSM (Canada Only)


Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 

Investigating

Incident Update: Identified

Our carrier partner continues to work with their roaming partner to resolve the issue impacting international cellular network communications. This event is affecting the 3G GSM network in Canada. While the 4G LTE network experience has been degraded, there has been some minor recovery from a carrier perspective. We are collaborating with our carrier partner to understand the full extent of this degradation and its impact. We will provide updates as more details become available. We sincerely apologize for the inconvenience.



Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 



Product Impact

Wireless Network Communications - 4G LTE Network (Roaming Canada)

Wireless Network Communications -3G GSM (Canada Only)


Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 

Investigating

Incident Update: Identified

Our investigation into the root cause is ongoing, but no discoveries have been made. Our carrier partner is actively working with their roaming partner to resolve the issue affecting international cellular network communications. This event impacts the 3G GSM network in Canada. While the 4G LTE network experience remains degraded, there has been some minor recovery. We are collaborating with our carrier partner to fully understand the extent of this degradation and its impact. Updates will be provided as more information becomes available. We sincerely apologize for the inconvenience.


Product Impact

Wireless Network Communications - 4G LTE Network (Roaming Canada)

Wireless Network Communications -3G GSM (Canada Only)


Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 

Investigating

Incident Update: Identified

Our carrier partners have resolved the issue affecting 4G LTE roaming in Canada. However, the investigation into the 3G GSM network of their roaming partners is still ongoing. No discoveries have been made so far, but engineers are continuing to troubleshoot. We will provide updates as more information becomes available.

Product Impact

Wireless Network Communications -3G GSM (Canada Only)


Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device) 

Monitoring

Incident Update: Monitoring 

Our carrier partners have resolved the issue affecting 4G LTE. However, there are still issues with the international 3G roaming network. We do not have an estimated time of resolution (ETR) at this time. We will continue to monitor the situation over the weekend and will update this incident as more information becomes available.

Product Impact

Wireless Network Communications -3G GSM (Canada Only)


Device Impact:

Device Communications: Device Logins, Device Messaging & Device Workflow (To and From device)

Resolved

Incident Update: Resolved

Our carrier partners resolved the 4G LTE issue on June 28 at 1:30 PM CT and the 3G GSM network issue on June 29 at 4:30 PM CT. As of June 29 at 4:30 PM, all international cellular network communications have been restored to normal functionality. We sincerely apologize for the delay and have requested a Root Cause Analysis (RCA) from our carrier provider.

Product Impact: Update

Wireless Network Communications (International - Canada) 

All cellular network communications have been restored.


Device Impact: Update

Device Communications (International - Canada) 

Device communications should now process in real-time.