All Systems Nominal

Premium workspaces on us-east-1 eare inaccessible

11:16 Investigating - Initial status of Investigating

11:22 Identified - Root cause identified, fix being applied

11:26 Resolved - Workspaces are accessible again.

Multiple workspaces unreachable due to provider maintenance (OVH)

13:45 Identified - Initial status of Identified

13:45 Update - Waiting for further updates from provider

14:40 Resolved - All workspaces are back to normal.

14:41 Resolved - OVH incident - https://network.status-ovhcloud.com/incidents/qgb1ynp8x0c4

Maintenance of service provider

07:49 Monitoring - Initial status of Monitoring

07:51 Update - Upstream OVH maintenance affects 4 of our nodes. We do not expect any downtime or disturbances. Situation is currently being monitored.

14:35 Resolved - Everything is back to normal.

premium-use1 outage

18:35 Monitoring - Initial status of Monitoring

18:36 Update - Due to dockerhub's pull limit, some images failed to pull during an update, rendering some workspaces unavailable.

19:17 Resolved - All issues resolved

360 Dialog unavailable

13:09 Investigating - Initial status of Investigating

13:11 Update - External provider 360 Dialog seems to be having some issues, affecting sending and receiving of Whatsapp messages

13:27 Update - External partner confirmed issues on their on-prem API impacting whatsapp360 app. Partner support is engaged and rocket.chat is in direct contact with them. No ETA

18:23 Resolved - All issues resolved

Premium SA East cluster is behaving unpredictably

20:21 Investigating - Initial status of Investigating

20:29 Resolved - Heavy operation caused slight hiccup, team got notified and checked quickly. System is stabilizing now.

SA East Dedicated cluster workspaces experiencing instability

12:22 Investigating - Initial status of Investigating

19:44 Resolved - Completed

EUC1 Database cluster down

07:18 Identified - Initial status of Identified

07:43 Update - Database nodes crashed, causing them to reboot. We are now slowly recovering as the nodes are coming back up.

07:49 Update - One of the nodes is still down. We are working on getting it back up. But workspaces will now start recovering.

09:15 Resolved - All issues resolved

Cloud Services degraded peformance

13:43 Identified - Initial status of Identified

14:04 Monitoring - Resolved but will continue to monitor

07:54 Resolved - n/a

Cloud services partial outage

14:10 Investigating - Initial status of Investigating

16:39 Monitoring - Data bearing node went down, system is now recovering from the failure

19:45 Monitoring - Systems have been performing normal over the past several hours. We are monitoring current system health.

07:55 Resolved - n/a

Some cloud services are not responding

06:44 Investigating - Initial status of Investigating

07:04 Resolved - Cloud services are back operational

Some workspace functionalities went inaccessible for a limited timeframe

22:17 Identified - Initial status of Identified

22:22 Resolved - Root cause was identified and quickly fixed afterwards, workspaces are now functioning normally

Some workspaces are showing delayed response time on Europe region

13:16 Investigating - Initial status of Investigating

14:20 Monitoring - Workspaces are back up, we are investigating performance and other potential sources of degradation

14:34 Resolved - Everything is back up as normal, workspaces are functional as usual

Europe cluster experiencing partial outage due to network proxies

08:03 Identified - Initial status of Identified

08:34 Monitoring - We have recovered the network proxies, workspaces are back to normal, we are monitoring

19:46 Resolved - Resolved

Europe cluster database node down

06:42 Identified - Initial status of Identified

06:52 Resolved - False alarm, db nodes are running fine

Some trial workspaces are suspended

17:03 Monitoring - Initial status of Monitoring

20:29 Resolved - Remaining suspended trials have been restored

EUC workspaces inaccessible

07:29 Investigating - Initial status of Investigating

08:42 Identified - DB outage, now recovering

10:05 Identified - Some issues with Kubernetes cluster

10:59 Monitoring - Workspaces recovering

11:27 Monitoring - Suspending all trials while recovering customers

11:47 Monitoring - All customers recovered and are fully operational

17:01 Resolved - All customer workspaces are fully operational

US Workspace disruption

15:35 Identified - Initial status of Identified

16:16 Monitoring - Database restored, network issue on db nodes

07:56 Resolved - n/a

Service disruption

18:13 Identified - Initial status of Identified

18:25 Resolved - Resolved

Service Disruption

22:42 Investigating - Initial status of Investigating

22:56 Monitoring - Service Recovered

07:56 Resolved - n/a

Mobile App Unavailable in Google Play

22:44 Identified - Initial status of Identified

22:44 Update - Our app is temporarily not available for download from the Google Play store. We are working with Google to address the issue as quickly. We apologize for the inconvenience

07:57 Resolved - The issue had been resolved.

Service Interruption in Cloud EU workspaces

22:40 Investigating - Initial status of Investigating

23:05 Identified - Edge node down, removed from rotation

23:14 Monitoring - Continuing to monitor for any issues

Service Interruption

06:58 Investigating - Initial status of Investigating

07:12 Monitoring - Found the issue in authorization service and resolved it

Interruption to some customers workspaces

21:49 Investigating - Initial status of Investigating

22:25 Resolved - False alarm, the issue was self contained to single workspace and issue resolved

Interruption in new trials

17:54 Identified - Initial status of Identified

17:59 Resolved - Should be fully resolved