20:52 Investigating - Initial status of Investigating
21:55 Monitoring - Bypassed infrastructure provider limitation on udp traffic by switching to DNS over TLS
19:16 Resolved - Setting as resolved
15:36 Identified - Initial status of Identified
17:42 Update - Core issue resolved leading to the email problem. Waiting on our email provider to restore email delivery
22:05 Update - Still waiting on our email service provider to restore email sending
22:10 Monitoring - Service provider updated ticket with re-instated. Will continue to monitor
19:16 Resolved - Setting as resolved
06:32 Identified - Initial status of Identified
08:42 Monitoring - Issue mitigated
23:22 Update - Still some customers experiencing slowness. We are working to move these customers to additional resources
00:27 Monitoring - Some customers moved. Monitoring situation to see if improved
02:39 Monitoring - Will continue to monitor over the weekend
19:16 Resolved - Setting to resolved
19:42 Identified - Initial status of Identified
20:53 Update - Some trials worksapces down at the moment
15:43 Resolved - Incident solved. Affected services are operational.
20:29 Identified - Initial status of Identified
20:53 Update - Issue with DNS resolving at infrastructure level - https://status.us.ovhcloud.com/pages/incident/59dd23da8827c804746f1664/62cf2350e828a90534973914
21:23 Monitoring - All routing and DNS issues seem to be resolved. Continuing to monitor the situation
02:08 Resolved - Preventative measures are being put in place we are considering the incident resolved
13:56 Identified - Initial status of Identified
18:07 Resolved - Issue with mail provider unable to be resolved. Switched mail sending provider
07:45 Identified - Initial status of Identified
09:10 Resolved - Incident solved. Affected services are operational.
19:39 Identified - Initial status of Identified
19:58 Monitoring - Database node issue discovered. Issue seems to be resolved. Continuing to monitor and investigate root cause
21:18 Monitoring - We will continue to monitor but services are nominal
14:06 Identified - Initial status of Identified
14:32 Resolved - Incident solved. Affected services are operational.
08:41 Monitoring - Initial status of Monitoring
19:33 Resolved - Incident solved. Affected services are operational.
13:53 Identified - Initial status of Identified
04:32 Investigating - Initial status of Investigating
04:32 Investigating - Initial status of Investigating
16:02 Investigating - Initial status of Investigating
18:35 Monitoring - All systems are nominal. We are going to continue to monitor and are provisioning new servers to help mitigate
18:34 Investigating - Initial status of Investigating
19:31 Identified - We identified a problem with a database node. It should be improving we are working to restore to full desired condition.
20:50 Update - The load increased. We have restricted connections
21:26 Update - We have lifted the restrictions. Still degraded but should be accessible
00:12 Monitoring - All should be performing as usual. We will continue to monitor and apply additional fixes to prevent the incident from reoccuring
04:45 Resolved - Further mitigations put in place
02:24 Investigating - Initial status of Investigating
02:27 Update - Faulty messages sent about push notification limits
02:28 Update - Bad deployment rolled back to stop any further alerts.
02:44 Update - Will be deducting the pushs caused by this issue from usage. Full investigation to follow to determine how this happened.
07:01 Investigating - Initial status of Investigating
07:17 Update - A loadbalancer node is down in use1. Working on resolving now. Still investigating why euc1 network is effected
07:23 Update - EU Workspaces should now be nominal. Will keep monitoring. US Workspaces should also be stablizing
07:43 Update - Workspaces in EU and US should be stablizing. Leaving as degraded as we continue to observe
08:20 Monitoring - All Workspaces currently nominal. Will continue to monitor
19:51 Resolved - All continues to be Nominal