VIP Manager issue
Incident Report for Symantec VIP Cloud
Postmortem

On January 11 at 01:12 UTC, Symantec VIP users may have experienced timeouts using the following services: VIP Manager, Authenticate User API, and VIP Self-Service Portal (SSP). The issue corrected itself by 02:50 UTC and the VIP services returned to normal operation.

We are actively investigating an internal networking error and preliminary corrective actions have been taken to prevent reoccurrence of this issue while the investigation is still ongoing. A full RCA will be provided to affected customers by the end of business on January 19, 2018.

The investigation into the issue continues. We have completed an analysis of the application servers and found no issues at this level, nor did we see a slow down on the time taken to process a request on the application level. However, our monitors indicated there was a load balancer failover due to a missed health check which terminated some inflight transactions. We have proactively tuned our network infrastructure to mitigate the issue, preventing the health check from failing prematurely, as we continue to investigate the root cause of the missed health check.

Posted over 1 year ago. Jan 12, 2018 - 18:31 UTC

Resolved
The service is now fully operational. We are monitoring the service and investigating the root cause of the issue. Please accept our apologies for inconveniences caused.
Posted over 1 year ago. Jan 11, 2018 - 16:37 UTC
Update
We continue to troubleshoot the issue. Customer impact remains unchanged. The next update will be at 17:00 UTC or sooner if more information becomes available.
Posted over 1 year ago. Jan 11, 2018 - 15:59 UTC
Update
Our investigation is ongoing and we have engaged additional internal and external resources to troubleshoot the issue. Customer impact remains unchanged. The next update will be at 1600 UTC or sooner if more information becomes available.
Posted over 1 year ago. Jan 11, 2018 - 15:35 UTC
Investigating
We are currently experiencing issues with our VIP manager. The symptoms include intermittent time-out errors shortly after logging in. We are currently investigating the issue with highest priority. We will provide an update at 15:30 UTC or sooner if more information becomes available.
Posted over 1 year ago. Jan 11, 2018 - 14:37 UTC