Is Syncro MSP down?

The team has deemed our Background Workers stable since our latest updates and are calling the all clear.

We will be sharing a retrospective on this incident and action items we will be taking to avoid incidents like this in the future.

This issue is now resolved, but if you have any further questions or concerns relating to this issue, please reach out to our support team at [email protected].

We are aware of delays in PSA import requests affecting Customer, Contact, and Product imports due to higher than usual import traffic since 12p PT today. Peak traffic is currently trending down and we will continue to be monitoring.

Please note that any import requests submitted during this time will still process as the delay period continues to trend down.

The deployment to resolve this issue has been completed. We have validated that hardware keys are operational again.

On Sunday, January 21 at 8:21 AM PT, a standard certificate renewal triggered a restart of our service that controls the Online/Offline status for our assets. While we have mechanisms in place to ensure purposeful restarts of the service are graceful, redistributing agent connections across our servers takes several days to complete.

By Monday, January 22 at 7:50 AM PT, as our traffic increased with the majority of our agent fleet coming back online after the weekend, our Online/Offline service drastically began to drop connections, leading to our agents falsely reporting offline. The problem occurred not only because the connections were still redistributing across the servers, but also the proxy server settings were not properly optimized to handle the influx of traffic.

As a response to the dropped connections, we temporarily disabled offline alerting in the app, adjusted our proxy server settings, and manually restarted each of our servers to forcefully optimize the redistribution of connections. By 11:21 AM PT on January 22, we found that these updates appeared to resolve the issue as our agent fleet held presence connections as expected, so we called the all clear on the incident and re-enabled agent offline notifications.

On Tuesday, January 23 at 8:20 AM PT, as our morning traffic once again increased and despite our changes the day prior, our Online/Offline service was still unable to support the influx of traffic and agent connections began to drop. Agent offline reporting was again disabled at 8:38 AM PT and the team continued work to optimize our server load configurations. After implementing these optimizations, we delayed reactivation of agent offline notifications until 9:00 AM PT on Wednesday, January 24, to confirm our system could reliably manage the morning traffic.

As an immediate response to this incident and to ensure a reliable presence system for our partners as they continue to grow their businesses in the years to come, our team is prioritizing additional optimizations of our Online/Offline system in a test environment simulating our current and projected traffic to ensure we can gracefully handle planned or unplanned restarts of the service in both the short and long term as our agent fleet continues to grow. This work is currently in progress and we are committed to investing whatever resources are necessary to mitigate the likelihood of future incidents.