Intermittent store availability issues
Incident Report for Shogun Frontend
Postmortem

Summary:

From 9:23 AM EST to 10:32 AM EST on January 5th, 2022, storefronts powered by Shogun Frontend were unavailable due to a misconfiguration within Shogun’s content delivery network. Affected consumers would have been presented with an unbranded error message during this timeframe. 

Details:

Shogun’s Engineering team regularly releases new functionality, as well as bug fixes and security updates. Shogun also utilizes a premier content delivery network to ensure the fastest delivery of content to shoppers. 

As part of the software update delivered on January 5th, 2022, Shogun’s Engineering team modified the storage location of credentials and other configuration data as part of routine maintenance to the platform. This change was tested in our pre-production environment with no ill effects prior to this release. Upon release to production it was discovered that the automated steps to remove the configuration data from the existing storage location and the automated steps to add the configuration data to the new location were not guaranteed to run in the correct order. When ran out of order the configuration data had not yet been added to the new storage location, but was removed prematurely from the existing storage location causing the storefront to fail to load.

After troubleshooting the issue and evaluating several options Shogun’s Engineering team began manually correcting the configuration to restore service.

Preventative Measures:

All future configuration storage changes will be completed in two phases. The first phase will put the configuration in the new storage location and update the code to read from both locations preferring the new location. The second phase will remove the configuration from the old storage location.

Shogun’s Engineering team is continually exploring additional ways to make our deployments more reliable.

Posted Jan 07, 2022 - 10:01 UTC

Resolved
The incident has been resolved.
Posted Jan 05, 2022 - 18:24 UTC
Update
Live preview within the XM will be impacted for ~1h. If you require this immediately, please contact support.
We will update status page once this is resolved.
Posted Jan 05, 2022 - 15:58 UTC
Update
All stores have been restored. We are continuing to monitor.
Posted Jan 05, 2022 - 15:33 UTC
Monitoring
All stores have been restored. We are continuing to monitor.
Posted Jan 05, 2022 - 15:32 UTC
Update
Shogun engineers are actively addressing the fix. Some sites have been restored, others will be restored shortly.
Posted Jan 05, 2022 - 15:26 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jan 05, 2022 - 14:51 UTC
Update
We are currently investigating an issue that is affecting all live stores. Our engineers are working hard to identify and resolve the issue.
Posted Jan 05, 2022 - 14:47 UTC
Investigating
We are currently investigating an issue that is affecting live stores. Our engineers are working hard to identify and resolve the issue.
Posted Jan 05, 2022 - 14:35 UTC
This incident affected: Shogun Core Services.