As of 10:50 UTC, the issues affecting the BigPanda pipeline have been mitigated, and all BigPanda services are now operating within normal operational levels.
We apologize for any inconvenience that this may have caused.
Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Posted Jun 13, 2021 - 11:39 UTC
Monitoring
As of 10:50 UTC, the issues affecting the BigPanda pipeline have been mitigated, and all BigPanda services are now operating within normal operational levels.
We apologize for any inconvenience that this may have caused.
We will continue to monitor systems for any change in operational capacity.
Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Posted Jun 13, 2021 - 11:08 UTC
Update
The team continues to work through the services and remediate issues for the remaining impacted customer subset.
Pipeline latencies for most customers have subsided, and BigPanda console functions for these customers should be operating as expected.
For a small subset of customers, some users may see updates to their console; however, it may take time for users to see 'current' data as the backlogged data continues being processed.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 13, 2021 - 10:37 UTC
Update
The team continues to work through the services and remediate issues.
Pipeline latencies for most customers have subsided, and BigPanda console functions for these customers should be operating as expected.
For a small subset of customers, some users may see updates to their console; however, it may take time for users to see 'current' data as the backlogged data continues being processed. These users may also be experiencing intermittent issues loading and leaving comments, loading the activity feed, and performing other UI actions such as manually resolving Incidents or performing Split/Merge.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 13, 2021 - 08:52 UTC
Update
The team continues to work through the services and remediate issues.
Pipeline latencies for most customers have subsided, and BigPanda console functions for these customers should be operating as expected.
For a small subset of customers, some users may see updates to their console; however, it may take time for users to see 'current' data as the backlogged data continues being processed. These users may also be experiencing intermittent issues loading and leaving comments, loading the activity feed, and performing other UI actions such as manually resolving Incidents or performing Split/Merge.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 13, 2021 - 08:05 UTC
Update
The team continues to work through the services and remediate issues.
Some users may be starting to see updates to their consoles, though it may take time for users to see 'current' data as the backlogged data is currently being processed.
Users may still be experiencing pipeline latencies, as well as intermittent issues loading and leaving comments, loading the activity feed, and performing other UI actions such as manually resolving Incidents.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 13, 2021 - 06:42 UTC
Update
The team continues to work through the services and remediate issues. At this time, we are working to mitigate issues within all pipeline service clusters, and we're continuing our effort related to making the pipeline fully operational in the second availability zone.
Some users may be starting to see updates to their consoles, though it may take time for users to see 'current' data as the backlogged data is currently being processed.
Users may still be experiencing pipeline latencies, as well as intermittent issues loading and leaving comments and loading the activity feed. Logging out and back into BigPanda may allow users to see the comments again.
The issues with AutoShares and manual sharing of Incidents should now be remedied.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 13, 2021 - 04:42 UTC
Update
The team continues to work through the services and remediate issues. At this time, we are working to mitigate issues within all pipeline service clusters, and we're continuing our effort related to making the pipeline fully operational in the second availability zone.
Some users may be starting to see updates to their consoles, though it may take time for users to see 'current' data as the backlogged data is currently being processed.
Users may still be experiencing pipeline latencies, as well as intermittent issues loading and leaving comments and loading the activity feed. Logging out and back into BigPanda may allow users to see the comments again.
Users may also see issues impacting AutoShares and manual sharing of Incidents.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 13, 2021 - 03:36 UTC
Update
The team continues to work through the services and remediate issues. At this time, we are working to mitigate issues within all pipeline service clusters, and we're continuing our effort related to making the pipeline fully operational in the second availability zone.
Some users may be starting to see updates to their consoles, though it may take time for users to see 'current' data as the backlogged data is currently being processed.
Users may still be experiencing pipeline latencies, as well as intermittent issues loading and leaving comments and loading the activity feed. Logging out and back into BigPanda may allow users to see the comments again.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 13, 2021 - 02:51 UTC
Update
The team continues to work through the services and remediate issues. At this time, we are working to mitigate issues within all pipeline service clusters, and we're continuing our effort related to making the pipeline fully operational in the second availability zone.
Some users may be starting to see updates to their consoles, though it may take time for users to see 'current' data as the backlogged data is currently being processed.
Users may still be experiencing pipeline latencies, as well as intermittent issues loading and leaving comments and loading the activity feed. Logging out and back into BigPanda may allow users to see the comments again.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 13, 2021 - 02:17 UTC
Update
Our Cloud Services Provider has repaired all affected hardware.
The team continues to work through the services and remediate issues. At this time, we are working to mitigate issues within all pipeline service clusters, and we're continuing our effort related to making the pipeline fully operational in the second availability zone.
Some users may be starting to see updates to their consoles, though it may take time for users to see 'current' data as the backlogged data is currently being processed.
Users may still be experiencing pipeline latencies, as well as intermittent issues loading and leaving comments and loading the activity feed. Logging out and back into BigPanda may allow users to see the comments again.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 13, 2021 - 00:48 UTC
Update
Our Cloud Services Provider continues to restore service to impacted components.
EC2 instances and EBS volumes were affected by an increase in relative humidity within a single Availability Zone. They have restored the relative humidity to normal levels are are working to recover the affected hardware.
We are anticipating being back to service with our Cloud Services Provider within the next 1-2 hours, possibly sooner.
In parallel, the team continues to work through the services and remediate issues. At this time, we are working to mitigate issues within all pipeline service clusters, and we're continuing our effort related to making the pipeline fully operational in the second availability zone.
Users may be experiencing pipeline latencies, as well as intermittent issues loading and leaving comments and loading the activity feed. Logging out and back into BigPanda may allow users to see the comments again.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 12, 2021 - 23:49 UTC
Update
Our Cloud Services Provider continues to restore service to impacted components.
EC2 instances and EBS volumes were affected by an increase in relative humidity within a single Availability Zone. They have restored the relative humidity to normal levels are are working to recover the affected hardware.
We are anticipating being back to service with our Cloud Services Provider within the next 1.5-2 hours, possibly sooner.
In parallel, we're continuing our effort related to making the pipeline fully operational in the second availability zone.
Users may be experiencing pipeline latencies, as well as intermittent issues loading and leaving comments and loading the activity feed. Logging out and back into BigPanda may allow users to see the comments again.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 12, 2021 - 22:42 UTC
Update
Our Cloud Services Provider has informed us that they've restored service to many impacted components.
We are anticipating being back to service with our Cloud Services Provider within the next 2-3 hours, possibly sooner.
In parallel, we're continuing our effort related to making the pipeline fully operational in the second availability zone.
Users may be experiencing pipeline latencies, as well as intermittent issues loading and leaving comments and loading the activity feed. Logging out and back into BigPanda may allow users to see the comments again.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 12, 2021 - 21:55 UTC
Update
Our Cloud Services Provider is currently experiencing issues and we are working directly with them to remediate the issue.
The team is working to add more nodes to the unaffected Availability Zone in our Provider's impacted region. Once those nodes are operational, the team intends to remove the nodes in the impacted Availability Zone from the cluster.
There are BigPanda services in a degraded state, and the team is working to add more nodes to affected service clusters in an effort to reduce load and process backlog once services return to normal operations.
Users may be experiencing pipeline latencies, as well as intermittent issues loading and leaving comments and loading the activity feed. Logging out and back into BigPanda may allow users to see the comments again.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 12, 2021 - 21:27 UTC
Update
Our Cloud Services Provider is currently experiencing issues and we are working directly with them to remediate the issue.
The team is working to add more nodes to the unaffected Availability Zone in our Provider's impacted region. Once those nodes are operational, the team intends to remove the nodes in the impacted Availability Zone from the cluster.
There are BigPanda services in a degraded state, and the team is working to add more nodes to affected service clusters in an effort to reduce load and process backlog once services return to normal operations.
Users may be experiencing pipeline latencies, issues loading and leaving comments, as well as loading the activity feed.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 12, 2021 - 20:52 UTC
Identified
We are continuing to investigate the issue and explore mitigation options.
Our Cloud Services Provider is currently experiencing issues and we are working directly with them to remediate the issue.
There are BigPanda services in a degraded state, and the team is working to add more nodes to affected service clusters in an effort to reduce load and process backlog once services return to normal operations.
Users may be experiencing pipeline latencies, issues loading and leaving comments, as well as loading the activity feed.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 12, 2021 - 20:01 UTC
Update
We are continuing to investigate the issue.
Our Cloud Services Provider is currently experiencing issues and we are working directly with them to remediate the issue.
There are BigPanda services in a degraded state, and the team is working to add more nodes to affected service clusters in an effort to reduce load and process backlog once services return to normal operations.
Users may be experiencing pipeline latencies, issues loading and leaving comments, as well as loading the activity feed.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 12, 2021 - 19:31 UTC
Update
We are continuing to investigate the issue.
In addition to pipeline latencies, users may experience issues loading and leaving comments as well as loading the activity feed.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.
Posted Jun 12, 2021 - 18:48 UTC
Investigating
We are currently seeing a delay for some customers affecting the BigPanda pipeline and are investigating the matter to identify the cause of the issue.
Additional updates will be shared as more details become available.
We apologize for any inconvenience that this may be causing.