Update - We are continuing to monitor for any further issues.
Apr 18, 2024 - 05:20 UTC
Update - Following our latest notice on Saturday April 13th, 2024, we are writing to provide our Customers with an update on the latest available information regarding the Sisense Data Breach.

On Tuesday April 16, 2024, BigPanda was informed by Sisense that the Breach involved Sisense data cubes.

With this new information, we are finalizing a process to export the data that was compromised in the Sisense cubes and provide it to our Customers. We expect this process to be complete by the end of the week, at which time we will be able to schedule individual times upon request.

Our investigation is still ongoing. For any additional requests or to arrange further discussion, please contact the BigPanda Trust & Security team by email at trust@bigpanda.io.

Apr 18, 2024 - 05:19 UTC
Update - BigPanda became aware of a Data Breach through its subprocessor Sisense on Thursday April 11th, 2024. We informed Customers that same day of the Breach via the BigPanda Status Page (status.bigpanda.io). We are writing now to notify our Customers and provide an update on the latest available information.

Incident Summary
A Threat Actor compromised the Sisense platform and gained access to BigPanda Customer Data through Snowflake. To our knowledge, the situation has been contained. Our investigation indicates that at no time did the Threat Actor gain access to BigPanda’s Core Platform.

BigPanda’s Use of Sisense and Snowflake
The BigPanda Core Platform stores metrics data in Snowflake, which is then loaded into Sisense to deliver metrics reporting via Unified Analytics.

Incident and Data Scope
Sisense has confirmed the Breach but has yet to provide detailed information about the incident.

After BigPanda’s internal investigation, it was confirmed that the Threat Actor accessed BigPanda Customer Data stored in Snowflake via Sisense. It was determined that a read-only service account credential for Snowflake was compromised due to the Breach, thus enabling the threat actor to make SQL queries on the Snowflake database.

The BigPanda Customer Data accessed included usernames (which contain BigPanda customer email addresses) and alert and incident tag information. This data in Snowflake enables customers to identify trends within their environments, examples are included below:
Username (email address), app_id, source_host, priority, start_time, end_time, date, etc

Incident Containment & Remediation
Upon becoming aware of the incident, the BigPanda Security Incident Response Team immediately launched an investigation, which included execution of the following steps:
- Requesting information and contacting Sisense;
- Reviewing all internal systems audit logs for abnormal activity;
- Performing programmatic updates to rotate user access keys and BigPanda’s SSO token via prepared scripts;
- Rotating service account keys for systems that feed data to Sisense cubes.
- Stopping any new Customer Data from being sent to Sisense. We will re-evaluate this decision in the days to come based on information that will be provided to us from Sisense.

A more detailed list of the actions taken by the BigPanda Security Incident Response team to ensure the containment of the Breach are as follows:

- Changed all Sisense-related passwords on my.sisense.com
- For non-Single Sign-On (SSO) access:
-- Replaced the Secret in the Base Configuration Security section with your GUID/UUID.
-- Reset passwords for all users in the Sisense application.
-- Logged out all users by running GET /api/v1/authentication/logout_all under Admin user.
- For SSO access:
-- Updated sso.shared_secret in Sisense and updated the newly generated value in the SSO handler.
-- Rotated the x.509 certificate in our SSO SAML identity provider.
-- Rotated the OpenID client secret.
-- Updated SSO settings in Sisense with the revised values.
-- Logged out all users by running GET /api/v1/authentication/logout_all under Admin user.
-- Reset credentials in the database used by the Sisense application.

Ongoing Investigation and Mitigation
BigPanda is continuing to investigate the issue and will conduct a full external forensic investigation of its Data Platform. We will continue to provide updates as they become available.

For any additional requests or to arrange further discussion, please contact the BigPanda Trust & Security team by email at trust@bigpanda.io.

Apr 13, 2024 - 20:15 UTC
Monitoring - We want to inform you that we have identified and stopped some suspicious activities from an unidentified threat actor. Pursuing our audit of this security incident, we were able to identify there was read-only access to some user data.

This incident follows a security incident experienced by one of our partners, Sisense. We wanted to let you know that BigPanda takes this matter extremely seriously. We have started a detailed investigation immediately to understand the full extent of what is happening. Our internal investigation determined the data compromise did not include operational data and was limited to analytical data used for our Unified Analytics offering.

As an immediate remediation, we updated all the user access keys and credentials for all our systems. This has terminated access for the threat actor and prevents further intrusion from the recent exploit. We monitored and confirmed there has been no threat actor activity after the user access keys were changed.

The threat actor was able to query a list of usernames, which are email addresses. There is no indication that customer API Keys were compromised since all API Keys are stored in a separate secured environment, which was not compromised as part of this event.

All user access credentials were rotated as of April 11th, 1:45 PM Pacific, and we have completed Sisense recommended procedures.

For any additional requests or to arrange further discussion, please contact BigPanda Security team by email trust@bigpanda.io.

Apr 12, 2024 - 07:08 UTC
Inbound Integrations and Event Processing Operational
Data Consumption Operational
Alert Enrichment Operational
Alert Filtering & Maintenance Plans Operational
Incident Enrichment & Environments Population Operational
Correlation Operational
Outbound Collaboration Operational
Incident Sharing Operational
Outbound Integrations Operational
ETL Pipeline Operational
Console Functions Operational
Login Operational
Incident Feed Operational
Incident Feed Search Operational
Automatic Incident Triage Operational
Root Cause Changes Operational
Administrator Screens Operational
Incident Actions Operational
Incident Activity Feed Operational
Unified Search Operational
Unified Analytics Operational
Integration Diagnostics / Troubleshooting Operational
APIs Operational
Tier 1 APIs: Automation & Event Processing Operational
Tier 2 APIs: Supplemental Operational
Tier 3 APIs: Tertiary Operational
Third Party Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Apr 18, 2024

Unresolved incident: Update on Sisense Customer Data Compromise.

Apr 17, 2024

No incidents reported.

Apr 16, 2024

No incidents reported.

Apr 15, 2024

No incidents reported.

Apr 14, 2024
Completed - The scheduled maintenance has been completed.
Apr 14, 11:50 UTC
Verifying - Verification is currently underway for the maintenance items.
Apr 14, 09:44 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 14, 07:30 UTC
Scheduled - In support of our commitment towards improving our services and infrastructure, we will be performing the following maintenance to the BigPanda platform.

Start Time: April 14, 2024 07:30 UTC
End Time: April 14, 2024 13:30 UTC
Duration: 6 hours

What's Happening?

During this maintenance, we will be upgrading our Mongo cluster from v5.0.24 to v6.0.13.

Component(s) undergoing maintenance:
• Pipeline
• Inbound Integrations and Event Processing
• Outbound Collaboration Services - Incident Sharing, Outbound Integrations, ETL Pipeline
• APIs

What's the Impact?

During the maintenance window - as most BigPanda services use Mongo, customers may expect up to 45 minutes of latencies within the BigPanda Pipeline services while we initiate service restarts to remediate experienced connectivity issues.

We will monitor our pipeline closely as changes are deployed.

Is User Action Required?

No user action is required.

The planned maintenance has successfully completed our QA validation and regression testing.
We will be closely monitoring the maintenance work, and BigPanda’s status page will be updated throughout the maintenance period with any changes to service operations.

We apologize for any inconvenience that this may cause. If you have any questions, please reach out to BigPanda Support via our live in-app chat or via support@bigpanda.io.

Apr 5, 21:06 UTC
Apr 13, 2024
Apr 12, 2024
Apr 11, 2024
Completed - The scheduled maintenance has been completed.
Apr 11, 15:36 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 11, 15:00 UTC
Scheduled - We want to inform you that we will be performing the following maintenance to the BigPanda platform:

Start Time: April 10, 2024 15:00 UTC
End Time: April 10, 2024 18:00 UTC
Duration: 3 hours

What's Happening?

During this maintenance, BigPanda will be performing updates to align access controls with our Zero Trust Network Architecture initiative. Specific security groups will be updated to improve operational access restrictions.

Component(s) undergoing maintenance:
• cloud infrastructure

What's the Impact?

No user impact

Is User Action Required?

No user action is required.

We will be closely monitoring the maintenance work, and BigPanda’s status page will be updated throughout the maintenance period with any changes to service operations.
We apologize for any inconvenience that this may cause. If you have any questions, please reach out to BigPanda Support via our live in-app chat or via support@bigpanda.io.

Apr 8, 19:36 UTC
Apr 10, 2024
Resolved - This incident has been resolved.
Apr 10, 15:24 UTC
Monitoring - A fix has been implemented and we are monitoring the results.
Apr 10, 15:01 UTC
Apr 9, 2024

No incidents reported.

Apr 8, 2024

No incidents reported.

Apr 7, 2024

No incidents reported.

Apr 6, 2024

No incidents reported.

Apr 5, 2024

No incidents reported.

Apr 4, 2024

No incidents reported.