Lessons Learned from a Cyber Incident Part 2: Common Questions and Lessons

Part of a blog series from the CoSN’s Cybersecurity Committee in preparation for and tied to Cybersecurity Month.
Continued from: Lessons Learned from a Cyber Incident Part 1: The Incident & The Aftermath

As the Director of Technology, I have recent, first-hand experience managing the incident response of a cyber event. We ultimately took a series of steps to minimize the incident, reduce its impact, and return our system to a normal state. We will continue incorporate what we have learned from this event into our cybersecurity practices.

Common Questions

As I’ve shared much of these details in a number of venues over the course of this year, several questions continue to arise, which I have answered below.

Do you know how the threat actors accessed your environment?
I can’t specifically address this for security purposes.

How did district leadership respond?
District leadership has been very supportive throughout the incident; this had a direct impact on the scope of our incident.

Who supported your response and recovery?
Our incident response team included the following staff: Technology Director, IT Support Manager, IT Infrastructure, Manager, Information Systems Manager, Senior Systems Administrator, Systems Analyst, Desktop Engineer, Project Coordinators, Assistant Superintendent of Secondary/Operations

I interacted regularly with our Superintendent, Executive Director of Business & Finance, Communications Manager, Digital Learning Manager, and Safety Coordinator

We contacted our insurance pool, FBI, and CISA.

Our ongoing incident support was provided by the district’s cyber insurance carrier, data privacy counsel, forensics consultants, and incident recovery consultants. We’ve leveraged additional partners for e-discovery and notification.

Did you follow an Incident Response plan? Is it being updated as a result?
We did not lean on an incident response plan throughout our incident. Our technology leadership team had completed CoSN’s Cybersecurity in K12 Course in the fall of last year, and that gave us all a good foundation and shared vision for how we would manage our incident. We also have a greater understanding now that whatever we document in an incident response plan needs to allow us to be nimble to the situational factors.

How much of day-to-day work was impacted by the incident?
For IT staff, roughly a month was heavily disrupted. Ultimately it was a bit more or a bit less for our staff based on their role.

For the district at large, it was about 3 weeks before all staff and all students were back into their systems with all major systems restored.

The last system restoration activity took place roughly 7 weeks after the incident began. This is also approximately when our elementary student login was reverted to passwords from badge authentication.

What worked well during the incident response?
Our people and processes worked very well. The support from district leadership was a strength, and the way in which we thoughtfully navigated the key decision points aided in our successful recovery as well.

Lessons Learned

Proactive Cybersecurity

1. The first and most important recommendation is to maintain a sense of urgency around this work. Identifying the projects that will most significantly reduce the risk to an organization, and even having plans or the funding in place to get that work done, is not enough. Nothing short of implementing the safeguards that have been identified as important to the cyber security posture of the environment is adequate.
2. In addition, constantly reevaluating each component of the cybersecurity system – even those that are doing their job well – against industry standards, can help reduce the potential for an incident. Business impacts should not be the primary driver for when substantial changes that will improve an organizations’ preparedness for a cyber incident can be completed.
3. Prioritize items that will have the most significant impact (reducing risk) and if those are unknown, find a partner who can help to assess and prioritize. Engage with a trusted cybersecurity consultant, work in partnership with them to evaluate the organization, and make the most of their recommendations.
4. It is no secret that regular cybersecurity education for all staff is a valuable way to increase awareness about cybersecurity threats. The amount of training that can be offered or required will depend on the comfort level of the organization. Each training is an opportunity to help staff understand that they are a key part in the district’s proactive cybersecurity efforts.
5. Take advantage of the experiences and challenges faced by other organizations to further stress the importance of the cybersecurity initiatives that are underway. Sharing timely details about significant cybersecurity incidents with members of the district leadership team or opening a monthly training session with a link to an article about major cyber threats affecting school districts can be very effective ways to build a shared understanding of why this work matters.

Incident Response

We had tremendous support from district leadership from the get-go, which was a substantial factor in reducing the impact of the incident. We also took some early actions that minimized the duration and severity of the incident. In addition to these, we quickly identified the right group of staff to serve as active members of the incident response.

1. The members of an incident response team will vary based on the situation. The incident may also require adding or removing members, as a direct response to the scenario being dealt with. Upon recognizing that a cyber incident was identified and pulling together the incident response team, we compiled a mix of technical expertise, project coordination, and leadership. In addition to these, it is important to have adequate district stakeholder input. In some cases, the technology leader serves in a cabinet-level role and brings this to the table. In our situation, the Assistant Superintendent overseeing Technology was included and played an active, supportive role throughout the incident response.
2. Most organizations rely on third parties throughout incident response. These may include legal expertise, forensics analysis, recovery support, or other roles. If at any time, a necessary role is not adequately represented, don’t wait to address the gap! Once you have the right partners, trust them, and lean into their expertise. For most district leaders, responding to a cyber incident is a very rare occurrence. That’s not the case for the threat actors involved, so relying on a team of partners that can bring additional expertise is one way to level the playing field.
3. In the heat of the moment, when the incident is actively being mitigated, the circumstances can be overwhelming. Take advantage of each decision to minimize exposure, reduce risk, and ultimately move from incident response to recovery.
4. Don’t make hasty decisions simply for the sake of moving forward. Encourage team members to speak up with any outstanding questions or concerns as you march toward recovery. At the expense of a quick restoration of services, and to prevent what could be a much more substantial issue, take your time and thoroughly evaluate each decision.

Author: Chris Bailey, Technology Director, Edmonds School District (WA)

Date: October 17, 2023

CoSN is vendor neutral and does not endorse products or services. Any mention of a specific solution is for contextual purposes.