Resilience: Insights from Live and Simulated Incident Response

Technology, by itself, is hardly ever the complete solution to a challenge. Complete solutions always involve shifting culture. Hacking machines is easy. Gratefully, for now at least, hacking people – not so much. Shifting culture successfully towards better practices requires finesse, empathy, and a commitment to validating people in an effort to move them forward.

Resilience is way of perceiving the world. It is the practice of protecting and empowering our organizations to be prepared to unplanned events of all kinds that will compromise our productivity, reputation, and bottom line. Some of these include natural disasters, cyber attacks, insiders and third-party threats, even file corruption and data loss.

While Resilience can only truly be tested in the midst of these events and in the forensic hours, days, and weeks following them, simulations help define its strengths and weaknesses in quantifiable and less quantifiable ways, saving our precious time, money, resources, and reputations. The less quantifiable but critical assets include concrete assets like trust, morale, and confidence levels on these Resilience Teams. There’s no understating how valuable these components are, even in average day-to-day operations. They are especially important in a crisis and often make the difference in the outcome when it matters most.

What follows are some insights gained from live and simulated exercises, giving internal Resilience Teams a unique opportunity to test and build deeper fluency and familiarity with each respective Resilience Strategy we’d designed and implemented.  While there are alternatives to live exercises, such as table-top ones, any type of exercise offers value in the context of practical and memorable learning to teams prior to responding to an unexpected incident. For the sake of discretion, specifics have been omitted.

It’s appropriate to say, however, that these groups literally celebrated how elevated and confident they became during and after these, both as teams and as individuals. They grew more competent in their understanding of how their business operates, its vulnerabilities, they improved their definition of the crown jewels and how to protect them, they value their own work more, as well as feel prepared – empowered – to protect their livelihoods when something unplanned happens.

These exercises are always empowering for me, too, being trusted enough to empower teams with practical tools and strategies they can put to use right away without adding additional complexity. Memorable, to say the least.

learning by sharingInsight #1: First of all, it is an understatement to say that access to these insights is rare. The importance of sharing these experiences and the lessons learned with organizations in helping spread literacy and preparation for their own incidents cannot be overstated. The task of researching post-mortems on unplanned events that compromise our productivity, reputation, and bottom line, especially cyber attacks, isn’t easy. Most organizations choose extreme discretion, even internally, and miss huge opportunities to share the lessons learned. While we can agree a certain level of discretion and privacy is appropriate to prevent creating additional risk, let us not overlook the fact that not sharing the insights we gain from these experience in appropriate ways arguably creates even more.

While I cannot argue with the merits of discretion, unfortunately this obscures much of the insights and specifics that could be gained, including common attack vectors (typically social engineering), defensive and offensive hardware and software in place during attacks that provide varying degrees of value, business impact and the outcomes and lessons learned from these inputs and diversity of controls. While there’s a significant amount of general information available on the internet about theoretical scenarios, it is only usually speculative due to it being unusual for organizations to be transparent (even appropriately so) about these types of situations. Research is often fruitful only in a general or vague sense. For this reason, it can take some real time to begin to synthesize an approach that aligns with the specific culture we are working to build a business case for.

Keep in mind, though, that being tight-lipped about breaches and how organizations handle them is becoming more and more challenging as the laws are changing, requiring more and more transparency. Be prepared to be open about real breaches. It will serve you well in the end, especially if you’re prepared. Openness can be a most powerful PR tool and create opportunities that even the most elaborate, globally integrated marketing strategy or ad campaign cannot rival.

On the one had, these exercises need not be so transparent in the context of publishing them in so much detail. It’s not advisable. Doing so can reveal specifics about an organization’s technology environment that may be used against them. However, internal exercises and training opportunities are their own rewards by elevating and uniting teams arguably like no other activity can.

Which leads me to discuss yet another context and arguably the most important part of this number 1 item: be transparent about these exercises with partners, vendors, and clients. Include as many of them as you can. Make them feel invested in the process in intentional ways. Doing so will build trust and confidence. Pure gold. Because when the real show comes to town, the first things partners, vendors, and clients do is assign blame. They want accountability. By including these components of your ecosystem in your live or tabletop exercises, your teams, clients, and vendors not only find some quick wins in the value of such work but have some fluency built-in when it matters most, when incidents occur, and see you as a victim and not part of the problem as they otherwise likely would.

My own clients are sick of hearing me say it: trust cannot be built in a better way, especially in this age of consent, of privacy and security breaches. Include the people who power your business in these exercises in intentional ways. Always. You will only regret it if you don’t.

At the same time, I have worked with organizations that have gone through exercises without proper guidance and found only frustration with too much ambiguity and/or a too rigid, highly-technical focus. It’s important to recognize there are some optimal ways to carry out these exercises and some less-optimal ways. It’s important the team feels successful in their efforts, too, in order to foster a sense of achievement and an elevated understanding of the environment they are being trusted to protect and respond on behalf of when things happen.

It is helpful to have some great tools for the team to use. For example, there are many great frameworks available, such as NIST’s ‘Computer Security Incident Handling Guide’, which provides a friendly balance of technical guidance and communication strategy that will be useful to organizations of any size or level of complexity. Build your own version from this friendly framework, using tools and language that resonates with your own team.

learn togetherInsight #2: Keep in mind it’s not essential that you or anyone on your team is a specifically a security expert. Exercises are about exposure and building familiarity with the processes and procedures of handling incidents. The more familiarity and confidence we and our teams have, the more effective we will all be when the real show comes to town. Resilience is about empowerment. It’s about identifying tools and strategies that make us feel empowered and confident to be prepared for anything.

During the exercises I’m speaking of, there were more non-technical people than technical ones. This was an intentional choice. The goal was to elevate everyone’s understanding of how the business uses technology, not just the engineers, developers, and IT support staff.

If you’re the business leader, your focus during the first 24 hours of an incident will be about strategic leadership. All the hidden machinery, the technical components of these exercises, will be carried out by your technical team, so let them focus on those details. Your focus won’t be about the attack itself or describing how it happened as much as the potential consequences of it. The rest will come but it’s important not to release inaccurate information too soon by making presumptions before facts can be verified. Conversely, it’s also important to be prepared enough in advance that you’re not releasing information too late to make it look like you’re dragging your feet. It’s a delicate dance, which is why these exercises are so valuable. And advisable.

GDPRInsight #3: Resilience promotes good data protection. Period. It’s crucial to be prepared in advance to present what information security your organization has in place, including any specific standards that guide your due diligence, such as ISO 27001, GDPR, HIPAA, NIST, etc. That way, during the breach, you and your team are quick to expedite the processes required to gather necessary forensic data while responding to external stakeholders and internal teams. Such preparation makes it easier to illustrate how your organization takes data integrity and information security seriously, that your Resilience program is audited internally and externally following industry practices.

During our exercises, teams uncover ways they could quickly improve current processes and workflows in simple, friendly ways. Some of these make sense from a productivity context, too, reducing complexity and time-to-deliver.

Insight #4: It’s critical to visualize how news of a data breach will be disseminated within your organization. How will you notify and brief the Resilience Team? Keep in mind: people will talk and spread misinformation. Quickly. There’s always a threat to morale when leadership mishandles these sensitive incidents. What tactics, tools and strategies will be used to make sure complete and accurate information with a positive voice is quickly available to those responsible for managing the incident? How can we ensure that the right people are empowered with enough latitude to make decisions based on the information they receive in a timely manner? Who within our organization is responsible for internal and external communication in response to a cyber attack? Are IT and senior management aware of how an incident would be managed? What components of a successful response are likely to be overlooked the first time?

These are the kinds of questions that most commonly come up during exercises and will likely come up during your first one, too. They always do. Answer them now, rather than in the midst of a crisis.

Insight #5: Let’s imagine your organization’s confidential personnel data was compromised, such as salary information, bank routing, and account numbers, birth dates and other Personally Identifiable Information (PII) or Personal Information, as per the GDPR. This is something we’ll want to be sure to prepare for ahead of time by authoring communication information to our staff regarding the breach and what the next steps will be. Next steps will include informing them of our commitment to helping them minimize their risk of identity theft/fraud and giving them useful advice and even tools/resources to assist them with achieving that.

During our exercises it’s clear this information/communication needs to be poised, packaged in friendly, bundled ways, and more or less ready to go because it takes some time to collect the information and put a package together that is accessible the audience can use right away. The longer we wait after discovering a breach, the greater our risks of identity theft and additional liability, too, that we do not want.

Insight #6: Even if only a small part of our technology infrastructure is hosted and/or managed by a third party, think through how our technical and operations teams will collaborate with those service providers to respond to some of the most common incidents and perhaps some less common ones. For example, if our Dev Team uses a hosted code repository solution that gets breached, what is the contingency plan? If the breach is related to a third party provider of identity access management, what resources are poised to work around a situation wherein your Directory Services are not available? What happens if a hosted environment (UTA, QA or live production) is compromised? How much fluency do your Dev, Sec (if you have one) and Ops teams have to quickly mitigate these scenarios and get back to work? Are they documented? Have they tested them? If not, how do they know they will work when needed? Just a hunch?

Figuring these scenarios out step-by-step in the midst of an incident is not awesome or optimal. During our exercises, coming up even with just some suggestions took an unacceptable amount of time, let alone planning out concrete, step-by-step checklists to act upon. When time is of the essence and getting third parties up to speed isn’t always easy, that combination can make or break a successful outcome, especially where our reputations are on the line and clients may be impacted by systems that are not directly under our control. Thinking these things through is a big focus of every exercise.

Insight #7: What information security strategies, tactics, and tools does our organization have in place? Do we have a socialized Resilience Strategy? Is it accessible? Do we have lists of hardware, software, and training materials? Are we supporting our team with time and proper tools to help them adhere to good data protection standards, such as ISO, GDPR, HIPAA or NIST?

In case of a breach or potential breach, we will want to immediately expedite incident response processes in accordance with our Resilience Strategy, aligned with these laws and their definitions and expectations. This is where Resilience goes beyond mere compliance.

When we protect our data using good Resilience practices, our due diligence will stand up to any framework it is measured against. It will be easy for us to illustrate how data integrity and information security is taken very seriously by our organization when we can quickly provide these details demonstrating how our Resilience Strategy is designed to protect our data as well as that of our clients and partners and that it’s revisited on a regular, periodic basis to ensure its ongoing value to adapt to an ever-evolving threat landscape, changing privacy and data protection laws, as well as our businesses and our clients, as our businesses continues to grow and succeed.

Insight #8: In the midst of an incident we will need to inform our stakeholders who are affected. It’s advisable that we author general communication ahead of time that transmits a clear message that an incident has occurred, while showing some restraint about details. The goal is to keep communication flowing but not make any presumptions before the facts can be verified and conversely, make sure not to downplay the breach in case it’s worse than initially understood.

On the flip-side, if we wait too long to author and send something it can appear as if we are not being transparent, which can foster more distrust and tendencies towards blaming before all the facts are in.

This is another reason why these exercises are great platforms to have strategic discussions with our teams + our clients rather than during an incident. It demonstrates initiative, is a differentiator from a competitive perspective and builds confidence and trust all around. Who would you prefer to hire? An organization that shows no sign of pro-actively preparing for cyber incidents? Or one that takes you on a friendly and informative journey to understand how to respond ahead of time so that, when something happens, everyone is in it together? United. Incidents, regardless of size or scope, are stressful enough without adding the additional stress of being completely unprepared, untrusted and blamed by partners, vendors, and clients. As I’ve already said, all of that adds to the stakes by putting our culture’s overall morale at risk.

The teams in these exercises typically haven’t made time to think this through, yet. Have these messages prepared ahead of time in template format. Get ahead of it.

Insight #9: It’s easy to overlook and/or neglect to consider insider threats. For example, technical contractors (Developers, Designers, IT Staff) who have direct access and work directly on our critical systems are always potential threats. Something happens to them, well, it happens to us. If their device(s) get malware on a public network at an airport or from a home wireless network, they then, in turn, deliver it to our infrastructure, too. There’s also the occasional disgruntled worker who maliciously destroys data. Be prepared for this because many of these nightmare scenarios are completely preventable.

Be sure to vet our contractors and keep detailed records of who has had access to what, when, and for how long. Make sure that, in addition to on-boarding procedures, our off-boarding procedures are comprehensive and designed with our own and our clients’ best interests in mind in the context of protecting our data and that of our partners, vendors + clients. It will count.

It’s also a competitive differentiator. Brag about it. See what happens. Others will look to us for guidance. We want to be leaders every chance we get.

Insight #10: Communication during a real incident is key. Make sure our plans don’t only deal with the technical side of things or are only communicating with stakeholders. Our communication plan will need to deal with an incident from many angles, including contexts like insurance, legal, technical, internal/external communication, etc . In our simulations, the technical teams had great tools and processes in place for technical staff but left most everyone else out of the loop.

Make sure to consider everyone as each organization’s culture is unique and often overlooks at least one or two key things, as is often the case in our simulations. Rumors. Rumors, man. Only communication can solve the rumor problem. Rumors create more problems than you can imagine. Can you imagine some of the problems rumors can create? Imagine yourself as one of your clients. What would make you feel most comfortable and confident in the midst of such a crisis? Make that your guiding principal.

Insight #11: Last but not least, remember that prevention is valuable on many levels. It’s always easier, cheaper, and less stressful than cure. Often, something as simple as correct hardware or software configuration (such as a properly configured firewall, wireless network, or hosting environment) can prevent incidents before they occur. In our simulations and live responses, improperly configured network devices almost always lead breaches that could have otherwise been very easily avoided. Invest in training your staff. They will thank you for it and one day, when something happens, your clients will, too.

Most importantly: being pro-active is the best cultural armor. Being unprepared always puts morale at risk. That’s even harder to recover from than a data breach.

Parting Thoughts

Two things are crucial to eliminating preventable risks and minimizing our acceptable risks. Number one is building awareness in our organizations that reduces the likelihood they’ll fall prey to social engineering like clicking on a link in a spear phishing email – part of a larger campaign to use us in their wire fraud heist. Criminals appreciate it when we configure something incorrectly or otherwise make a mistake that puts the entire organization at risk. Resilience needs to be integrated into everything we do. Everything. Project management proposals, creative briefs, you name it, it has to be even just a mention in a five-minute window of that meeting.

A quick win for all of us is a better, more comprehensive understanding of our organization’s technology environment, by defining and prioritizing our biggest risks and most valuable components by establishing a well thought out Resilience Strategy. When our teams have a complete and accurate understanding of exactly how our business uses technology, its risk and priorities, we’re able to determine our organization’s tolerance for a broad spectrum of unplanned events, incidents of all kinds. From this very straightforward planning, our Resilience Strategy is as effective as it can be, able to adapt to our business as it grows and the threat landscape as it continues to evolve.

Thanks for reading. I hope this is valuable to someone out there. Good luck.