8 Lessons Austin Business Owners Should Learn from the CrowdStrike Disaster

It was essentially what everyone expected Y2K to be, over 24 years ago.

The world’s biggest (so far) IT catastrophe forced hospitals to cancel surgeries, shuttered 911 call centers, crippled banks, and left travelers stranded at airports.

It wasn’t a cyberattack, and it didn’t come with a warning, but it had a massive impact on the world and is causing business owners and decision makers to think a little differently about their technology. Let’s talk about the CrowdStrike outage and what we all need to take away from this disaster.

What Caused the CrowdStrike Outage?

We’re going to keep this pretty short and simple. CrowdStrike is a popular suite of enterprise cybersecurity tools. The company has been around since 2011, and they’ve had a pretty big impact on the cybersecurity industry. They are definitely one of the heavy-hitters with over 29,000 business customers, and over 500 of those customers are on the Fortune 1000 list. 

Shortly after midnight on Friday, July 19th, CrowdStrike pushed a routine update to its CrowdStrike Falcon cybersecurity platform. There was a bug in the update that caused Windows devices to crash. The crashes appeared to be a generic blue screen of death, and it wasn’t immediately apparent that the issue was due to the CrowdStrike update. At first, many thought it was some sort of Microsoft issue.

The update was only available for 78 minutes before CrowdStrike took it down. In that short amount of time, it was installed on 8.5 million devices. 

It wasn’t a cyberattack, it wasn’t malware, and as far as we know, there was no malevolence behind this massive disaster. It was just a mistake.

It’s estimated that the insured losses from the CrowdStrike outage could be over $1.5 billion.

This disaster was impactful enough that the world will likely be talking about it for months or even years, and we will likely see legislation put into place to try to soften the blow of future disasters like the CrowdStrike outage.

Now that you are caught up, let’s take a look at some lessons we can all learn from the CrowdStrike outage.

Lesson 1 - Disasters Happen Fast

CrowdStrike Only Took Minutes to Cripple Massive Businesses
The fact that this all happened in the middle of the night with a routine, automated update that was only available for 78 minutes is a huge indicator of just how quickly a problem can escalate. For some organizations, like hospitals and airlines who are more-or-less operating 24/7, this meant waking up IT technicians who weren’t on shift to jump in and figure out this catastrophic issue. Other businesses would open a few hours later on Friday morning and learn that their computers were crashing. 

The cost of downtime, especially when it affects most or all of your endpoints, is extremely high. The more a problem shutters your business, the more expensive it is.

Lesson 2 - Business IT is Extremely Fragile

Not just business IT, but all of this technology that we use every day is extremely delicate, just not necessarily in a physical sense (although that too). Technology evolves because we continuously stack new innovations on top of old ones. It’s like a big Jenga tower. Sometimes we can replace some of the pieces at the foundation, but that’s usually very challenging. The parts of the tower that grow the fastest rely on the technologies that they are stacked on top of.

In fact, related to the CrowdStrike outage, Southwest Airlines was allegedly able to scrape by because some of their systems were still using Windows 3.1. Windows 3.1 was released in April of 1992. Back in 2015, an airport in Paris was forced to shut down because they were using the ancient Windows 3.1. This prehistoric operating system was still in use in Paris because a critical program used by air traffic controllers to communicate weather information to pilots wasn’t compatible with modern operating systems. 

Another example; a vast majority of financial transactions are handled with COBOL, a computer programming language developed in 1959. If you haven’t heard of the term COBOL, you aren’t alone. The computer programming language is so old that there are only a handful of COBOL programmers left in the world, but the world continues to rely on it.

This sort of thing always reminds me of a 2020 comic by XKCD:

Lesson 3 - Always Have a Backup and Test it Regularly

While most business backup solutions wouldn’t necessarily have helped with the CrowdStrike outage, they do serve as the last line of defense against some of the worst types of disasters, such as data loss, ransomware, natural disasters, hardware failures, and user error.

If you don’t test your data backup, you won’t be able to guarantee that it will be there for you when you need it. It’s critical that you test it regularly and ensure that your business can survive off of the data that is being backed up. 

The biggest mistake businesses make regarding their backup solution is not understanding what is getting backed up and what isn’t.

Lesson 4 - Train Employees to Spot and Report Anything Suspicious

While many disasters spread very quickly, having diligent employees who quickly report issues and oddities can result in faster resolution times, and could potentially stop a disaster from occurring. Again, we’re talking a little bit about cybersecurity, which the CrowdStrike outage falls outside of, but once a device starts acting strangely, it’s a good time to get IT on the horn.

This goes double for odd emails, strange behaviors in apps, missing data, and other anomalies. These days, there’s no problem too small that shouldn’t be reported to IT.

Lesson 5 - Being Proactive is Everything

While CrowdStrike itself has a pretty clean history, a disaster like this would have been a lot worse if the vendor wasn’t as diligent or responsive to the issue. We just recently saw a major outage in the car dealership industry. Software provider CDK Global provides dealership software for around a fifth of all car dealerships in the nation. They suffered from two cybersecurity incidents that left car dealerships struggling for about two weeks. This was a big deal for car dealerships, because it happened just before the 4th of July weekend and at the end of a month when financials were due. Depending on the state, it also made it harder for vehicles to get registered at the DMV. 

While coverage of the CDK cyberattack hasn’t been very comprehensive considering the scope, it’s pretty likely that the vendor wasn’t able to restore from a backup, and had to pay the cybercriminals a $25 million ransom to restore business.

That’s not a very good look. 

Similar to the CrowdStrike Outage, the CDK cyberattacks are estimated to have cost the car industry up to $1 billion.

Lesson 6 - Understand the Value of Your IT

One of the big points we’ve been trying to hammer in is just how expensive all of this is. For a smaller business, it’s hard to compare your ledgers with numbers in the billions of dollars, but smaller businesses also have a lot more to lose when disasters strike.

Some of these losses are hard to measure, but they have a major impact on your organization, such as:

• Loss of productivity and downtime - This is the obvious one, but if nobody can work for a day, a week, or a month, due to some sort of disaster, you still have to dish out the paychecks. That’s a lot of money down the drain.
• Loss of potential revenue - How many opportunities did you miss during the downtime? How many customers didn’t call back? How many potential sales did you miss? You should also think about how much time your staff spends struggling to get back up to speed before things go back to normal… it’s a lot longer than you think.
• Loss of reputation - Disasters can impact your clients and their opinions of your organization. If you miss an order, lose customer data, or can’t provide your services in a timely fashion due to an outage or cyberthreat, some customers simply won’t give you a second chance.
• Loss of morale - How strenuous was the disaster on your staff? Is it impacting their bonuses, raises, or growth within the company? Are they dealing with putting out fires? This can take a toll on a workforce.

Like it or not, most modern businesses absolutely rely on their technology to operate. Even if you can run your business without computers, networks, and fancy software, the speed and effectiveness that your business runs with technology is exponentially higher because of it.

Lesson 7 - Look into Business Insurance and Cybersecurity Insurance

There’s never been a better time to review your business insurance and fully understand what it covers, look at your options, and make sure that you are protected from outages that are outside of your control. It’s also recommended to look at cybersecurity insurance, as it’s not a matter of if, but when you’ll suffer from a major cyberattack.

These days, most business insurance providers are already asking questions about an organization’s cyber protections, and they are making decisions on what to cover and how much to charge based on how well your business is protected. 

Lesson 8 - Partner with a Managed IT Provider you can Trust

The real heroes of the CrowdStrike outage were the dedicated IT professionals who logged in at 3am Friday morning to the world’s biggest IT disaster. They worked tirelessly through the night and in many cases, throughout the weekend, to bring the world back to normal.

It wasn’t their fault, but they did the hard work so planes could fly and hospitals could treat patients and business could go back to normal.

Your business needs to have a trusted IT provider that it can call on for both the big and small issues. It’s best to be proactive, which is why Capstone Works is one of the best choices for Texas businesses. We act as your internal IT department, with a strong focus on preventing issues and sustaining normality. Your staff can get the technical support they need, and your decision-makers can work with our professional IT consultants to ensure that your technology aligns with your business goals to help your business be as effective as possible.

Get a Handle on Your IT So Your Business Isn’t Left in the Dark

Technology is only going to get more complicated. That’s why Texas businesses trust Capstone for all of their IT. We are like having a dedicated in-house IT department that truly understands your business and prioritizes your technology to meet your goals. To get started, give us a call at (512) 343-8891.