Random photos, comics, etc.

[Randy Stankey]

You're right. Sic the lawyers on 'em!

Just as you suggest, I'm sure that there are plenty of lawyers already knocking at their proverbial doors.

From a quick check of the internet, the company's net income for FY-24 was close to $90 million. Damage estimates for this incident are likely to top a billion! I know that finances don't work exactly like this but, from some thumbnail math, Crowdstrike has likely screwed themselves for the next ten years.

I certainly wouldn't want to be the guy who pushed the button that sent out that update!

[Leo Enticknap]

My second call would be to the guy in charge of CrowdStrike... The entire conversation would be something like, "We're done." <click>

I suspect that in many of these scenarios, the second call will be to an attorney, with his or her brief being to get medieval on CrowdStrike's ass. CrowdStrike's liability insurer has likely received a few calls by now, too.

[Martin McCaffery]

"Oops! My bad," is NOT an acceptable answer!

Fortunately, customer service is NEVER allowed to say the company made a mistake.

[Randy Stankey]

In your opinion, what would be an acceptable answer?

When I was managing the stage at Mercyhurst, I had to train new students to work in the loading gallery, 50 ft. above deck. I taught them how to load 20 kg., cast iron counterweights onto the appropriate line sets. The rule was that you NEVER held weights with only one hand. Always two hands. I explained that, if a weight falls fifty feet to the deck, equipment is going to be damaged and people could easily be killed.

I also told students, "There is no punishment for dropping counterweights." When people ask, "Why? What do you mean?" my answer is, "Because it doesn't happen!"

In the same respect, I can't think of an acceptable answer because it shouldn't have happened in the first place.

I'm not trying to be a hard ass. I don't expect things to be perfect all the time. Yes, I understand that people make mistakes. So do I.
All my life, since I can remember, people told me to work hard, to do things right, to try my hardest. All my life, I struggled to get things right. Come to find out, I have a learning disability that I didn't discover until I was over forty, almost fifty. However, I got through, okay, because I tried my hardest. I learned to cope with problems and work around them. I know how hard it can be but it pisses me off when people who don't have problems like I do slack off, fuck up and don't seem to care.

If I can learn to do things right when I have to try twice as hard just to be thought of as half as good, why can't other people, who don't have disabilities, do their jobs right the first time?

Airlines, at least, should have a separate data center of their own at each airport. If this country ever had a terrorist attack we'd be at a giant shand still again.

Yeah! You, me and a bunch of other people at FT have been saying the same kinds of things for how long?

How long will it be before we get another, "Oops! My bad!"

[Mark Gulbrandsen]

Frank, Southwest likely has their own data centers. I think you'll see more airlines building their own centers.

[Frank Cox]

"Oops! My bad," is NOT an acceptable answer!

In your opinion, what would be an acceptable answer?

Southwest is still flying high, unaffected by the outage that’s plaguing the world today, and that’s apparently because it’s using Windows 3.1.

This article is highly misleading. Southwest is still flying because they are not a Crowdstrike subscriber. Just like any other company that's not a Crowdstrike subscriber, regardless of whether they're using an obsolete version of Microsoft Windows, a current version of Microsoft Windows, or Linux or anything else.

[Mark Gulbrandsen]

Screw Crowd Strike! Airlines, at least, should have a separate data center of their own at each airport. If this country ever had a terrorist attack we'd be at a giant shand still again.

[Randy Stankey]

That was more of a rhetorical statement, of course.

Still, I would have my people looking into alternatives beside CrowdStrike and I would be calling CrowdStrike up to say, "You fucked up! What are you going to do about it?" I wouldn't accept any pat answers and I certainly wouldn't let them hide behind any flimsy terms in some EULA.

Their stupid and avoidable screw up cost just cost thousands of companies, uncountable millions of dollars that are far in excess of the money they charged for their software. Sure, people shouldn't allow software to be updated automatically but, when you're talking about critical infrastructure the stakes are a lot higher and everybody who's got skin in the game needs to take responsibility.

When it all comes out in the wash, though, the buck stops at Crowdstike's doorstep. They are the ones who, ultimately, need to make things right.

"Oops! My bad," is NOT an acceptable answer!

[Mark Gulbrandsen]

Randy,
There may not be an option for many of these system users to move, and no one updates millions of computers all at once, that is beyond stupid. It will take weeks, if not months to get all this mess cleaned up. End of story.

[Randy Stankey]

The bigger problem is that it's millions of computers that are all over the world...

No, the problem is that it happened on mission critical computers, all over the world.

I would agree that the chances of something like this happening are small... less than 1%. However, if something happens, the chances of people being hurt, killed or negatively affected in some way is 100%.

If computers at banks go down, people can't access money and their lives grind to a halt. If somebody needs that money for an important purpose, they could be harmed. Even if that doesn't happen, businesses could be negatively impacted.

If computers at the airlines go down, millions of people are affected or, in an extreme situation, people could be killed.

If this problem only happened to home computers, it wouldn't be such a big deal. In a situation like that, edge cases don't matter so much. However, if you're talking about critical infrastructure, edge cases DO matter!

It's common for people to decide how much time, effort and money to put into preventing problems based on the probability of something bad happening. That makes sense. People don't hire armed guards to protect their houses, 24/7 because the chances of being burgled are pretty low. However, I'm certainly going to have armed guards protecting the White House because, even though the chances of somebody getting in are relatively low, IF somebody gets in, it could cause problems for the whole country.

In other words, gauge the amount of trouble that something might cause to determine how much of your resources to put into prevention of a problem, not just the chances of it happening.

If I was the boss of a company that got affected by this malfunction at CrowdStrike, I'd be making two phone calls:

My first call would be to my the head of my IT department to tell them to get something else, beside Crowdstrike, immediately and forthwith.

My second call would be to the guy in charge of CrowdStrike... The entire conversation would be something like, "We're done." <click>

[Mark Gulbrandsen]

Leo, there are mini computers today, used for ticketing a lot in Cinemas, that could easily serve as local computers on a big network. I got rid of my big workstation several years ago and run an HP Z2 G6 Mini computer. It has most of the features of the larger workstations. There are also monitors from all the big manufacturers that have the Mini computer built in.

Now if it's a VM setup, that makes the fix far more difficult as you pointed out. I suspect that in the end computer techs are going to have to go to every site and do the fixes locally, then verify the fix.

What a nightmare!

[Leo Enticknap]

I'm sure those batteries are also used in ahem, "personal pleasure items" thus the category.

I can't claim to be any sort of an expert on this category of consumer tech, but I'd be surprised if one could get much personal pleasure out of 3v at a few millliamps. I would expect to find three or four 18650s in such a device at the very least!

The funny part is that the fix is uber easy to do.

From what I've been reading, the problem is now threefold. The fix (booting into safe mode, deleting a specific file, but which is in a Windows folder that is hidden by default, and then rebooting back into regular mode) is easy for someone who knows their way around Windows, but is just above the skill level of your average office worker. So in any given megacorp, a small crew of IT techs have to go around all the office PCs, fixing them manually, which will take some time. The second problem is that a lot of the affected devices are Windows running inside VMs, which can't be booted into safe mode. You would then need to connect the VM file or drive to a second VM in order to gain access to it to delete the file, which needs a proper techie. The third problem is that if the drive containing the affected Windows install (either physical or VM) is encrypted using Bitlocker, you need the key.

BTW, a somewhat surprising reason emerged why Southwest was almost unaffected by the Crowdstrike SNAFU:

Windows version from 1992 is saving Southwest’s butt right now

By Jacob Roach July 19, 2024 10:19AM

Nearly every flight in the U.S. is grounded right now following a CrowdStrike system update error that’s affecting everything from travel to mobile ordering at Starbucks — but not Southwest Airlines flights. Southwest is still flying high, unaffected by the outage that’s plaguing the world today, and that’s apparently because it’s using Windows 3.1.

Yes, Windows 3.1 — an operating system that is 32 years old. Southwest, along with UPS and FedEx, haven’t had any issues with the CrowdStrike outage. In responses to CNN, Delta, American, Spirit, Frontier, United, and Allegiant all said they were having issues, but Southwest told the outlet that its operations are going off without a hitch.

Some are attributing that to Windows 3.1. Major portions of Southwest’s systems are reportedly built on Windows 95 and Windows 3.1, which is something the company has come under fire for in the past several years. It should go without saying that Southwest needs to update its system, but in this case, the ancient operating system seems to be doing the airline some favors to avoid a complete Y2K-level apocalypse.

If you aren’t flying Southwest, you’re out of luck right now. Airports around the world had their scheduling systems crash in the wake of the CrowdStrike update, sending millions of travelers into a frenzy. The Federal Aviation Administration (FAA) said it’s working with several airlines on the outage. Thankfully, the FAA itself hasn’t been affected.

Microsoft, who has been at the center of this fiasco with CrowdStrike, says that the root cause of the issue has been fixed. It could take days before everything is sorted out, though. Microsoft’s CEO Satya Nadella commented on the issue on X (formerly Twitter), saying, “We are aware of this issue and are working closely with CrowdStrike and across the industry to provide customers technical guidance and support to safely bring their systems back online.”

That shows the scale of this problem. Microsoft has outages all the time, but none of them are worth commenting on from the CEO of the company. This is a different beast entirely, affecting millions of servers running on Windows. Southwest seems to have saved itself from any trouble by being woefully late to upgrade.

I'm guessing that their W3.1 systems must almost all be in VMs - there surely can't be many, if any, motherboards and processors made in recent years that are capable of running it natively.

[Mark Gulbrandsen]

Here is how you fix the problem.

[Mark Gulbrandsen]

I used to work for a Fortune 500 corporation as a Sr. Systems Analyst. We never allowed any software provider, including Microsoft, to automatically update our systems. We also never allowed any computer that was not owned by our company to even attach to our network. I suspect there will be many people fired for allowing CrowdStrike to update the affected systems. I wouldn't want to be an employee at CrowdStrike. It reminds me of the old quote, "'A sure way to empty a hotel fast: is to drop an elevator."

The funny part is that the fix is uber easy to do. The start point is the "Blue Screen Of Death Page". There are already "How To" videos on Instagram and Tic Toc.. Takes about 15 seconds to do, then just reboot the computer and you're back home... The bigger problem is that it's millions of computers that are all over the world...

[Ed Gordon]

It's good to have backup systems!

On system updates, there is a script that pushes updates to USL/QSC/MIT IRC systems. The script checks that the system comes back up again and reports the new version number before going on to the next unit. The script exits on the first failure so we don't end up with hundreds of dead systems.

Weren't there a couple of guys, here, at Film-Tech who always used to say that you should never accept automatic software updates but nobody seemed to want to listen?

I used to work for a Fortune 500 corporation as a Sr. Systems Analyst. We never allowed any software provider, including Microsoft, to automatically update our systems. We also never allowed any computer that was not owned by our company to even attach to our network. I suspect there will be many people fired for allowing CrowdStrike to update the affected systems. I wouldn't want to be an employee at CrowdStrike. It reminds me of the old quote, "'A sure way to empty a hotel fast: is to drop an elevator."