Random photos, comics, etc.

[Mark Gulbrandsen]

Screw Crowd Strike! Airlines, at least, should have a separate data center of their own at each airport. If this country ever had a terrorist attack we'd be at a giant shand still again.

[Randy Stankey]

That was more of a rhetorical statement, of course.

Still, I would have my people looking into alternatives beside CrowdStrike and I would be calling CrowdStrike up to say, "You fucked up! What are you going to do about it?" I wouldn't accept any pat answers and I certainly wouldn't let them hide behind any flimsy terms in some EULA.

Their stupid and avoidable screw up cost just cost thousands of companies, uncountable millions of dollars that are far in excess of the money they charged for their software. Sure, people shouldn't allow software to be updated automatically but, when you're talking about critical infrastructure the stakes are a lot higher and everybody who's got skin in the game needs to take responsibility.

When it all comes out in the wash, though, the buck stops at Crowdstike's doorstep. They are the ones who, ultimately, need to make things right.

"Oops! My bad," is NOT an acceptable answer!

[Mark Gulbrandsen]

Randy,
There may not be an option for many of these system users to move, and no one updates millions of computers all at once, that is beyond stupid. It will take weeks, if not months to get all this mess cleaned up. End of story.

[Randy Stankey]

The bigger problem is that it's millions of computers that are all over the world...

No, the problem is that it happened on mission critical computers, all over the world.

I would agree that the chances of something like this happening are small... less than 1%. However, if something happens, the chances of people being hurt, killed or negatively affected in some way is 100%.

If computers at banks go down, people can't access money and their lives grind to a halt. If somebody needs that money for an important purpose, they could be harmed. Even if that doesn't happen, businesses could be negatively impacted.

If computers at the airlines go down, millions of people are affected or, in an extreme situation, people could be killed.

If this problem only happened to home computers, it wouldn't be such a big deal. In a situation like that, edge cases don't matter so much. However, if you're talking about critical infrastructure, edge cases DO matter!

It's common for people to decide how much time, effort and money to put into preventing problems based on the probability of something bad happening. That makes sense. People don't hire armed guards to protect their houses, 24/7 because the chances of being burgled are pretty low. However, I'm certainly going to have armed guards protecting the White House because, even though the chances of somebody getting in are relatively low, IF somebody gets in, it could cause problems for the whole country.

In other words, gauge the amount of trouble that something might cause to determine how much of your resources to put into prevention of a problem, not just the chances of it happening.

If I was the boss of a company that got affected by this malfunction at CrowdStrike, I'd be making two phone calls:

My first call would be to my the head of my IT department to tell them to get something else, beside Crowdstrike, immediately and forthwith.

My second call would be to the guy in charge of CrowdStrike... The entire conversation would be something like, "We're done." <click>

[Mark Gulbrandsen]

Leo, there are mini computers today, used for ticketing a lot in Cinemas, that could easily serve as local computers on a big network. I got rid of my big workstation several years ago and run an HP Z2 G6 Mini computer. It has most of the features of the larger workstations. There are also monitors from all the big manufacturers that have the Mini computer built in.

Now if it's a VM setup, that makes the fix far more difficult as you pointed out. I suspect that in the end computer techs are going to have to go to every site and do the fixes locally, then verify the fix.

What a nightmare!

[Leo Enticknap]

I'm sure those batteries are also used in ahem, "personal pleasure items" thus the category.

I can't claim to be any sort of an expert on this category of consumer tech, but I'd be surprised if one could get much personal pleasure out of 3v at a few millliamps. I would expect to find three or four 18650s in such a device at the very least!

The funny part is that the fix is uber easy to do.

From what I've been reading, the problem is now threefold. The fix (booting into safe mode, deleting a specific file, but which is in a Windows folder that is hidden by default, and then rebooting back into regular mode) is easy for someone who knows their way around Windows, but is just above the skill level of your average office worker. So in any given megacorp, a small crew of IT techs have to go around all the office PCs, fixing them manually, which will take some time. The second problem is that a lot of the affected devices are Windows running inside VMs, which can't be booted into safe mode. You would then need to connect the VM file or drive to a second VM in order to gain access to it to delete the file, which needs a proper techie. The third problem is that if the drive containing the affected Windows install (either physical or VM) is encrypted using Bitlocker, you need the key.

BTW, a somewhat surprising reason emerged why Southwest was almost unaffected by the Crowdstrike SNAFU:

Windows version from 1992 is saving Southwest’s butt right now

By Jacob Roach July 19, 2024 10:19AM

Nearly every flight in the U.S. is grounded right now following a CrowdStrike system update error that’s affecting everything from travel to mobile ordering at Starbucks — but not Southwest Airlines flights. Southwest is still flying high, unaffected by the outage that’s plaguing the world today, and that’s apparently because it’s using Windows 3.1.

Yes, Windows 3.1 — an operating system that is 32 years old. Southwest, along with UPS and FedEx, haven’t had any issues with the CrowdStrike outage. In responses to CNN, Delta, American, Spirit, Frontier, United, and Allegiant all said they were having issues, but Southwest told the outlet that its operations are going off without a hitch.

Some are attributing that to Windows 3.1. Major portions of Southwest’s systems are reportedly built on Windows 95 and Windows 3.1, which is something the company has come under fire for in the past several years. It should go without saying that Southwest needs to update its system, but in this case, the ancient operating system seems to be doing the airline some favors to avoid a complete Y2K-level apocalypse.

If you aren’t flying Southwest, you’re out of luck right now. Airports around the world had their scheduling systems crash in the wake of the CrowdStrike update, sending millions of travelers into a frenzy. The Federal Aviation Administration (FAA) said it’s working with several airlines on the outage. Thankfully, the FAA itself hasn’t been affected.

Microsoft, who has been at the center of this fiasco with CrowdStrike, says that the root cause of the issue has been fixed. It could take days before everything is sorted out, though. Microsoft’s CEO Satya Nadella commented on the issue on X (formerly Twitter), saying, “We are aware of this issue and are working closely with CrowdStrike and across the industry to provide customers technical guidance and support to safely bring their systems back online.”

That shows the scale of this problem. Microsoft has outages all the time, but none of them are worth commenting on from the CEO of the company. This is a different beast entirely, affecting millions of servers running on Windows. Southwest seems to have saved itself from any trouble by being woefully late to upgrade.

I'm guessing that their W3.1 systems must almost all be in VMs - there surely can't be many, if any, motherboards and processors made in recent years that are capable of running it natively.

[Mark Gulbrandsen]

Here is how you fix the problem.

[Mark Gulbrandsen]

I used to work for a Fortune 500 corporation as a Sr. Systems Analyst. We never allowed any software provider, including Microsoft, to automatically update our systems. We also never allowed any computer that was not owned by our company to even attach to our network. I suspect there will be many people fired for allowing CrowdStrike to update the affected systems. I wouldn't want to be an employee at CrowdStrike. It reminds me of the old quote, "'A sure way to empty a hotel fast: is to drop an elevator."

The funny part is that the fix is uber easy to do. The start point is the "Blue Screen Of Death Page". There are already "How To" videos on Instagram and Tic Toc.. Takes about 15 seconds to do, then just reboot the computer and you're back home... The bigger problem is that it's millions of computers that are all over the world...

[Ed Gordon]

It's good to have backup systems!

On system updates, there is a script that pushes updates to USL/QSC/MIT IRC systems. The script checks that the system comes back up again and reports the new version number before going on to the next unit. The script exits on the first failure so we don't end up with hundreds of dead systems.

Weren't there a couple of guys, here, at Film-Tech who always used to say that you should never accept automatic software updates but nobody seemed to want to listen?

I used to work for a Fortune 500 corporation as a Sr. Systems Analyst. We never allowed any software provider, including Microsoft, to automatically update our systems. We also never allowed any computer that was not owned by our company to even attach to our network. I suspect there will be many people fired for allowing CrowdStrike to update the affected systems. I wouldn't want to be an employee at CrowdStrike. It reminds me of the old quote, "'A sure way to empty a hotel fast: is to drop an elevator."

[Tony Bandiera Jr]

By my calculation, Young Boozer II would have had to violate the Volstead Act to have lived up to his name!

Changing the subject somewhat:

image.png

So Barco Alchemy certificate batteries are a "health and beauty" product, according to Amazon! Knowing that an ICMP won't bork, resulting in a panicked call or email, is not bad for my health, I guess, but I'm still not sure that I get the beauty part...

I'm sure those batteries are also used in ahem, "personal pleasure items" thus the category.

[Randy Stankey]

Weren't there a couple of guys, here, at Film-Tech who always used to say that you should never accept automatic software updates but nobody seemed to want to listen?

[Harold Hallikainen]

[ATTACH=JSON]
I felt somewhat smug this morning at Starbucks: they could only accept cash, and I was one of very few people in line who actually had some.

It's good to have backup systems!

On system updates, there is a script that pushes updates to USL/QSC/MIT IRC systems. The script checks that the system comes back up again and reports the new version number before going on to the next unit. The script exits on the first failure so we don't end up with hundreds of dead systems.

[Leo Enticknap]

image.png

I felt somewhat smug this morning at Starbucks: they could only accept cash, and I was one of very few people in line who actually had some.

[Randy Stankey]

That's pretty freakin' amazing!

The scroll quilting must have taken forever, even if done on a machine!

[Frank Cox]

450905735_880815164073737_4190213294174939287_n.jpg
Here is a picture of the whole quilt.