Internet Insanity

[Jim Cassedy]

I mentioned in a post the other day that we had a major internet outage here last
Thursday, that made the workday very challenging for everyone here. I actually
didn't have it too bad, since I was still able to communicate internally with all the
projectionstuff. But the box office and the kitchen and wait-staff had a really hard
time because all their fancy new POS terminals were down, and box office couldn't
sell or verify online ticket sales. At first they couldn't run credit cards, but I think they
finally figured a way around that using someone's phone. It was definitely an ISP
problem, but nonetheless, I spent 9 hours on the phone troubleshooting with their
tech-support people, and running around between our main terminal room, our
server room, and my main work-station, which are all on different floors and widely
separated from each other. (This is a HUGE building that occupies the full depth
of a city block) It took a 14hour day to fix, but it was working when I went home.
(and collapsed!)

So, today, the fire alarm people were here early this morning, doing some sort
of testing, and they somehow managed to break one of the sprinkler heads in
the main terminal room. (our MPOE, for those of you who know Telco terminology)
- - the sprinkler was right above the backboard where all the incoming copper &
fiber lines terminate, and all the equipment got soaked, the AC circuit breakers
tripped, once again taking us totally off the 'net. I'm not sure how long it's going
to take to repair. At least today, I won't spend another 9hours on the phone, since
we KNOW what the problem is, and at this point there's not a thing I can do.

As I've said before: it's a good thing my venue isn't closer to the Golden Gate
Bridge, or I would have jumped a long time ago.

Fortunately, I was made aware of this before I left for work today, and I was able
to download some content I needed for a special event tonight, and for a couple
of shows tomorrow, in case we're still down. I have a super fast fiber connection
at home, so I was able to download what i needed & still get to work on time,
and with any luck, I may even get home at a reasonable hour tonight.

[Leo Enticknap]

Oof! The fire alarm company will likely see a hike in their liability insurance premium at the next renewal! Drying out the MPOE room (which could take days), replacing potentially an an entire room full of equipment, the installation labor costs, and the lost business while all of this is done, could easily cost in the low to mid six figures, I'd have thought.

My initial reaction after reading the first paragraph is that Internet access is now as important as electricity to many businesses, because without it, you simply can't do your core business; and that therefore, loss of Internet service should be factored into disaster planning that publicly traded companies are required to do by law, and private ones and nonprofits should also do if their leadership has any sense. That could be as simple as having a cellular router/access point available as an alternative, all the way up to more robust and pricey strategies.

However, there really isn't much planning you can do to mitigate the risk of a catastrophic but good faith error by a vendor or contractor like that, apart from only hiring reputable ones.

[Jim Cassedy]

I could easily understand if some tradesman or some theater employee had
inadvertently set off one of the sprinklers- - but these were THE FIRE ALARM
GUYS. Even though the ceiling in that room is a little low, they still should have
known better. (and the sprinkler head is protected by a little cage, so you really
needed to be really careless or bump it really hard to set it off) And not only that,
but the main sprinkler shut-off valve is located in that same room, less than 10ft
away from the sprinkler they set off, and from what I was told, it still took them
several minutes to stop the water flow. Maybe they just didn't want to get wet
while turning it off. I wasn't here, but there are so many unanswered questions.
> And, once I found out what happened, I had to get here early enough to re-set
all the auditorium servers, since they had all gone into the "emergency shutdown"
mode. The theater wasn't open when this happened, but the 1st round of shows
wouldn't have started until the auditorium servers were manually re-set.
"It's always something"

[Leo Enticknap]

And not only that, but the main sprinkler shut-off valve is located in that same room, less than 10ft away from the sprinkler they set off, and from what I was told, it still took them several minutes to stop the water flow. Maybe they just didn't want to get wet while turning it off.

I feel your pain. These people sound like they would make Laurel and Hardy look like consummate professionals (jump to approx. 4'05" for the scene that would likely give anyone working for the California contractors' licensing board multiple heart attacks)...

On a more serious note, what happened in Spain and Portugal earlier today appears to have been a lesson in just how quickly and severely things go south if the Internet goes down. Even stores, gas stations, etc. that had emergency generators or could function at some level without electricity could only accept cash, because no Internet meant that the infrastructure needed to process electronic payments wasn't working.

[Ryan Gallagher]

I've heard of numerous arena concert load-ins during my career that involved an unusually tall road box going down a hallway with protruding sprinkler heads. You can imagine what happens next.

EDIT: Ohh also usually those systems are pressurized, so even if you shut it off immediately it will still drain a ton of water out of the pipes, usually gross rusty water if it's an older building. They probably shut it off as quick as they could.

I would think a failover 2nd ISP at least for the POS systems would be something most heavily traffic'd retail establishments had these days. I guess not.

On a related internet note. We learned the hard way that our only booth laptop with adequate storage for long-form DCP downloads is too old to have a gigabit network card, or something else is going on. It was fetching 100mb/s while our newer laptop (which was tasked elsewhere for the show), with an inadequate C: drive pulls 900mb/s (but that one is IT managed and we can't install apps such as frame.io downloader or signiant). We can download directly to USB caddies but that doesn't help much if we can't install the apps on short notice.

It derailed a rather disorganized tech-check opporutnity for my recent festival that provided all their DCPs as a single Frame.io project link (including their contracted helper's DCP-o-matic project folders, so 2x the needed size) via email 30min before our only 4 hour opportunity to do the full load in and also check any of their content. Despite numerous occasions of them being instructed to bring physical drives if it was going to be day-of. We got there in the end for all the screenings (cause I downloaded just the DCP content myself at home the night before festival kickoff).

My world for a modern desktop machine I have admin privs on, and a local file-server for download destinations and network ingest to the server. Lacking them had never impacted a show before, and now it has, so maybe my chances are better.

[Jim Cassedy]

The internet at my venue is still down. The equipment that got waterlogged
has been replaced, and now It's a carrier issue. Our data transport is handled
by two different entities (the ISP and the local incumbent carrier) and each
one is blaming the other. If my internet at home was down for 4 days, I'd
be pretty upset, so I don't understand why someone higher up the corporate
ladder isn't on the phone screaming at our service providers to get this fixed.

Anything that was wasn't already scheduled before Monday, just doesn't
exist, as far as our venue systems are concerned, which means that as of
today, there are only 4 shows scheduled to play, and less than half a dozen
others scattered over the weekend. ("Sinners" is toast as of yesterday)

Yes, I have content and keys, almost all of which I had downloaded at
home during the week, but none of our shows are appearing on our website,
and we can't sell tickets, because the shows "don't exist". We can't process
credit cards. All the electronic poster frames are displaying nonsense
graphics, because they too are managed by some central server which
I think is up in Sacramento. At the moment, even our box office doesn't
know what our show times are supposed to be, or even what auditorium
a show is supposed to play in. And to top it all off, all of our pinball and
game machines are dead, because about a year ago, the company
that operates them converted them from coin operation to a system
that only takes pre-paid game cards and credit cards, and with the
internet down, the games have no way of processing payments.

I'm going in to work later this morning, but I literally have nothing to do.
(Oh, I'll find something to keep me busy. I've got some incoming and
outgoing 35mm prints and some paperwork to deal with, and there's
some cleaning I can accomplish, but if the 'net is still down at noon,
- - I'm probably going home. )

[Ryan Gallagher]

....We can't process
credit cards. All the electronic poster frames are displaying nonsense
graphics, because they too are managed by some central server which
I think is up in Sacramento. At the moment, even our box office doesn't
know what our show times are supposed to be, or even what auditorium
a show is supposed to play in. And to top it all off, all of our pinball and
game machines are dead, because about a year ago, the company
that operates them converted them from coin operation to a system
that only takes pre-paid game cards and credit cards, and with the
internet down, the games have no way of processing payments.....

Wow, when pinball machines no longer work you know we have put the internet in too many things! LOL

Best of luck to them and getting up and running again. Sounds like even redundant internet for POS systems wouldn't have fully saved this outage. Too much stuff in the cloud!

[Mark Gulbrandsen]

Satellite ISP cures those problems. And I'm not teferring to Starlink. I have friends that tried that a couple of years ago and found it to be too inconsistant.

[Frank Cox]

I posted here a while back that I have two internet services here from two different ISP's. One cable and one DSL. (Supposedly the DSL is going to get converted to fiber but that's been "any day now" for about the past four years.)

I don't know why much bigger theatres than me don't have two internet services. That way if one is down, or slow, or whatever, you can just instantly switch to the other. And you can do some rudimentary load balancing too.

[Ryan Gallagher]

I posted here a while back that I have two internet services here from two different ISP's. One cable and one DSL. (Supposedly the DSL is going to get converted to fiber but that's been "any day now" for about the past four years.)

I don't know why much bigger theatres than me don't have two internet services. That way if one is down, or slow, or whatever, you can just instantly switch to the other. And you can do some rudimentary load balancing too.

Sounds like maybe they did have two ISPs based on Jim's last post... but the location of the water damage probably took them both out. Which is configured to do what and if they have any redundancy configured is a different question of course. But won't matter if one flooded equipment room kills em both.

[Mark Gulbrandsen]

Fiber is awesome! It never goes out here, even if the streets flood or what ever. The only thing that killed ours is when some dude blew up the main phone exchange down town. Still, they had cellular and internet back up in just over two days using portable equipment.

[Mike Blakesley]

all of our pinball and
game machines are dead, because about a year ago, the company
that operates them converted them from coin operation to a system
that only takes pre-paid game cards and credit cards, and with the
internet down, the games have no way of processing payments.

You'd think there would be some way of bypassing all that crap and just installing a good ol fashioned coin box (or bill reader). Any game tech should be able to do that, shouldn't they?

[Ryan Gallagher]

You'd think there would be some way of bypassing all that crap and just installing a good ol fashioned coin box (or bill reader). Any game tech should be able to do that, shouldn't they?

Except no one carries cash anymore, if they want them to get used gotta come up with something else.

i would have thought tokens you could buy at the concessions would have been easier than some cloud connected arcade card. The pinball arcade in town is all coin/token operated, contributes to the nostalgia selling points.

[Mark Gulbrandsen]

Except no one carries crash anymore, if they want them to get used gotta come up with something else.

i would have thought tokens you could buy at the concessions would have been easier than some cloud connected arcade card. The pinball arcade in town is all coin/token operated, contributes to the nostalgia selling points.

The use of coins or tokens eliminates bank expenses on card readers.

[Mike Blakesley]

Many arcades have machines which will turn a credit card into tokens. (Or bills, too.)