Internet Insanity

[Jim Cassedy]

I mentioned in a post the other day that we had a major internet outage here last
Thursday, that made the workday very challenging for everyone here. I actually
didn't have it too bad, since I was still able to communicate internally with all the
projectionstuff. But the box office and the kitchen and wait-staff had a really hard
time because all their fancy new POS terminals were down, and box office couldn't
sell or verify online ticket sales. At first they couldn't run credit cards, but I think they
finally figured a way around that using someone's phone. It was definitely an ISP
problem, but nonetheless, I spent 9 hours on the phone troubleshooting with their
tech-support people, and running around between our main terminal room, our
server room, and my main work-station, which are all on different floors and widely
separated from each other. (This is a HUGE building that occupies the full depth
of a city block) It took a 14hour day to fix, but it was working when I went home.
(and collapsed!)

So, today, the fire alarm people were here early this morning, doing some sort
of testing, and they somehow managed to break one of the sprinkler heads in
the main terminal room. (our MPOE, for those of you who know Telco terminology)
- - the sprinkler was right above the backboard where all the incoming copper &
fiber lines terminate, and all the equipment got soaked, the AC circuit breakers
tripped, once again taking us totally off the 'net. I'm not sure how long it's going
to take to repair. At least today, I won't spend another 9hours on the phone, since
we KNOW what the problem is, and at this point there's not a thing I can do.

As I've said before: it's a good thing my venue isn't closer to the Golden Gate
Bridge, or I would have jumped a long time ago.

Fortunately, I was made aware of this before I left for work today, and I was able
to download some content I needed for a special event tonight, and for a couple
of shows tomorrow, in case we're still down. I have a super fast fiber connection
at home, so I was able to download what i needed & still get to work on time,
and with any luck, I may even get home at a reasonable hour tonight.

[Leo Enticknap]

Oof! The fire alarm company will likely see a hike in their liability insurance premium at the next renewal! Drying out the MPOE room (which could take days), replacing potentially an an entire room full of equipment, the installation labor costs, and the lost business while all of this is done, could easily cost in the low to mid six figures, I'd have thought.

My initial reaction after reading the first paragraph is that Internet access is now as important as electricity to many businesses, because without it, you simply can't do your core business; and that therefore, loss of Internet service should be factored into disaster planning that publicly traded companies are required to do by law, and private ones and nonprofits should also do if their leadership has any sense. That could be as simple as having a cellular router/access point available as an alternative, all the way up to more robust and pricey strategies.

However, there really isn't much planning you can do to mitigate the risk of a catastrophic but good faith error by a vendor or contractor like that, apart from only hiring reputable ones.

[Jim Cassedy]

I could easily understand if some tradesman or some theater employee had
inadvertently set off one of the sprinklers- - but these were THE FIRE ALARM
GUYS. Even though the ceiling in that room is a little low, they still should have
known better. (and the sprinkler head is protected by a little cage, so you really
needed to be really careless or bump it really hard to set it off) And not only that,
but the main sprinkler shut-off valve is located in that same room, less than 10ft
away from the sprinkler they set off, and from what I was told, it still took them
several minutes to stop the water flow. Maybe they just didn't want to get wet
while turning it off. I wasn't here, but there are so many unanswered questions.
> And, once I found out what happened, I had to get here early enough to re-set
all the auditorium servers, since they had all gone into the "emergency shutdown"
mode. The theater wasn't open when this happened, but the 1st round of shows
wouldn't have started until the auditorium servers were manually re-set.
"It's always something"

[Leo Enticknap]

And not only that, but the main sprinkler shut-off valve is located in that same room, less than 10ft away from the sprinkler they set off, and from what I was told, it still took them several minutes to stop the water flow. Maybe they just didn't want to get wet while turning it off.

I feel your pain. These people sound like they would make Laurel and Hardy look like consummate professionals (jump to approx. 4'05" for the scene that would likely give anyone working for the California contractors' licensing board multiple heart attacks)...

On a more serious note, what happened in Spain and Portugal earlier today appears to have been a lesson in just how quickly and severely things go south if the Internet goes down. Even stores, gas stations, etc. that had emergency generators or could function at some level without electricity could only accept cash, because no Internet meant that the infrastructure needed to process electronic payments wasn't working.

[Ryan Gallagher]

I've heard of numerous arena concert load-ins during my career that involved an unusually tall road box going down a hallway with protruding sprinkler heads. You can imagine what happens next.

EDIT: Ohh also usually those systems are pressurized, so even if you shut it off immediately it will still drain a ton of water out of the pipes, usually gross rusty water if it's an older building. They probably shut it off as quick as they could.

I would think a failover 2nd ISP at least for the POS systems would be something most heavily traffic'd retail establishments had these days. I guess not.

On a related internet note. We learned the hard way that our only booth laptop with adequate storage for long-form DCP downloads is too old to have a gigabit network card, or something else is going on. It was fetching 100mb/s while our newer laptop (which was tasked elsewhere for the show), with an inadequate C: drive pulls 900mb/s (but that one is IT managed and we can't install apps such as frame.io downloader or signiant). We can download directly to USB caddies but that doesn't help much if we can't install the apps on short notice.

It derailed a rather disorganized tech-check opporutnity for my recent festival that provided all their DCPs as a single Frame.io project link (including their contracted helper's DCP-o-matic project folders, so 2x the needed size) via email 30min before our only 4 hour opportunity to do the full load in and also check any of their content. Despite numerous occasions of them being instructed to bring physical drives if it was going to be day-of. We got there in the end for all the screenings (cause I downloaded just the DCP content myself at home the night before festival kickoff).

My world for a modern desktop machine I have admin privs on, and a local file-server for download destinations and network ingest to the server. Lacking them had never impacted a show before, and now it has, so maybe my chances are better.