Avoiding Key Disasters

[Ryan Gallagher]

I'm realizing after a near disaster with a key for a different VF and a much delayed correction from Deluxe/Disney that a few things would really help, some of which I've considered as capability improvements prior, but maybe now will get some traction from mgmt. I'm not sure anything can be done to make the emergency key issues go faster.

(We are rolling on time, but held doors 30 min while I frantically set up two stream backup options seeking best quality).

1. Ability to remote into our booth network on a VPN, if I get key on a dark day I can ingest immediately at home and check validity instead of 2 hours before show time.
2. Record CPL names and UUID last digits in our content tracking spreadsheets. We track most things, but if you are at home you have no way to even glance if the Key CPL/UUID matches the delivered film, unless you are really paying attention and check the shipping notice CPL name, at risk of getting ahead of yourself if they deliver something else.
3. Have a ready to go wired network stream device for punting to a stream if you absolutely have to. I got lucky there happened to be a fire-stick that I knew the location of because of children's performances in the summer (they watch a movie on the event projector during lunch break). That and the film had a rentable HD source. Laptop and browser won't cut it on most platforms due to web HDCP restrictions.

We know we have a key or not ahead, and validity checking kinda depends on who is in the booth the couple days prior (worst case, no one).

Would any of this info be more available by getting access to our deluxe portal? Currently only our Programmer and TD have logins, though we projectionists have email access to a key destination email.

Deluxe didn't help much in that they had to contact the booker to release a key, and that part nearly didn't happen. Got a valid key 90min after the request, proved a backup plan, cued up, checked Audio, and was about to open doors. Pivoted back to DCP in the last second.

I came in early today to haul 70mm cans upstairs, had I come in with usual lead time for a DCP this one might not have happened at all.

[Leo Enticknap]

This all sounds completely sensible. The only thing I'd flag up is that 1 (remote access to the booth) risks being a double-edged sword. In my final projectionist job before moving to a service tech role, I set this up both for this reason (dealing with KDM emergencies), and to be able to support other projectionists as and when they encountered issues. The problem was that management and co-workers then expected me to be available to "remote in" basically whenever any show was being prepared or in progress, but management was a lot less enthusiastic about me putting the hours on my timesheet when I was called upon to do so. I handed in my notice and quit while Local 33 was trying to figure out what to do about this, and don't know what deal (if any) was finally struck to cover what happens when a projectionist calls on the chief for assistance from off site. As this was an intensively programmed arthouse that often had a throughput of 30-40 DCPs and film prints a week, and lots of labor intensive shows involving multiple clips, bespoke AV setups, etc., those calls for assistance were frequent and nontrivial.

So my suggestion would be that before any remote access capability is put in place, work-at-home expectations and boundaries be established.

[Tony Bandiera Jr]

This all sounds completely sensible. The only thing I'd flag up is that 1 (remote access to the booth) risks being a double-edged sword. In my final projectionist job before moving to a service tech role, I set this up both for this reason (dealing with KDM emergencies), and to be able to support other projectionists as and when they encountered issues. The problem was that management and co-workers then expected me to be available to "remote in" basically whenever any show was being prepared or in progress, but management was a lot less enthusiastic about me putting the hours on my timesheet when I was called upon to do so. I handed in my notice and quit while Local 33 was trying to figure out what to do about this, and don't know what deal (if any) was finally struck to cover what happens when a projectionist calls on the chief for assistance from off site. As this was an intensively programmed arthouse that often had a throughput of 30-40 DCPs and film prints a week, and lots of labor intensive shows involving multiple clips, bespoke AV setups, etc., those calls for assistance were frequent and nontrivial.

So my suggestion would be that before any remote access capability is put in place, work-at-home expectations and boundaries be established.

I second everything Leo said and on that last bit, get it writing, with a defined duration of validity (contract.) If you are Union (I'm pretty sure Ryan is) then do all of that with the Local's business agent and attorney involved.

For anyone else thinking of setting up a "work at home" via remote login, be sure you have any documents reviewed and blessed by a good attorney knowledgeable of contract laws. Otherwise you WILL end up getting screwed.

[Leo Enticknap]

I was dealing with this 8-9 years ago, and so hopefully the situation has gotten better; but a major aspect to the problem was that Internet-based remote access was a totally alien concept to the union's business agent and attorney. Even the fact that we were talking about having the ability to operate and troubleshoot equipment in the booth but from an employee's home was totally new to them. When I tried to explain what VPNs, Teamviewer, etc. did and enabled, they were like rabbits in headlights. Eventually they understood what the issue was: that without a clear agreement as to when I would be available to do this work and what and how I would be paid for doing it, there risked being a default expectation that I would be on call 24/7, but only paid when I was physically present in the booth. Then the horse trading with management began, during which I quit.

This was pre-covid, and the widespread growth of working from home that it caused has hopefully resulted in unions' business agents and attorneys coming up to speed on the issues, and being able to negotiate them.

[Ryan Gallagher]

All very sound advice. I was thinking of the network security double edge sword, but the compensation framework one is equally if not more valid. If our network was airgapped as it ought to be (it is not, that is a longer IT story than I have time for), then it would be a device that straddles both networks and hosts the VPN gateway, raspberry pi's are popular. But i'm sure our IT person has their own ideas about how a booth VPN should work.

I think I'll go down the documentation rabbit hole first, at least at a glance I can see what UUID was ingested, and use a key-viewer to confirm everything looks kosher. And only for those occasions when keys show up and no one else is paying attention such as dark days ahead of films.

That and push for an "oh shit" stream device with a company account and payment methods. (I had to boot our EDU department off the fire-stick and put my own account on it for emergency payment).

And yes I am Union here. They are somewhat comfortable with me putting hours down when I have to rip a non-us region blu-ray into a DCP, or god forbid record a stream to DCP for a 35/70mm backup, or build a client's entire presentation, QLab, or OBS configurations. But general "on call" hours is not something formalized... though we are usually responsive when our brothers and sisters need assistance on our day off (that goes for all departments).

[Ryan Gallagher]

But how much of this snag would have been visible in deluxe portal and maybe forseen? I've never even seen what it exposes to the customer.

We've had close calls with keys especially during festivals... but this probably as close as we've come to cancelling a show in a while.

[James Gardiner]

As I use to run a integrator, from my perspective, support networks that can talk directly with all your equipment is an essential skill any decent integrator should be using.
These days, it's all about pre-emptive analysis and rectification. Fix it before it results in a dark screen. To do this, you need to be able to collect telemetry from all devices in the projection booth directly. (Not a remote desktop)

My current business is running 3 regional cinemas. I got out of installation and integration as in my region its too.. lets say heavy fingers are on the scale and I could not be bothered dealing with the BS.
If the independents are happy to look past unethical and corrupt behaviour. They screwed themselves, as this has resulted in - the only integrators in our country are owned by majors, or their competitors. You can imagine how that goes down. They made their bed. I'll look after myself (And a few friends I feel are good people)

My cinemas operate autonomously, and I operate and run them remotely, 1x4hour drive, 1x8hour drive, 1x2hour domestic flight (Australia is big). Obviously, this all relies on secure private networks. But then again, I can go on holiday with my family, All I have to do is open my laptop and I can run the circuit from anywhere.

I also look after another network of 4 cinemas in my spare time, plus a Mini Major of 6 sites (ACE TMS support mainly). And a Post Production Facility. (I use to have a post house when I was younger)
Generally my cinema systems are as automated as I could make them. I just have to do the booking, which I have built tools to make that super fast, Once booked, the booking offer in my database goes directly into the cinema software and automates doing everything else. From posters to trailer for the website. This is how I still have plenty of spare time even though I run 3 locations myself with no staff. (I do have a book keeper doing the invoicing, 1 day a week)

Generally, the conditions I have with these entities I do work for is. They can call me or leave me a message, but I don't get back to them until it's suitable to me. And generally they are fine with this as. Having the experience I have, they know if I look into the problem it will be fixed or rectified fast. So they are happy to work to my schedule. Otherwise, I just charge them an hourly rate.

Secure network wise, I host EVERYTHING myself. From network overlays to a teamviewer equivalent. Its all my own infrastructure. No service costs. All built for my use in doing the support I give.. I am not open to any global hack.
This all lives on my own small data centre I built in my office, with fiber, non contended directly to an Equinix backbone. (One of the biggest interconnect companies in the world)

I know quite a few independents, and they ALL use either VPNs (I setup for them) or good old TeamViewer. It is their business and business is tight right now. It makes sense, they take on the advantages of bringing in remote access at critical times to save them from a dark screen.

Finally, I am trying to put some time into coding my admin.d-cine.net website into a new version based on faster modern frameworks. I plan, once I convert the old into the new framework, to implement a KDM service on it. (Register a CPL and point and click to dispatch KDMs, targeting self distributors) Plus internationalise some tools so they work for other than Australian cinema operators. Thats what I do with any time I have left during the week.
I also have cinema-catcher-app I created. I may come back to that in a few years. Upgrade it to do minimal TMS functions. Free entry level TMS for smaller locations. Possibly add the QC fatures seen on admin.d-cine.com so it would create mp4 of all adv, TLR with DCP analysis and Leq(m) reports. For example, I find it useful that all my TLRs go into my d-cine system, and if I need a trailer as a mp4, I just go there and get it for website etc.

[James Gardiner]

Ok the KDM issue.
Yes KDMs can cause issues. Its the main feature of cinema-catcher-app. It scans your KDM email account, and drags ALL kdms into a database. I datbase you can search and use to analyse the KDMs that arrives. (Or don't arrive as is often the issue, and with this tool you can figure that out very quickly) ie. ff a KDM issues happens, its fast to diagnose why something may not be working. You also know when they arrive, when they are ingested into a player. It basically pushes ALL KDM for a specific player into that player, for all VO and VF versions. If a KDM exists for your player, its ON that player. It also has a tool that sends a report every morning via Email indicagtin if any SPL has a CPL in it that a KDM does not exist for. Early warning system so you can get onto the distributor the day before if a KDM have been forgotten. Happens at least once per month. It used to be happen more but they are quite good now. But then again, the system also sends the distributors a report every week telling them exactly WHAT KDMs they need to send. I find it smooth as these days. I have not had an emergency KDM issue for years.

If an emergency KDM is coming, you can just log in and tell it to scan emails NOW, otherwise it does it every 30min or 1h (Configurable)

I have one instance of cinema-catcher-app, all it does it read the KDM email for 6 decent sized sites, and FTPs the KDM directly into ACE TMS systems every 30min. It has shown me where I need to do some improvements. The table of KDMs downloads all active KDMs, for 6 locations that's a LARGE list, makes the interface sluggish, next time I will do paged access to the database to speed it up a lot.