Avoiding Key Disasters

[Ryan Gallagher]

I'm realizing after a near disaster with a key for a different VF and a much delayed correction from Deluxe/Disney that a few things would really help, some of which I've considered as capability improvements prior, but maybe now will get some traction from mgmt. I'm not sure anything can be done to make the emergency key issues go faster.

(We are rolling on time, but held doors 30 min while I frantically set up two stream backup options seeking best quality).

1. Ability to remote into our booth network on a VPN, if I get key on a dark day I can ingest immediately at home and check validity instead of 2 hours before show time.
2. Record CPL names and UUID last digits in our content tracking spreadsheets. We track most things, but if you are at home you have no way to even glance if the Key CPL/UUID matches the delivered film, unless you are really paying attention and check the shipping notice CPL name, at risk of getting ahead of yourself if they deliver something else.
3. Have a ready to go wired network stream device for punting to a stream if you absolutely have to. I got lucky there happened to be a fire-stick that I knew the location of because of children's performances in the summer (they watch a movie on the event projector during lunch break). That and the film had a rentable HD source. Laptop and browser won't cut it on most platforms due to web HDCP restrictions.

We know we have a key or not ahead, and validity checking kinda depends on who is in the booth the couple days prior (worst case, no one).

Would any of this info be more available by getting access to our deluxe portal? Currently only our Programmer and TD have logins, though we projectionists have email access to a key destination email.

Deluxe didn't help much in that they had to contact the booker to release a key, and that part nearly didn't happen. Got a valid key 90min after the request, proved a backup plan, cued up, checked Audio, and was about to open doors. Pivoted back to DCP in the last second.

I came in early today to haul 70mm cans upstairs, had I come in with usual lead time for a DCP this one might not have happened at all.

[Leo Enticknap]

This all sounds completely sensible. The only thing I'd flag up is that 1 (remote access to the booth) risks being a double-edged sword. In my final projectionist job before moving to a service tech role, I set this up both for this reason (dealing with KDM emergencies), and to be able to support other projectionists as and when they encountered issues. The problem was that management and co-workers then expected me to be available to "remote in" basically whenever any show was being prepared or in progress, but management was a lot less enthusiastic about me putting the hours on my timesheet when I was called upon to do so. I handed in my notice and quit while Local 33 was trying to figure out what to do about this, and don't know what deal (if any) was finally struck to cover what happens when a projectionist calls on the chief for assistance from off site. As this was an intensively programmed arthouse that often had a throughput of 30-40 DCPs and film prints a week, and lots of labor intensive shows involving multiple clips, bespoke AV setups, etc., those calls for assistance were frequent and nontrivial.

So my suggestion would be that before any remote access capability is put in place, work-at-home expectations and boundaries be established.

[Tony Bandiera Jr]

This all sounds completely sensible. The only thing I'd flag up is that 1 (remote access to the booth) risks being a double-edged sword. In my final projectionist job before moving to a service tech role, I set this up both for this reason (dealing with KDM emergencies), and to be able to support other projectionists as and when they encountered issues. The problem was that management and co-workers then expected me to be available to "remote in" basically whenever any show was being prepared or in progress, but management was a lot less enthusiastic about me putting the hours on my timesheet when I was called upon to do so. I handed in my notice and quit while Local 33 was trying to figure out what to do about this, and don't know what deal (if any) was finally struck to cover what happens when a projectionist calls on the chief for assistance from off site. As this was an intensively programmed arthouse that often had a throughput of 30-40 DCPs and film prints a week, and lots of labor intensive shows involving multiple clips, bespoke AV setups, etc., those calls for assistance were frequent and nontrivial.

So my suggestion would be that before any remote access capability is put in place, work-at-home expectations and boundaries be established.

I second everything Leo said and on that last bit, get it writing, with a defined duration of validity (contract.) If you are Union (I'm pretty sure Ryan is) then do all of that with the Local's business agent and attorney involved.

For anyone else thinking of setting up a "work at home" via remote login, be sure you have any documents reviewed and blessed by a good attorney knowledgeable of contract laws. Otherwise you WILL end up getting screwed.

[Leo Enticknap]

I was dealing with this 8-9 years ago, and so hopefully the situation has gotten better; but a major aspect to the problem was that Internet-based remote access was a totally alien concept to the union's business agent and attorney. Even the fact that we were talking about having the ability to operate and troubleshoot equipment in the booth but from an employee's home was totally new to them. When I tried to explain what VPNs, Teamviewer, etc. did and enabled, they were like rabbits in headlights. Eventually they understood what the issue was: that without a clear agreement as to when I would be available to do this work and what and how I would be paid for doing it, there risked being a default expectation that I would be on call 24/7, but only paid when I was physically present in the booth. Then the horse trading with management began, during which I quit.

This was pre-covid, and the widespread growth of working from home that it caused has hopefully resulted in unions' business agents and attorneys coming up to speed on the issues, and being able to negotiate them.

[Ryan Gallagher]

All very sound advice. I was thinking of the network security double edge sword, but the compensation framework one is equally if not more valid. If our network was airgapped as it ought to be (it is not, that is a longer IT story than I have time for), then it would be a device that straddles both networks and hosts the VPN gateway, raspberry pi's are popular. But i'm sure our IT person has their own ideas about how a booth VPN should work.

I think I'll go down the documentation rabbit hole first, at least at a glance I can see what UUID was ingested, and use a key-viewer to confirm everything looks kosher. And only for those occasions when keys show up and no one else is paying attention such as dark days ahead of films.

That and push for an "oh shit" stream device with a company account and payment methods. (I had to boot our EDU department off the fire-stick and put my own account on it for emergency payment).

And yes I am Union here. They are somewhat comfortable with me putting hours down when I have to rip a non-us region blu-ray into a DCP, or god forbid record a stream to DCP for a 35/70mm backup, or build a client's entire presentation, QLab, or OBS configurations. But general "on call" hours is not something formalized... though we are usually responsive when our brothers and sisters need assistance on our day off (that goes for all departments).

[Ryan Gallagher]

But how much of this snag would have been visible in deluxe portal and maybe forseen? I've never even seen what it exposes to the customer.

We've had close calls with keys especially during festivals... but this probably as close as we've come to cancelling a show in a while.

[James Gardiner]

As I use to run a integrator, from my perspective, support networks that can talk directly with all your equipment is an essential skill any decent integrator should be using.
These days, it's all about pre-emptive analysis and rectification. Fix it before it results in a dark screen. To do this, you need to be able to collect telemetry from all devices in the projection booth directly. (Not a remote desktop)

My current business is running 3 regional cinemas. I got out of installation and integration as in my region its too.. lets say heavy fingers are on the scale and I could not be bothered dealing with the BS.
If the independents are happy to look past unethical and corrupt behaviour. They screwed themselves, as this has resulted in - the only integrators in our country are owned by majors, or their competitors. You can imagine how that goes down. They made their bed. I'll look after myself (And a few friends I feel are good people)

My cinemas operate autonomously, and I operate and run them remotely, 1x4hour drive, 1x8hour drive, 1x2hour domestic flight (Australia is big). Obviously, this all relies on secure private networks. But then again, I can go on holiday with my family, All I have to do is open my laptop and I can run the circuit from anywhere.

I also look after another network of 4 cinemas in my spare time, plus a Mini Major of 6 sites (ACE TMS support mainly). And a Post Production Facility. (I use to have a post house when I was younger)
Generally my cinema systems are as automated as I could make them. I just have to do the booking, which I have built tools to make that super fast, Once booked, the booking offer in my database goes directly into the cinema software and automates doing everything else. From posters to trailer for the website. This is how I still have plenty of spare time even though I run 3 locations myself with no staff. (I do have a book keeper doing the invoicing, 1 day a week)

Generally, the conditions I have with these entities I do work for is. They can call me or leave me a message, but I don't get back to them until it's suitable to me. And generally they are fine with this as. Having the experience I have, they know if I look into the problem it will be fixed or rectified fast. So they are happy to work to my schedule. Otherwise, I just charge them an hourly rate.

Secure network wise, I host EVERYTHING myself. From network overlays to a teamviewer equivalent. Its all my own infrastructure. No service costs. All built for my use in doing the support I give.. I am not open to any global hack.
This all lives on my own small data centre I built in my office, with fiber, non contended directly to an Equinix backbone. (One of the biggest interconnect companies in the world)

I know quite a few independents, and they ALL use either VPNs (I setup for them) or good old TeamViewer. It is their business and business is tight right now. It makes sense, they take on the advantages of bringing in remote access at critical times to save them from a dark screen.

Finally, I am trying to put some time into coding my admin.d-cine.net website into a new version based on faster modern frameworks. I plan, once I convert the old into the new framework, to implement a KDM service on it. (Register a CPL and point and click to dispatch KDMs, targeting self distributors) Plus internationalise some tools so they work for other than Australian cinema operators. Thats what I do with any time I have left during the week.
I also have cinema-catcher-app I created. I may come back to that in a few years. Upgrade it to do minimal TMS functions. Free entry level TMS for smaller locations. Possibly add the QC fatures seen on admin.d-cine.com so it would create mp4 of all adv, TLR with DCP analysis and Leq(m) reports. For example, I find it useful that all my TLRs go into my d-cine system, and if I need a trailer as a mp4, I just go there and get it for website etc.

[James Gardiner]

Ok the KDM issue.
Yes KDMs can cause issues. Its the main feature of cinema-catcher-app. It scans your KDM email account, and drags ALL kdms into a database. I datbase you can search and use to analyse the KDMs that arrives. (Or don't arrive as is often the issue, and with this tool you can figure that out very quickly) ie. ff a KDM issues happens, its fast to diagnose why something may not be working. You also know when they arrive, when they are ingested into a player. It basically pushes ALL KDM for a specific player into that player, for all VO and VF versions. If a KDM exists for your player, its ON that player. It also has a tool that sends a report every morning via Email indicagtin if any SPL has a CPL in it that a KDM does not exist for. Early warning system so you can get onto the distributor the day before if a KDM have been forgotten. Happens at least once per month. It used to be happen more but they are quite good now. But then again, the system also sends the distributors a report every week telling them exactly WHAT KDMs they need to send. I find it smooth as these days. I have not had an emergency KDM issue for years.

If an emergency KDM is coming, you can just log in and tell it to scan emails NOW, otherwise it does it every 30min or 1h (Configurable)

I have one instance of cinema-catcher-app, all it does it read the KDM email for 6 decent sized sites, and FTPs the KDM directly into ACE TMS systems every 30min. It has shown me where I need to do some improvements. The table of KDMs downloads all active KDMs, for 6 locations that's a LARGE list, makes the interface sluggish, next time I will do paged access to the database to speed it up a lot.

[Carsten Kurz]

https://www.kdm-inspector.com


If someone working for a cinema is able to solve a problem, he should be enabled to do so whenever the need comes up. If 99% of issues are taken care of in regular working time, really who cares for the 1% that need to be taken care of remotely?

We used to use Teamviewer to remote in on a PC that is connected to our network. Now, we use AnyDesk.
I also have means to power-on that PC remotely.
I can do this from my laptop or from my iPad or iPhone.
That PC allows me to solve all sorts of issues, KDMs, preshow, playlist programming, ingest.

[Steve Guttag]

A substantial part of our service model is via remote-in access. I can (and so) support 100s of screens. I can't be at all of them at once. But, I can certainly support multiple screens, at the same time, via remote. We do use Teamviewer (used to use Logmein) and have considered going with a route closer to James'...but I, very much, lack the expertise for that. There are costs to the remote-in concept, including the cost of the service, if you use one, plus maintaining the various remote-in PCs. So, that has to be factored in. But, I dare say, this model has proven to be successful and cheaper to the end-user who doesn't have to hire an in-house tech and we are able to, faster, resolve problems as well as diagnose problems so when we do go on-site, we come prepared with the right tools and parts for the specific problem.

One nice aspect of Teamviewer is it does log the amount of time you are spending on a site. So, as far as justifying one's hours, be they an employee or service, it is easy to show. And, if you note why you were logged in (comes up every time you log out), it is easy to document. The only time that can get fouled up (for me) is if you remain logged in for a protracted period of time (have to jump on another site or are doing some long-term testing/monitoring where you may be logged in for the day or even days. So, you may be logged in but you aren't actively doing anything. Yes, you might as well have logged out and just log back in to check on what you are working on but that doesn't always happen.

And, while I'm on this tangent...as you all know, I'm pro-Q-SYS for cinema. You CAN get Q-SYS to have NOC like reporting. It can email as well as they have a whole cloud based "Reflect" where you can set it up to monitor your equipment, including those without plugins (though they will be much more limited in reporting). With plugins, you could have it telling you about lamp expiry or other equipment failures/warnings. There is quite a bit of power there. Then again, most forms of using that will entail a license to QSC, of course ("It's a profit thing.")

[Leo Enticknap]

James and Steve describe a slightly different process to the one Ryan is contemplating, whereby remote support is provided either by a third party vendor via a service contract, or individuals working for a chain but not based at the site being supported. For a typical mainstream movie theater, either scenario makes sense, mainly because it likely won't have staff based on site with the skills and knowledge needed to provide the technical support that a site manager will need from time to time. Like James and Steve, I, as one of these external service vendors, also look after many of these service contracts. For a regular movie theater that plays the same first run movie in multiple auditoria and several times a day, it's a support model that usually works well for everyone involved.

If I understand Ryan's situation correctly, it's a mixed use venue that plays a lot of arthouse/rep/release movies, and with a multiple projectionists and technicians on the payroll, some with more advanced skill sets than others. What is being contemplated is giving the chief projectionist/technical manager the tools for Internet-based remote access from home, for use when he isn't formally at work. In a theater such as this one, the chief projectionist/technical manager is likely to have a higher level of skills and experience than would be found among the staff of a typical mall multiplex, and therefore leveraging that takes the place of a service contract with an outside technical support vendor.

Tony's and my point is that doing this can work and be beneficial for all involved, but there is a potential danger if unscrupulous management is in the mix, that sees this approach as a way to provide technical support to address the occasional emergency without having to pay for it. So guardrails have to be built, not least because all it takes is one retirement or resignation at a pay grade above the booth, and competent, sensible management can be transformed into the exact opposite. I've experienced this many times.

[Martin McCaffery]

Question/rant from one of the little guys (single screen arthouse): Do multiplexes test their shows before going public? Test them any amount at all?

Lately the keys do not become active until the day before the show, which gives me very little time to run a full test.

As we've been having problems with playback bugs, running the film all the way through makes me feel a little better (but even that doesn't catch all erratic problems, but that's a different discussion).

Today I got the key for a show we have tomorrow. It is not active until 8am tomorrow. We have children's matinees in the morning, so I wouldn't be able to test it until tomorrow afternoon. If there is something wrong, I don't have much time to fix and retest. Certainly no time to order another drive if that is the problem.

Yes, I call and order a QC key, and Deluxe was very good about doing it. But I shouldn't have to.

I remember writing in the early days of the change to DCI that is was all about studio control, and this is just another example of it.
The the notes from Deluxe always say to test the film as soon as possible is just rubbing salt in the wound.

How do other people deal with testing their shows when the keys are only available a day or less before?

[Frank Cox]

How do other people deal with testing their shows when the keys are only available a day or less before?

In all of the time that I've had my digital setup I can count on one hand (with fingers left over) the number of times that I've played a movie for the public without checking it myself in advance. And those were the rare occasions when the CRU drive didn't show up here until after 4pm for that evening's show and there simply wasn't enough time.

I generally watch 'em on Wednesday night after the regular show or Thursday afternoon. Or Thursday night, depending on when the key is valid.

[Ryan Gallagher]

If I understand Ryan's situation correctly, it's a mixed use venue that plays a lot of arthouse/rep/release movies, and with a multiple projectionists and technicians on the payroll, some with more advanced skill sets than others. What is being contemplated is giving the chief projectionist/technical manager the tools for Internet-based remote access from home, for use when he isn't formally at work. In a theater such as this one, the chief projectionist/technical manager is likely to have a higher level of skills and experience than would be found among the staff of a typical mall multiplex, and therefore leveraging that takes the place of a service contract with an outside technical support vendor.

You did understand our situation correctly Leo. While mostly a live theatre, we are a classic film venue in the summer with about 10 unique films a week for 3 months, holiday season films, 3 annual festivals (one of which is SXSW), and a scattering of studio premieres, roadshow films, and film rental events. Union venue, manually operated booth with staff, still showing 35mm and 70mm too.

For whatever reasons our TD still prefers to get DCPs on CRU drives too!! We are a fossil. Which get shipped to his house (or porch) cause I guess at some point in the past they decided they needed weekend delivery to cover their butts? That is a little crazy pants to me.

We facilitate a lot of things here that would normally involve a tech, but we are not actual techs. Of the venue staff that would be able to offer offsite help it's really mostly me as the principle video/projectionist FT person whom has started down the cinema tech path of knowledge, there is a 2nd but he covers Audio as well in the smaller room and is less available (he's my typical partner for 35mm/70mm days). Perhaps our TD if he is inclined, but he usually only gets involved if there are intractable issues. Replacement/alternate projectionists are rare but they do happen, and we are trying to train up more. Sometimes other house IA department heads will cover in a pinch for DCPs if they have been cross-trained.

And honestly the prospect of remotely helping others is valid, but really i'm just trying to cover my own butt for my own shows, and have some oversight on key validity as they arrive when no one is in the theatre.

KDM Inspector is a great resource, I also use the IOS app of the equivalent at times. But i'm just as comfortable reading the XML to check for our card and server ID match. Our catch was we were not recording CPL names or UUIDs in our track sheet as films arrive. So when a key arrived, other than ingesting it to test, I had no remote way to have a sense of a looming problem unless I thought to go back and check shipping notice emails for CPL name match (that would have caught my most recent issue, but sometimes it's just the UUID that is different).

Cinema-Catcher email monitoring app James describes is something I'll have to look into. Sounds like it can get keys ingested immediately, but i'll have to investigate if it covers validity reporting in any way that precludes VPN access to the server (Doremi DCP2K4 in this case). There is also the issue that we don't leave our server on 247, but could if we wanted to.

How do other people deal with testing their shows when the keys are only available a day or less before?

I get multis and independents doing full screen tests for films that will play multiple days. Just not an scheduling/budget option for the art-houses typically that might show 3 unique films a day. Spot check only if you are lucky. Key windows will always be an issue it seems. Less so with rep/classics, but still. There seems to be no standard adopted for how long a key should be open for a single screening from deluxe or the respective agents. Even when you have a separate QC key, now you have 2 keys that need to be valid, just cause the QC key was good doesn't guarantee the primary is... if it ingests and associates you should be fine of course. Our problem was key ingest is often not being done until the projectionists call for that day.

One happy accident here is that our Doremi server time has drifted (ahead), and as it provides extra tech check time on the primary key, we haven't bothered to fix it. Eventually it might get so bad we have to do the correction though. We don't use a TMS or scheduling, operated booth, so the only real downside is with very short tech keys for festivals, if the talent shows up late and you didn't leave the SPL loaded while waiting for them, you run the risk of them arriving within their window but your key being no longer good.

Our system is reliable as long as we have content and keys at this time. We don't do full DCP play through tests (except when festivals schedule full techs), but we will spot check and look for gotchyas (especially with squirrely aspect ratio changes). We schedule non-festival films such that there is at least an hour and a half of time before doors where I can use the room for testing SPL, confirming aspect ratio and credits timing, checking fader levels, watching the crawl, picking walkin music, yada yada.

[Jim Cassedy]

For several years I worked at a private screening room that was used mainly for press
& preview screenings and occasional special corporate events. Very often, keys or
content did not arrive until the last minute, or, especially in the case of press shows,
there was a last minute change or a 'new version' they wanted to screen.

First of all, I just became very good at 'reading' KDM's & DCP cpls, to spot
differences or discrepancies in file creation dates, version numbers and UUID's.

SInce it was not open every day, and because I was working at several venues, I often
didn't arrive at the venue to physically ingest content and KDM's until the last minute.
The Deluxe online portal was the answer to my prayers, and made it easier for me to
check content & key compatibility somewhat in advance of arrival. When I got too busy
with other (and more lucrative) free-lance projection gigs, I turned most of that job over
to another projectionguy. But he too had a semi busy schedule. This venue had no
ability to receive DCP's electronically, so, often,when I saw via tracking or Deluxe that
a DCP had arrived had arrived, I would stop by the venue for a few minutes on my way
to my main projection gig, and stick the DCP in the server to start ingestion so that it
would be done when he got there later in the day. Sometimes the reverse would happen,
and he would stop by the venue on his way to or from somewhere else to start ingestion
for a show I might be doing in the evening. He was a good projectionist,
but multiple versions and of content & KDM's would sometimes befuddle him. I eventually
set things up in the booth laptop so that I could remote in, via TeamViewer, and push keys
to the server (Dolby DSS 200) and also, VNC directly to the DSS, so I could help him out
if he ran into an issue with a playlist or show. This worked well for a while, but there came
a point where the TeamViewer folks started cracking down on people who were abusing
the 'free' version of TeamViewer for business purposes, and we got kicked off, as did
several other venues in town who were doing the same thing.

"The other guy" eventually got better at the CPL/UUID/KDM decoding and didn't need
my help much more by then anyway.

In my present job, my company uses a corproate SPLASHTOP", which seems pretty
secure andbeing as how I have a high-speed fiber connection at home, it's pretty much
"just like being there" when I have to remote in to my venue. If I'm at work and I need
to pull a graphic or corporate logo DCP I've renedered at home and forgot to bring with
me on a thumb drive, I can remote into my home computer using "Remote PC" for which
I have a paid account, and transfer it directly into our TMS library.

But again- - going back to basics, the first thing I told this other projectionist when he
was taking over the screening room job is that the best thing he could do to avoid KDM
katastrophies was to get a good understanding of opening and reading KDM files to
check UUID's and server serial numbers, etc.