The Y2k24 BUG! MAJOR DIGITAL OUTAGE TODAY!

[Carsten Kurz]

Well, it's only a small part of repertoire titles, actually. It doesn't happen with interop DCPs, and SMPTE DCPs are only around since 2 years or so (we still get many interop titles today). I don't expect companies to ignore future cert expirations from now - commercial mastering tools will certainly warn or display expiration dates.

As this is easy to check, it would certainly be nice to get an idea about how many titles will have issues in forthcoming years.

We'll see how Sony does with this. There are many systems still under active service contracts, and the code change needed is small.

It is certainly a good idea to develop some scheme to check incoming CPLs signer certs for repertoire titles though. Clairmeta and dcp_inspect are there to aid.

[Paul H. Rayton]

Happily, the ISDCF (Intersociety Digital Cinema Forum) meets this Thursday (Jan. 11 at 10AM, Pacific time), and (we can hope) they'll take this issue up, as they are the ones who represent both the cinemas and the equipment manufacturers -- all those who are supposed to make d-cinema "go". I've personally informed one active member, and if there are any other ISDCF members reading this, please reserve the time on Thursday to get this mess sorted out once and for all! Talk about the "law of unintended consequences"!

[Frank Cox]

I don't see anything "unintended" here at all. The certificate expired and it quit.

Invalid certificate = no movie for you.

Negligent may be the word that fits the situation better.

[James Gardiner]

ISDCF meeting will definitely devote some time to this issue, and how to avoid it in the future.
The problem, in my opinion, is just an unfortunate set of events that led to an undesirable result. For example, the mission to the moon to land a craft. A small issue likely, not considered that serious was not given the time to make it solid. has lead to a failed mission. Similar to what has happened here. The complexity of DCI makes it inevitable that a few issues like this may fall through the cracks. Luckily the damage was very limited.

I expect the discussion will go along the lines that vendors (manufacturers and service providers) are encouraged to develop mitigation projects to catch this and similar possible issues going into the future.

DCI has already addressed the error in that, it did not define what should happen when this issue occurred.
i.e, typically if a cert connected to a process is expired, it would void that process.
But. the signature is NOT technically part of the security checking process the media block uses to play encrypted content. It's a GRAY area.

DCI has now updated the spec to specify that the signing certs used in the process, and its expiry date are not a condition to refuse to play.

All I see is a lot of engineers working overtime for many months.

My biggest worry is.. It shows that equipment like Sony and other EOL and non supported equipment, really need to be deprecated. Any idea how many are in the field?
Its not really a good time for many cinemas to have to re-invest in a 10-year plant and equipment purchase. Attendance levels are still 20%+ down in most territories.

[Carsten Kurz]

Why don't we wait for an actual definitive statement from Sony before we call any equipment deprecated?

[James Gardiner]

Why don't we wait for an actual definitive statement from Sony before we call any equipment deprecated?

Deprecated my be the wrong word here. It's not that it shouldn't be used anymore, but that it requires a specific constraint to make sure it works going forward. It's also not just Sony that may be affected.

In any case, I don't see Sony pulling the development team back together simply to address this issue. Sony Digital Cinema is gone from my understanding. It would not be trivial to get into a position where they can do revisions of the software. I expect the development labs and equipment to be gone. It would take a long time to get into a position to even address the issue. I don't see it happening. (But who knows)

Considering now, the way Sony players work but are now not compliant with the DCI specification as updated in the recent release. I know ISDCF was looking into nomenclature for all the possible conditions equipment can be in. I am unsure what to call this.

Sony and other equipment will still work, but is now severely limited by content that may be affected by this issue with signing certs.
I see the major KDM management entities just putting in processes to ensure signing certs are not expired.

Still, this bug will likely cause more issues with entities trying to play older and long lived DCPs.

[Frank Cox]

Is the software that runs these things really such a rats nest of garbage code that it would take a team of engineers several months to comment out one if block?

*boggle*

The right thing to do on the part of Sony (and any other outfit who is in the same position) would be to assign an engineer to retrieve the code from the archive, comment out the offending function and recompile it for distribution to those who have affected equipment. With a properly maintained archive and a qualified person to do the job he should be able to get the job done before lunch. Anything past that means that there are problems that extend past a bug in the software. It is, after all, just one function. Or it should be, and every keyboard has a ; key.

"We may not be contractually obligated to provide a fix for this issue but we recognize that this problem has been caused through no fault of our customers, the same customers who have given us a great deal of money for our equipment in the past."

The relevant concepts here are good will and customer service, which has always been shown to pay for itself many times over.

[James Gardiner]

Frank, its not about code quality. Its about building on top of other complex systems.
Firstly, you have to setup a development environment, and as DCI is a snowflake-type environment. How many programmers in the world do you think there are that understand DCI and its complex implementations? Then finding a coder, that understands the development stack chosen. build environment, languages used. etc etc. Then having a knowledge of the DCI implementation. etc.

The typical rule of thumb is a new coder in a larger enterprise takes 3 months minimum before they are considered up to speed.. And that's on a standard every day build environment with other developers who know the environment helping them. Not a custom hardware etc like we see in DCI.

Typically a new coder has other coders who know how all this works and guide him through the process.
Unless the old coders who build the Sony system and on hand. (And not left and have a new job with other companies) it would take a very long time indeed to get to a position to compile the code let alone work on the issue.
And the issue is not a BUG in the code. Its a change in expected implementation.

final note:
There are $100 of millions in projects by other entities on that rocket that is going to crash into the moon. Its the risk of doing business.
As the Sony player can continue with the issue if proper care is taken in how certificate signing is done. Thats how I expect it to shake out.

[Frank Cox]

It's not a complete rewrite of the code that runs the server. It's one if block.

Retrieve the relevant code from the archive.

Set up the build environement.

Edit the code to remove the offending block.

Recompile.

You don't even need to hunt through everything for what calls that function. Just have it return "true". (Or false. Whichever we need here.) Done.

If software is written such that it can't be reproduced on demand then it's unfit for purpose and it needs to be refactored and rewritten until is reproducible before it's distributed in the first place.

I've read many arguments stating that software programmers should never be allowed to call themselves an "engineer" and it's situations like this that make me think they're right.

[James Gardiner]

Frank, I feel you underestimate the issue here.

Firstly, how they build the certificate chain authentication process is likely very complex. You will need to understand that complexity and implementation. Then based on the specific error occurring, you need to filter it out or change the code to ignore the certificate signing part of the process.. How this is implemented is far more involved than an IF statement. I have dabbled in SSL and certificate issues. It's not for the faint of heart.
Then, after you build the development environment, which could take weeks for someone never exposed to this complex one that is likely connected to custom hardware, likely requiring complex cross compilation etc.
THEN you need to understand the custom build code signing mechanism they must use to build an update patch. Again likely super complex and snowflake custom.
If the development team is completely gone, and they cannot get key personnel back. it would be very difficult even to change a line of code if starting from scratch.

Software development is very expensive, time-consuming and ultimately very hard to do. Especially these esoteric security-heavy implementations. And even more so for custom hardware.

Those entities that have an active team that are continuing to improve the software, building new versions every day. Looking over the code base every day. Yes, they will likely to able to fix this in a few days. But once you disband that team and the years of understanding they have absorbed to get to that position are gone. Those years of understanding or part thereof need to be re-implemented to put you in a position to release an updated version of a codebase.

[Frank Cox]

"Sucks to be you" shouldn't be an acceptable answer for people who have spent tens or hundreds of thousands of dollars on their hardware.

I've heard people saying "glad I didn't buy a Sony" in other contexts; I guess this is turning into another example.

But... I'm glad I didn't buy a Sony.

[Marcel Birgelen]

Software development is very expensive, time-consuming and ultimately very hard to do. Especially these esoteric security-heavy implementations. And even more so for custom hardware.

That's exactly my argument against this entire DCI security spaghetti.

Does it add any substantial extra security to the current market, where most movies are available in HD formats via alternate routes? No.
Does it add quality to the presentation standards? No.

So, it only ads complexity on all levels and it greatly reduces maintainability of the fleet of machinery we have out there.

You think it's hard to play a 35mm movie in 2024? Wait until someone wants to play a DCP mastered in 2024 in 2050...

[Leo Enticknap]

That analogy speaks to part of the issue. In the 35mm days, the companies that made projectors and related hardware had adapted to a business model whereby their customers only expected to replace their projectors every 30 to 50 years or so. They were long established, small (rarely more than 100 employees), low staff turnover, and their designs and infrastructure evolved at relatively low cost, rather than frequently being revolutionized after a large R & D investment. If you bought a Cinemeccanica Vic 8 in 2008, it was almost a carbon copy of the projector you would have bought in 1968.

The manufacturers of digital cinema projectors, servers, and supporting IT hardware are megacorps whose customers think in terms of five to ten-year lifecycles. For the likes of Barco, Sony, and SharpNEC, cinema is essentially a side hustle, given the volume of sales they're looking at. The first mass manufactured (by cinema industry standards) generation of DCI equipment is now hitting EOL in significant numbers of units, and we're seeing the business models clash between industries that are geared to producing units in quantities of tens of thousands a year with a lifecycle of five to ten years, and customers that are geared to buying equipment made in quantities of a few hundred a year, and with a lifecycle of 30 to 50 years.

[Carsten Kurz]

Sony issued an official statement about continuing support until the end of 2027. They have systems still under SLAs, also their TMS. So, they clearly need to sustain a minimum crew to fulfill that obligation. We still get parts. So, someone is there.

[Steve Guttag]

There is also the realities that, particularly in exhibition, there are not financial resources/ability to have short equipment lifecycles. That said, it would appear that "DCI Compliance" was a "spectrum" and there was room for interpretation. So, everyone can claim DCI compliance and meeting such and such standard and yet there is incompatibility. This is a problem that we really didn't have with film, particularly once the SMPE->SMPTE standardized things. The film width, perforation size/pitch/location were all well defined and, as such, there isn't a question of a 35mm film made in 2000 running on a projector made in 1930 (or even 1920 and earlier...give or take sound).

There is no question that this industry is over-zealous when it comes to the whole KDM security thing. Nothing highlights it more than studios using KDMs for content that has already made it to the home market via media or streaming. It just becomes a nuisance at that point with no security value (since a bad actor could rip a disc, copy a stream...etc.).

Note, on the Sony stuff, it is the LATER models, the 515 and 815 that had the issues, not the 220 and 320.

We have a decent amount of Dolby DSS200 servers installed...there is now a sizable number of 4K content with a bad header block (or incompatible header block) that will cause image issues. I've talked to Dolby and it isn't a matter of them recoding something as it baked into the components they were using. They'd have to go back to those manufactures and have them crack open their code to fix something that is now approaching 10-years after the last one was sold. Note, this can affect other mediablocks of that vintage too) Or...the software that authors content could be fixed to not have that header issue...which is easier to accomplish? Clearly it is the authoring software...but okay...brand A fixes it and they author compatible DCPs....and 6 months or 2-years later another company brings out an authoring software and...ding...the problem starts up again.

It almost should be a DCI requirement for the authoring software to test their output on legacy servers/mediablocks until such time as the industry, as a whole, declares something deprecated but, in no case should that ever be less than 15-years after the last one sold. Of course, once you set a time frame, you, practically, declare the life cycle of the product. This industry cannot afford 7-10 year life cycles. it is one thing for wear and tear items like carpet and seats...servers, projectors, sound systems should last longer and be serviceable.