Suspected cat745 in its death throes - thoughts appreciated

[Leo Enticknap]

So here I am in the middle of a holiday weekend, having dragged myself out of bed at 4.30am to troubleshoot a fault for one of our service contract customers: intermittent cracks on the left channel, and horizontal white lines occasionally flashing on the picture throughout yesterday's shows.

System is a Barco DP2K-32B, with a DSS220/cat745 and a CP950.

Started by doing the DeOxit and reseat ritual, then played a bunch of trailers. I could not see or hear anything wrong. The only alarm bell was that the CP950's display had a yellow "check cables" warning. AES1-8 is looped through a Fidelio HI/VI breakout box. Through trial and error, I established that if I make a straight shot connection from the cat745 to the CP950, I get a green "connected" indication, but if it's looped through the Fidelio, I get a yellow "check cables."

The CP950's log indicates AES sync problems when the Fidelio loop through is in place.

CP950_errors.jpg

This is where it gets interesting. Through more trial and error, I established that the left channel cracks and white flashes happen when playing encrypted content, but not unencrypted. This is definitely consistent behavior, established by around an hour of playing a random five minutes from two encrypted features, followed by a couple of trailers.

I then swapped cat745s between two screens, and the fault moved.

My hypothesis, therefore, is that the cat745 is starting to fail. It is still able to handle the processing tasks necessary to play unencrypted stuff, but decryption is sapping too much power out of it. The yellow audio warning when looped through the Fidelio is because there is a slight voltage drop on the AES output: not enough to cause a problem with the connection is a straight shot, but enough to degrade the signal after the Fidelio has sapped it a bit more. Does this sound feasible?

Anyways, I suspect that an IMS3000 sale is imminent. Hope they don't have to wait too long for it.

[Harold Hallikainen]

The audio problem is interesting. I don't know how the Fidelio taps onto the AES audio. Possibilities include bridging across the line with a high impedance (seems most likely); terminating the line, receiving the audio, and either transmitting it or leaving pair 4 empty; a splitting pad or transformer to split the AES between the HI/VI device and the pair 4 output. But, AES3 is VERY tolerant. It is hard to believe the voltage would drop far enough with the Fidelio inline to cause the CP950 to have trouble receiving it. But, you have verified that SOMETHING is happening. Interesting! The encrypted versus non-encrypted is also interesting. But, I don't know enough about the CAT745 to offer any solutions.

Good luck!

Harold

[Leo Enticknap]

Audio still plays when the CP950 is in the "warning - check cables" state, so the three AES pairs that the Fidelio is passing through clearly aren't degraded enough that the CP950 can't play them. The Fidelio does leave pair 4 empty on the output end, because the CP950 registers no signal at all on it.

This doesn't affect the intermittent clicking / flashing behavior: it's audible and visible when encrypted content is being played, but not unencrypted, regardless of whether AES 1-8 is looped through the Fidelio or not.

I'm hoping that they will decide to upgrade this house to an IMS3000, because unlike the DSS220, it allows the reconfiguration of DCP to AES output channels in the media block. So with one of those, I could send HI and VI to 15 and 16 rather than 7 and 8, and the rear surrounds to 7 and 8 rather than 11 and 12; then hook AES 1-8 as a straight shot into the CP950, and 9-16 directly into the Fidelio. With that arrangement, any malfunction of the Fidelio could not disrupt the main audio.

Eliminating that vulnerability is the main advantage of the USL HI/VI system over the DolReMi one. The disadvantage is that you have to have an audio processor that will output the HI and VI channels as line level analog (or an external AES to analog converter somewhere in the signal chain and a physical adapter to separate one pair from the others coming out of the media block), whereas with Fidelio, you don't.

[Mattias Mattsson]

Could it be that somehow the Fidelio messes up the forensic markings of the audio tracks on this specific encrypted DCP? Does the problem occur on all encrypted content? What channels get forensically marked could be different for different DCPs.

[Harold Hallikainen]

OK, I did not know a whole lot about Fidelio. If it interrupts pair 4, I suspect the warnings about it (probably the sample rate 0 log entries) are normal and do not indicate a problem. But what are the other log entries about no b-chain config? That almost sounds like a configuration issue instead of a signal input issue.

On forensic marking, I doubt the CP950 cares. It just passes the marked audio through and does not evaluate it at all. It does, of course, generate forensic marking when running immersive audio.

So, the audio warnings in the log may not be related to the issue. But the cracks in the left channel and the horizontal white lines DO indicate an issue and are MAYBE related to each other. The mention of forensic marking reminds me that it was a challenge to get forensic marking working on a large number of channels. Failures would be a crackling in the audio. But the forensic marking is handled in a DSP (or maybe two), while the video is all handled in the FPGA. Video decryption is definitely in the FPGA. Audio decryption MIGHT have been in the main CPU (I get the different models of the IMB mixed up). But audio and video both pass frame buffers run by the FPGA on the way out.

Good luck!

Harold

[Leo Enticknap]

There were two encrypted DCPs with open keys on that server when I found it (Uncharted and Dog). I tried both of them, and observed the cracks and flashes on both of them.

The no B-chain config on array X errors are likely because I used Dolby Atmos Designer to set up the 950, but for regular 7.1 only. You have to do it this way in order to use the 950's internal crossover, which I needed to, because the stage channels are tri-amped (the main reason we installed the 950 was because an external crossover had failed, and the pre-existing CP500 and DAX-602 combo likely didn't have long to go). So it's complaining about arrays that don't exist (there are no top surrounds, bass management, etc.), and therefore I'm pretty sure that these messages are nothing to worry about.

I suppose the definitive test would be to try an Interop encrypted DCP (audio cannot be encrypted), to see if the fault presents. If it doesn't, we possibly are looking at an issue with watermarking in SMPTE encrypted DCPs. That having been said, Uncharted was the movie that they are currently playing in this screen and for which they reported the problem. It is also playing in two other DSS220/cat745 screens, one of which with a 950 and the other with a 750, and they are not reporting this fault in any of the others. All the DSS220/cat745s are on the same version, too (4.9.5.2). So on balance, I still suspect a hardware fault with this specific IMB.

Thanks, folks.

[Harold Hallikainen]

I also suspect a fault in the IMB. On the audio, are all DCPs the same number of channels (5.1 or 7.1)? For a 5.1, the CPL may say to mark channels 1 - 8, while with 7.1, it might say to mark 1-12, which is harder for the watermarking DSP to do. But, maybe not! I think there are two watermarking DSPs, each one handling 8 channels.

I know I'm not being much help here!

Harold

[Leo Enticknap]

You, and Matthias, are being a very big help - watermarking being behind this problem is not something I'd have thought of by myself. Just checked remotely - both Uncharted and Dog on that server are the 7.1 versions, but two other screens that are not wired for 7.1 and in which the fault has not been reported, are playing the 5.1 version of Uncharted. So I think you could be on to something. If I had more time to investigate, I'd ask them to get a 5.1 key for the 7.1 screen's IMB, and try that version. Sadly, I'm completely slammed for the rest of this week. Anyways, this is their biggest screen (and the only one with leather recliners and a higher ticket price), and they want a guaranteed, permanent fix, ASAP. Replacing that DSS200/cat745 seems to me to be the most sensible option at this point.

[Stefan Scholz]

What I meanwhile like about the old Dolby system is the hardware, which seems to be commercial grade quality, over the usual industrial grade used for the other vendor's stuff. It should, besides battery depleation, be quite stable from my investigations on the ones I was given.
But as it is a product with End of Life in Oct 18 (?), for safety reasons in an important screen, I'd personally go the way of the IMS 3000. It's just the long lead time to get one, which makes it difficult these days. Still waiting for mine ordered in Dec...

[Carsten Kurz]

Leo - encrypted Interop DCPs will not have subtitles encrypted, but audio still. However, Watermarking can be disabled/enabled on a per channel basis in the KDM (in order e.g. to protect sync/data tracks). The recent version of DCP-o-matic allows you to activate/deactivate Watermarking for video and audio when you create a KDM.

[Leo Enticknap]

Confirmed that it was a bad cat745. Replaced it with an Alchemy today (wait time for an IMS3000 ruled that option out): no cracks, flashes, or errors generated by the CP950 playing the same DCP that gave the trouble last week.

One advantage gained was that the Alchemy let me map LRS and RRS to 7 and 8, and HI/VI to 15 and 16. So AES1-8 is now a straight shot to the 950, and AES 9-16 goes to the Fidelio. No need to loop anything through it, and so a single point of failure is now removed from the audio signal path.

[Steve Guttag]

Until the mapping un-maps itself or until the next ICMP swap. I do try to avoid mapping in the server...though sometimes it is unavoidable. I tend to make custom cables and do the swapping in a DB25. Or...Q-SYS...put things where you want them!

[Leo Enticknap]

Custom cables create an equal risk of that outcome. If that cable breaks and is replaced by someone other than the tech who made and/or installed it, they likely won't know that the pin routing is customized, replace it with an off-the-shelf one (or one with standard pinouts for the equipment), and then wonder why things don't work. This happens to me once or twice a year.

[Steve Guttag]

Hmm...I've never had one of my cables "break" like that. Seriously...in 40+ years...never once. Channel mapping not working...been burned by both GDC and Doremi...multiple times. It is not an equal risk.