IMB Server Restart Issue Mid-Playback Intermittantly for 3 months

[Ivy Engwall]

This is a Christie Cinelife CP2411 RBe with a GDC Standalone IMB SR-1000. We use a GDC PSD with 3 RAIDs. We just had this laser projector installed in April, and the issue started a few days later, and now it happens every 3-5 days.

It has happened multiple different points in playback, with many different movies (and even during trailers), and sometimes while there is nothing playing. The server will just initiate a restart process (or it disconnects and just immediately tries to reconnect), so playback stops and it shows as disconnected in Cinema ACE. Navigating to the server, you'll find it already starting up again and initializing SMS. When it first started doing this, it would just get stuck and we had to reboot, but now it always completes the reconnect process. Once it comes back up, the server says that "last playback was abnormally interrupted" and we can just hit play (and strike the laser again).

Our tech support has told us they have been in touch with GDC and Christie several times, and we've tried to no avail:

• reseating eSATA cable
• reseating PSD drives
• projector software upgrade at Christie's suggestion
• an entirely new loaner IMB sent from GDC

Has anyone else experienced anything like this? Our tech support has told us that GDC wants to replace the server, but we're already using the loaner IMB, which didn't solve anything. Due to the fact that these issues started after the laser install makes me wonder if it's something to do with how the server connects to the projector. I'm not an tech expert though, it is mostly just theory-crafting based on my 11 years working with digital projection. I'm wondering if anyone here has any ideas or suggestions. Thanks!

[James Gardiner]

If its new, I would ask them to replace the IMB first.
It seems like a internal reset is being triggered in the IMB.
I could be a power issues. Voltages. Power supply in the Projector could also be a cause. (As it supplies the power to the IMB)
Those are my guesses.

[Shibu Paul]

"Based on my experience, GDC with Christie 4th Gen is not an ideal combination, as I have been facing recurring issues with this setup."

[Ioannis Syrogiannis]

[...] and we can just hit play (and strike the laser again).[...]

You mean that the light source would turn off by its own, during the server/IMB disconnection?

All this reminds me of RAM problems, but if you have a standalone IMB, that means you use a NAS or something similar for storage, and not a separate server. So, if the RAM issue is on the IMB, the change of the IMB would have solved it. That is, if I understand correctly.
Then, you write that GDC wants to replace the server. (?)

Besides RAM, James makes perfect sense, writing about power issues (and, hence, maybe projector-IMB connection issues). Cleaning the connectors between the IMB and the projector could be a solution to that. Otherwise, a power drop on the projector could be shown at its logs.

I am not familiar with GDC, so take all I say with a pinch of salt.

[Steve Guttag]

Thus far, I have not had this combination. The SR-1000, for me, has been one of the most reliable with a near 100% up time. Now, my storage has been Enterprise and 4TB Cine-Cache. In the past, I've had nothing but misery on the PSD (the "classic" one with 3-4 laptop drives, not the new SSD version).

To trigger a reboot leads me to believe

• Power fluctuation that the SR-1000 cannot deal with.
• Command(s) that the server cannot deal with and triggers a reboot (for instance, if the projector is polling for information or some security integrity check that overwhelms the server/floods it).

Both of those point to the projector and its interaction. You do have ACE in there that is polling too and perhaps the Christie/GDC combination with those specific models may alter how it interacts. I'd say you can reduce/eliminate the server because you said you already tried a different one with the same results. So, it isn't hardware. GDC could, possibly, alter its code to deal better with how these outside forces are interacting with it.

If GDC is offering a server swap, you have nothing to lose there but I wouldn't have high hopes either. I would be interesting if GDC monitors its voltage rails to see if they see any fluctuations or if that projector is running low at that point in the card cage. You could have a projector power supply that is outputting the proper voltage but once loaded and going through the harnesses and PCB, the level could be sagging.

My experience has been that GDC is pretty good/fast at diagnosing their stuff. I can't remember ever being lead down the wrong path. Christie has been more mixed on tech support. Some Level-1 people just know a script and can't seem to actually diagnose a problem while others are quite sharp. Hopefully, you have a relationship with a dealer/service company that can find the right person at Christie to talk to (as well as GDC but again, they have been more consistent with me on getting to the crux of a problem, if is theirs).

[Marco Giustini]

Originally posted by Ivy Engwall View Post
[...] and we can just hit play (and strike the laser again).[...]

Ioannis has a good point.
If it's just the server rebooting, you shouldn't have to strike the lasers again.

Could it be that you think the server reboots because you see that UI rebooting but actually the whole projector restarts?

You should be able to see what happens from the logs. I feel you might have some supply power issues.

[Mete Tunca]

Which SR-1000 version do you have ? 1K/1Q or 1X version? Which build are you on ? Can you post the dashboard screen ?

Did your service company updated the server to the latest version ?

Also there is a ethernet cable on SR-1000 that connects 2 module. If there is a problem with this cable and its connection you'll experience similiar problem. Did GDC support suggested to check this cable ? Its a small black cat6 cable. Perhaps this needs to be checked.

[Leo Enticknap]

The original poster writes that among the troubleshooting steps tried was installing an entirely new, replacement SR-1000; and that the same fault continued to occur afterwards (unwanted spontaneous reboots). That, IMHO, removes any reasonable doubt that Ivy's suspicion and Steve's diagnosis is the right one, i.e. that the cause is in the interaction between the SR-1000 and the projector.

I don't know if this is relevant, but this is all I can think of. A couple of weeks ago I was called to an issue with an SR-1000 (with Cinecache, not an external RAID box) in an NC1202L. For about a month prior, the theater manager reported that the SR-1000 would spontaneously lose communication with the (GDC) TMS. Scheduled and ingested shows would continue to play, but communication with the TMS was not restored until both the projector and IMS had been rebooted. He didn't know if the SR-1000's web UI went out as well, because he never used it. However, the day before I was called in, a show stopped midway through as well as the communication loss.

I could not reproduce the fault when I was on site: everything was behaving normally. I tested both network cables coming out of the SR-1000, and they came up clean. The projector was on a rather ancient software/firmware bundle version, but I dismissed that as the cause, because the SR-1000 hadn't been updated recently either, and both had worked well together with those versions until the fault started to occur, with nothing having been changed recently (including the TMS version).

I sent a log package to GDC, who responded almost immediately to the effect that they suspected a known issue, and asked me to install a beta version of the SR-1000 software/firmware bundle (build 52; the SR-1000 was on build 39 when the fault started to occur). This I did, tested functionality, and advised the site manager to contact me if the fault reappeared. That was on July 2, and I've heard nothing from him since; therefore, I assume that this update fixed it.

It appears that Ivy's fault is not identical ... but ... in my case and on reflection, it's possible that this SR-1000 could have actually have rebooted itself without the manager noticing, especially if the fault occurred when no show was running. He did tell me that on some of the occasions, he noticed it when entering the booth first thing in the morning to use the TMS, suggesting that the fault had appeared overnight.

Bottom line - maybe it might be a way forward if Ivy (or her tech) sent a log package to GDC and asked if installing build 52 might be worth a shot? If this is a fix, then it would have to be because it modifies the way that the SR-1000 and the projector interact with each other, such as to work around any bug on the Christie side.

[Steve Guttag]

You gotta watch out on the SR-1000 firmware/builds. They have two disparate paths (used to be 3 and possibly more) that "someday maybe" will be merged into 1. There is the 17.3 path and the 17.5 path, at the moment.

There could be something to the lasers needing to be restruck. That starts to point to the projector or possibly its power. Do you have a UPS? If so, any chance it is dropping power? If not, have you tried putting on one line. I'd also check the connections where a UPS can be spliced in, in case the power there is flaky. Then again, the Christie logs should show if there are power interruptions.

[Stefan Scholz]

I did not find the SR 1k to work unreliably in S4 Christie lasers, most of the ones we did seem to run reliably.
I had 2 which stopped mid show and needed a reboot, where actually the short CAT 6 cable interconnecting the board and the Raspie board seemed to show contact issues. But then, a whole bunch of error messages appeared on the Christie screen, all security related issues from tamper switch to tamper batterie were triggered. It disappeared after reboot. In the last case, smacking on the card cage helped.
In my first case, GDC was super quick after receiving the log, with the suggestion to reseat the small cable, and immediately sent a new one.
I replaced with one from our stock, problem gone.

It should not trigger the laser status, this requires a command to be sent over the control network. This indicates power issues, but those, as Steve stated, would be found in the Christie logs.

Also watch out for control network issues. There is no real QOS implemented, and with one setup we had a lot of issues. Often the show stopped midway, and the SR restarted. Here, a common network was used to interconnect the house, private apartment, box office, customer WiFi. Driven by a consumer/ home cable router that was set to control the DHCP leases regardless of fixed adresses, conflict led to loosing the communication between SR1k and S4 projectors. So it mostly occurred when the tenants of the flat returned home, or the fore hall had customers logging into their WiFi.
Disappeared after I restructured the control network into a virtual segment.