Lamp Not Turning On, Dowser Not Opening?

[Nathan Paris]

Hey everyone! We use Christie CP2220s, Dolby DSS200s, and Cat862s.

I have 7 screens at my theater, I have had an issue the past few weeks with two of my projectors not having the lamp turn on or the dowser open when it runs a playlist. It ignores the cue so we have to manually turn the lamp on and open the dowser and then when the show ends it also ignores the cue and we have to go and turn the lamp off and shut the dowser. The house lights still went down as normal according to their cue.

I've had this happen once in a while across my time working but it has been happening repeatedly the past few days for about a week. Restarting everything after the first show fixes it for the next (and remaining) show of the day but then the next day when we power on all our equipment again it has a chance of happening again.

One other issue related that we had is that one of those screens was constantly repeating the first few seconds of the playlists when we looked at it on our Arts Alliance Media Screenwriter and when trying to override it on the DSS200 the show control was locked.

Any help would be great! I've been trying to look for any existing topic but all I seem to find is bulb related issues, nothing on the lamp/dowser just not turning on or off despite having cues to do so.

[Leo Enticknap]

To distill this a bit, is the situation that all cues sent from the server to the projector are failing to work? If so that, combined with the repeating the first few seconds behavior, suggests to me that the fault could be in the DSS200.

If it were me, the first troubleshooting step would be to try to send API commands to the projector from another device (even manually via an SSH or Telnet connection if necessary: don't know what Christie uses). If they work OK, that would narrow it down to the DSS200. Next, can you ping the projector from the DSS200 successfully? If no, there is a communications problem (maybe another device has the projector's port locked open?). If yes, something is not right with what the DSS200 is sending to the projector. In that scenario, my suggestion would be to "nuke" the DSS200, do a clean install of the software, reconfigure it, reingest everything from the TMS, and see if the problem goes away.

and when trying to override it on the DSS200 the show control was locked.

You can unlock it in the system > theatre tab, as so:

image.png

[Ryan Gallagher]

Not an expert, but this feels automation/network related. Are the lamp and shutter commands via TCP/IP or some other method?

One thing that happens with our CP2220 and TCP/IP commands (and I haven't learned exact cause) is that the TPC embedded windows will get very laggy over time or randomly. When it's in that laggy state VNC connections are difficult to use, as well as incoming TCP/IP automation commands will execute extremely late, perhaps in worst case get missed?

Our "band-aid" unfortunately is to force the TPC to reboot, and while you can do with the lamp on and content running, you lose some lamp power management features until the next full projector standby cycle. We have avoided doing it during shows. But I will reboot the pedestal or TPU only if I start the day with that interface feeling sluggish, and have mostly avoided cue lag issues since adopting that practice.

I suspect our network infrastructure doesn't have the Booth equipment as isolated as it should be, and the system can get kinda clobbered dealing with broadcast type network traffic and ARP requests, ending up in some kind of less than responsive state. I need to poke our IT dept to look into it.

[Nathan Paris]

To distill this a bit, is the situation that all cues sent from the server to the projector are failing to work? If so that, combined with the repeating the first few seconds behavior, suggests to me that the fault could be in the DSS200.

If it were me, the first troubleshooting step would be to try to send API commands to the projector from another device (even manually via an SSH or Telnet connection if necessary: don't know what Christie uses). If they work OK, that would narrow it down to the DSS200. Next, can you ping the projector from the DSS200 successfully? If no, there is a communications problem (maybe another device has the projector's port locked open?). If yes, something is not right with what the DSS200 is sending to the projector. In that scenario, my suggestion would be to "nuke" the DSS200, do a clean install of the software, reconfigure it, reingest everything from the TMS, and see if the problem goes away.



You can unlock it in the system > theatre tab, as so:

image.png

So I believe some cues are still coming through as our house lights will still go down as normal when it hits that cue in the playlist, it's just specifically been the dowser and lamp that isn't turning on/opening. I'll try pinging the projector. I've had to run a clean install of the DSS200 setup before so if it keeps happening then I'll do that!

[Nathan Paris]

Not an expert, but this feels automation/network related. Are the lamp and shutter commands via TCP/IP or some other method?

One thing that happens with our CP2220 and TCP/IP commands (and I haven't learned exact cause) is that the TPC embedded windows will get very laggy over time or randomly. When it's in that laggy state VNC connections are difficult to use, as well as incoming TCP/IP automation commands will execute extremely late, perhaps in worst case get missed?

Our "band-aid" unfortunately is to force the TPC to reboot, and while you can do with the lamp on and content running, you lose some lamp power management features until the next full projector standby cycle. We have avoided doing it during shows. But I will reboot the pedestal or TPU only if I start the day with that interface feeling sluggish, and have mostly avoided cue lag issues since adopting that practice.

I suspect our network infrastructure doesn't have the Booth equipment as isolated as it should be, and the system can get kinda clobbered dealing with broadcast type network traffic and ARP requests, ending up in some kind of less than responsive state. I need to poke our IT dept to look into it.

Our TPC has been very laggy in any of the instances I've dealt with it, so I think you're spot on with that. We shut off all of our equipment nightly so that would functionally cause the TPC to reboot, wouldn't it? Or is there a more official way to specifically force a reboot of the TPC? I believe our cues run through IP? It's all serial automation through the DSS200.

[Ioannis Syrogiannis]

[...]
So I believe some cues are still coming through as our house lights will still go down as normal when it hits that cue in the playlist, it's just specifically been the dowser and lamp that isn't turning on/opening. I'll try pinging the projector. I've had to run a clean install of the DSS200 setup before so if it keeps happening then I'll do that!

Yet, the light cues are not cues that are sent to the projector, right? We don't even know if the light cues are running through the auditorium/control network or serial connection.
If it was the lens' cues that were still working, then it would have been meaningful.

I support Leo's suggestions. The problem seems to be on the network connection between the screen server and the projector (Ryan also mentioned it). I would (besides that) underline the fact that you use a third party TMS (not the Dolby one, that would have needed license, I suppose, for seven screens(?) and is abandoned by Dolby for the sake of the Doremi one, that was... also abandoned).
So, before a fresh install, I would first check if the problem resolves itself by
1) checking -and maybe reducing- the polling frequency between the TMS and the projectors
2) while keeping the projector fully on (the lamp makes no difference) and connected to the server, running anew the configuration setup (the server is retrieving the automation cues/channels during that procedure)

Best of luck!

[Ryan Gallagher]

Our TPC has been very laggy in any of the instances I've dealt with it, so I think you're spot on with that. We shut off all of our equipment nightly so that would functionally cause the TPC to reboot, wouldn't it? Or is there a more official way to specifically force a reboot of the TPC? I believe our cues run through IP? It's all serial automation through the DSS200.

I agree it depends a bit on which cues are working and which are failing. Could be a mix of serial/TCP/gpio and different destinations. If ALL projector cues are suffering but nothing else is, that kinda focuses your attention.

When you say power down nightly, do you mean “stand-by” mode, or completely removing power? Stand-by notably DOES NOT cause the TPC to reboot.

I bet there is a more kosher way to do a clean shutdown of just the TPC, but simply unplugging it and replugging it will force it to reboot. At the risk of damaging it’s little connector if done all the time.

Again, this, if it helps at all, is just a band-aid and not really solving the underlying problem.

Before I started rebooting ours everyone avoided projector automations like a plague cause they were so unreliable in their timing. I would prefer some automation in some situations so had to wrestle with it to find a way forward.

[Ryan Gallagher]

Sharing which firmwares and software versions the CP2220 is on might also be useful info. I know ours is probably behind a couple versions... but it's not the kind of thing my org likes to touch unless something is not working. (Obviously something isn't working quite right, but it hasn't risen to the level of making my superiors want try updating to fix it yet).

It's a hard thing to recreate under test conditions... but if you can get a moment when it is misbehaving, I agree with Leo that having a second device you can try sending projector commands from would be good. If it is non-responsive or super delayed on those too then that narrows it down to the christie automation handling or the network situation. I should do so myself!

[Caleb Williams]

CAT862's in 2023? Wowza, that's impressive.

The Christie series 2 models require the TPC to be fully booted and functional to accept and process automation command sent via tcp/ip on port 5000 (Christie calls it serial even though its not). There is a suite of applications that run on the TPC that communicate with the PIB and manage the rest of the projector. Occasionally, the process responsible for listening for commands on port 5000 crashes out or just doesn't start when the projector boots up. The only way to fix this is by power-cycling the TPC, either by powering off the whole projector, accessing the system via the TPC menus, or using a telnet connection to trigger it. We have a lot of series 2's and there seem to be a few projectors that consistently have this issue, while the rest of the fleet works just fine, or only very rarely has this issue. If the TPC is "laggy" then the flash card in it may be experiencing a slow failure. Corrupted files on the flash can also cause parts of the system to not work (like automation-related processes).

If other automations are working from the DSS200 like the house lights and others, I'd bet money on the TPC not functioning correctly. As others have mentioned, it really isn't a good idea to unplug the cable between the TPC and the projector a lot, else it may get damaged. The safest and easiest way to soft-reboot the TPC is to go to the menu, login, then go Service Setup->System Access->Explorer. Then you can click on the start button, tap shutdown, and then make sure "Restart" is selected before continuing. For the houses we have that constantly have this issue, we kind of Russian-roulette-restart the projector until everything is working, then tape the breaker off so it doesn't get shut off and only power the projector down to standby mode. After doing that we can go months without issues.

You can easily confirm if the projector is having issues by using a command line utility to send it some commands, and see what the responses are, provided you have a machine on the same network. If you use putty, create a new raw connection pointed to the projector's IP address and use port 5000. Once the connection is open, type the string (PWR?) and press enter. If you get something back like (PWR!003 "Power Off") or something similar, then the automation system on the projector is working. If you don't get anything back within a couple seconds then try again. If nothing comes back still then the TPC automation process is the problem. You can also use net-cat, a linux utility, to accomplish the same test if putty isn't your jam.

At one location, I have a bash script deployed via cron-job that will detect the failed process and reboot the TCP automatically. Thus far, problems caused by this issue (like channel not changing or lamp not turning on), have been non-existent. I've only deployed that at one location for testing, because I don't know the long-term consequences of rebooting the TPC many times unattended. Based on the logs my script has kept, every series two projector at that location has had this issue at least once in the last month, though I don't currently have a way to tell if the issue happens at bootup or sometime during the day. If anyone is interested I can provide the source here.

[Oleg Batiaiev]

At one location, I have a bash script deployed via cron-job that will detect the failed process and reboot the TCP automatically. Thus far, problems caused by this issue (like channel not changing or lamp not turning on), have been non-existent. I've only deployed that at one location for testing, because I don't know the long-term consequences of rebooting the TPC many times unattended. Based on the logs my script has kept, every series two projector at that location has had this issue at least once in the last month, though I don't currently have a way to tell if the issue happens at bootup or sometime during the day. If anyone is interested I can provide the source here.

Hi, that's interesting. Can you explain how the script reloads the TPC if it turns out that the automation system is not working?

[Steve Guttag]

A high percentage of my CAT862 systems are still alive...despite my dire warnings that they are due for sudden-death as I have seen it happen a bit.

[Ryan Gallagher]

CAT862's in 2023? Wowza, that's impressive.

The Christie series 2 models require the TPC to be fully booted and functional to accept and process automation command sent via tcp/ip on port 5000 (Christie calls it serial even though its not). There is a suite of applications that run on the TPC that communicate with the PIB and manage the rest of the projector. Occasionally, the process responsible for listening for commands on port 5000 crashes out or just doesn't start when the projector boots up. The only way to fix this is by power-cycling the TPC, either by powering off the whole projector, accessing the system via the TPC menus, or using a telnet connection to trigger it. We have a lot of series 2's and there seem to be a few projectors that consistently have this issue, while the rest of the fleet works just fine, or only very rarely has this issue. If the TPC is "laggy" then the flash card in it may be experiencing a slow failure. Corrupted files on the flash can also cause parts of the system to not work (like automation-related processes).

If other automations are working from the DSS200 like the house lights and others, I'd bet money on the TPC not functioning correctly. As others have mentioned, it really isn't a good idea to unplug the cable between the TPC and the projector a lot, else it may get damaged. The safest and easiest way to soft-reboot the TPC is to go to the menu, login, then go Service Setup->System Access->Explorer. Then you can click on the start button, tap shutdown, and then make sure "Restart" is selected before continuing. For the houses we have that constantly have this issue, we kind of Russian-roulette-restart the projector until everything is working, then tape the breaker off so it doesn't get shut off and only power the projector down to standby mode. After doing that we can go months without issues.

You can easily confirm if the projector is having issues by using a command line utility to send it some commands, and see what the responses are, provided you have a machine on the same network. If you use putty, create a new raw connection pointed to the projector's IP address and use port 5000. Once the connection is open, type the string (PWR?) and press enter. If you get something back like (PWR!003 "Power Off") or something similar, then the automation system on the projector is working. If you don't get anything back within a couple seconds then try again. If nothing comes back still then the TPC automation process is the problem. You can also use net-cat, a linux utility, to accomplish the same test if putty isn't your jam.

At one location, I have a bash script deployed via cron-job that will detect the failed process and reboot the TCP automatically. Thus far, problems caused by this issue (like channel not changing or lamp not turning on), have been non-existent. I've only deployed that at one location for testing, because I don't know the long-term consequences of rebooting the TPC many times unattended. Based on the logs my script has kept, every series two projector at that location has had this issue at least once in the last month, though I don't currently have a way to tell if the issue happens at bootup or sometime during the day. If anyone is interested I can provide the source here.

Great information Caleb. We'll switch to the soft reboot methods for sure in our booth. The cron script is also a great idea... I don't currently have a machine to run it on though. Maybe one day.

I can confirm at least from our experience that we've never encountered the issue immediately upon reboot, it's always something that grows or appears on it's own at a later date, and coincides with the TPC UI becoming very sluggish to redraw your requested changes (like if you open the menu or go to an info or configuration screen). Next time I catch ours doing it I'll try exiting to explorer and check the task manager to see where memory usage and cpu usage are at. It could be some misbehaving process and it starts getting laggy when it is trying memory "swap" using the CF card.

What is the network command you are using to tell the TPC os to reboot? I'm not too familiar with windows remote commands. If you shut it down does it just auto-reboot because it's powered from the projector still?

[Caleb Williams]

The script is as follows:

#!/bin/bash

# COLOR MESSAGES (use with echo -e and quotes)
error_txt="[ \e[31mERROR\e[0m ]"
ok_txt="[ \e[32mOK\e[0m ]"
warn_txt="[ \e[33mWARN\e[0m ]"
info_txt="[ \e[34mINFO\e[0m ]"

PROJ=$1

if [[ -z $PROJ ]]; then
echo "Usage: <script.sh> [TARGET-IP]"
exit 1
fi

log_message()
{
msg=$1
if [[ $2 == "ok" ]]; then
echo -e "[ $(date +%Y-%m-%d) $(date +%H:%M:%S) ] ${ok_txt} $msg"
elif [[ $2 == "err" ]]; then
echo -e "[ $(date +%Y-%m-%d) $(date +%H:%M:%S) ] ${error_txt} $msg"
elif [[ $2 == "warn" ]]; then
echo -e "[ $(date +%Y-%m-%d) $(date +%H:%M:%S) ] ${warn_txt} $msg"
else
echo -e "[ $(date +%Y-%m-%d) $(date +%H:%M:%S) ] ${info_txt} $msg"
fi
}

get_projector_online()
{
ping -c 1 -q $PROJ &>/dev/null
if [[ $? == 0 ]]; then
echo 0
else
echo 1
fi
}

get_projector_healthy()
{
H=$(echo '(PWR?)' | nc -w 2 $PROJ 5000)
if [[ -n $H ]]; then
echo 0
else
echo 1
fi
}

tpc_reboot()
{
{
sleep 2
echo -e "[USER]\r"
sleep 2
echo -e "[PASSWORD]\r"
sleep 1
echo -e "shutdown /r /t 0\r"
sleep 1
echo -e "exit\r"
} | telnet $PROJ &>/dev/null
}

# 1. Get Projector online status w/ping
P_STATUS=$(get_projector_online)
if [[ $P_STATUS == 0 ]]; then
ONLINE=$true
log_message "Projector at ${PROJ} is online" "ok"
else
# no logging if projector fails ping test
#log_message "Projector at ${PROJ} is offline" "err"
# try again in case projector is booting
ONLINE=$false
sleep 300
$P_STATUS=$(get_projector_online)
if [[ $P_STATUS == 0 ]]; then
$ONLINE=$true
else
# projector is offline and we failed check
exit 1
fi
fi

# 2. Get API status via query on port 5000
# check ping status 1st
if [[ $ONLINE == $true ]]; then
H_STAT=$(get_projector_healthy)
if [[ $H_STAT == 0 ]]; then
# Projector is healthy. Nothing to do
log_message "Projector at ${PROJ} is healthy" "ok"
exit 0
else
log_message "Projector at ${PROJ} is not healthy. Waiting a few minutes to confirm" "err"
# Projector is not healthy. Double check
# if its consistent, then reboot tpc if needed
sleep 180
H_STAT=$(get_projector_healthy)
if [[ $H_STAT == 0 ]]; then
log_message "Projector at ${PROJ} is healthy" "ok"
exit 0
else
# run tpc reboot
log_message "Projector at ${PROJ}: Rebooting tpc..." "warn"
tpc_reboot
sleep 180
# see if it worked
H_STAT=$(get_projector_healthy)
if [[ $H_STAT == 0 ]]; then
# we are done
log_message "Projector at ${PROJ} is now healthy" "ok"
exit 0
else
# human intervention required at this point
log_message "Projector at ${PROJ}: TPC reboot did not fix the problem" "err"
exit 1
fi
fi
fi
else
# Projector is offline, nothing to do if we made it this far
exit 1
fi


You can run this script by calling it like /bin/bash script.sh [projector IP]. I have ommitted the account and password info for the telnet user, but you can pm me for those if you don't know what they are. In order to target all of our projectors I have a helper script that is called by cron.d and loops through the IP scheme of our projection equipment. For the log_message() function, you can pipe the output of the script into a file if you'd like, which I do in crontab, otherwise it will just run and echo to whatever tty it was called from.

The methodology is this:

1) Ping the projector and see if it's reachable on the network (we power off our stuff at night)
2) if step 1 was successful, then send a message on port 5000. (If response was null we have a potential problem)
2.5) if there was a problem, wait for a time, just in case we caught the TPC booting
3) repeat step 2. If issue still exists, then go ahead and reboot
4) confirm if step 3 complete successfully, then repeat step 2. If problem is fixed, log it and exit, otherwise log the error and still exit.

If the TPC doesn't come back up after 1 reboot we don't spam it with repeated reboots, in case there is a different problem. However, the entire script will execute again after 90 minutes (from cron).

Theoretically, as long as you have a server with bash you can run this. The DSS200's run Debian so they're a perfect target, although I haven't tested running this on a production server.

The TPC should come right back on after a reboot even unattended

[Oleg Batiaiev]

Caleb, Thank you for the useful information about soft rebooting the TPC and the script provided, because we have many screens in remote control so far without remote power control and therefore had to send local managers to reboot the projectors. Now this will simplify and reduce the time to resolve this issue.

[Ioannis Syrogiannis]

Nice input, Caleb.
I wonder if it would cover more cases when you were to add that "force" operator:
shutdown /r /f /t 0
(I was under the impression that it was "-" instead of "/", but I haven't used that command since... Windows XP?)

Ryan, if there isn't any linux machine at the booth, an old generation* Raspberry π might have been a nice idea.
Those "Windows Linux Subsystem" implementations on recent Windows OSes are nice. You can ssh in a server without having to use PuTTY, for instance. But I don't think they fit well the idea of cron scheduled tasks.
(*No substantial benefit from a younger generation, unless you are looking for high speed Ethernet.)