Christie Series 2 TPC issues

[Ryan Gallagher]

Imagine getting this to work on something like the RasPad. Its such a cheap part compared to the oem TPC which may not even be available much longer. This would be a really interesting project on it's own and I'm willing to take a deep-dive on it.

Is there anyone not running the latest 4.8.1(3) software for some reason?

Just more fodder... but it appears booting a CE image on a regular PC is possible from a HDD. Booting a pre-existing CE image might be quite a bit harder to get needed driver support. CE apparently does not even have a built in "safe mode", and driver installers don't behave the same... drivers are all pre-loaded and built using image creation tools, are built using .cab files it seems.

But here is a tutorial on getting a vanilla CE image running on bare metal PC:
https://www.e-consystems.com/article...gWindowsCE.asp

I do agree that although more technical, a VM would be more readily available to everyone. If you were successful against a particular PC or laptop booting on hardware, now THAT becomes the dependent hardware... perhaps just as rare as the TPC (although maybe less expensive).

Several legacy microsoft emulators supported CE and mobile, depending on which version you are targeting, for example "Microsoft Virtual PC 2007":
https://archive.org/details/virtual-pc

VirtualBox claims to support CE images, and virtualbox runs on linux, so maybe your RasPad dream is not impossible?

After a little reading, perhaps QEMU is more needed than virtualbox. As there was no pi arm version of CE. Is the TPC version we are discussing ARM or x86?

[Marcel Birgelen]

Gen1 TPCs run on Windows CE, which is not really a traditional Windows, as it uses a different kernel architecture. Gen 2 TPCs run on Windows XP Embedded, which is essentially a dressed down version of Windows XP.

Correction: Gen1 is an ARM Based system and Gen2 is an Intel x86-based AMD Geode system.

I think it will be easier to customize an XPe image than a CE image.

[Ryan Gallagher]

I think it will be easier to customize an XPe image than a CE image.

That does sound like the path of least resistance. I was not aware there was a generation that moved to XPe.

[Caleb Williams]

I think I could build a basic linux system and then virtualize either generation on top of it using qemu. I need to figure out how to make sure the inputs for keyboard, mouse, and maybe touch get passed through directly to the virtual machine, but its very doable using a NUC. I'll whip something up and do some testing with performance. Also the networking will need to be addressed in the base system so that a static address on the vm is passed directly to the projector. I'm thinking of something like a Beelink nuc as the host

https://www.newegg.com/p/2SW-0012-00...0aAvcTEALw_wcB

Its very overkill with the memory and disk space (TPC sdd is 4-8 GB), but that allows for some flexibility in the host system. I already have a few of these on my workbench we've been developing in-house signage with, so I could use one of those to start testing very soon. The bottleneck will definitely be the cpu, but running linux should make the usage of the host system way smaller.

If we really wanted to, it would be easy to get that NUC to run on the 24v power provided by the projector, we could find a touch panel online, then design a case not too dissimilar from the TPC to 3d print. In that way, it could be a bolt-on solution when the TPC is no longer available. Is it worth all of the work? I don't know.

[Ryan Gallagher]

Is there anyone not running the latest 4.8.1(3) software for some reason?

Not for a valid reason, other than the venue’s “if it ain’t broke don’t change it” default position.

But checked and we are on 4.7.2(4) still. Should I be pushing them harder to update?

[Caleb Williams]

Not for a valid reason, other than the venue’s “if it ain’t broke don’t change it” default position.

But checked and we are on 4.7.2(4) still. Should I be pushing them harder to update?

I was just thinking, if we go to the effort of building basically a replacement TPC, it would be helpful if we didn't have to worry about different software versions. *Technically* a virtualized tpc should not cause any problems upgrading or downgrading projector components, but that is only proof of concept. There may turn out to be issues that can only be resolved by virtualizing exactly the version the tpc was on before it was replaced. I never got release notes for 4.8. I'll see i can find out what they've changed.

[Ryan Gallagher]

I was just thinking, if we go to the effort of building basically a replacement TPC, it would be helpful if we didn't have to worry about different software versions. *Technically* a virtualized tpc should not cause any problems upgrading or downgrading projector components, but that is only proof of concept. There may turn out to be issues that can only be resolved by virtualizing exactly the version the tpc was on before it was replaced. I never got release notes for 4.8. I'll see i can find out what they've changed.

I think by the time you are having to replace a TPC with 3rd party home spun solutions, the projector basically becomes a time capsule with no further real prospect of updates (unless you get lucky). Part of trying to keep legacy gear alive. I would think just one version might be adequate. The perk is you could always backup your image and "try" updates, without much risk involved. But this is all well outside the scope of normal support/service/tech work. (All of it). haha.

[Ryan Gallagher]

Reading it was possible to license or lock XPe to specific hardware, that might be a gotchya if employed.

[Caleb Williams]

Huh.

Okay so for the TPC650H I was able to virtualize it really easily using the following steps on a Debian 12 install with only x11 and openbox installed:

1) use a flash card reader to take a dd image of the tpc ssd
2) install qemu-system on the target host for virtualization
3) convert the disk image from step one from .dd to .qcow2, a dynamic disk format familiar to qemu (similar to .vhd on windows)
4) create the command for launching the virtual tpc:
"sudo qemu-system-i386 -hda tpc_t650H_ssd.qcow2 -m 2048 -display sdl,gl=off,show-cursor=on -vga std -boot c -cpu host -enable-kvm -net nic,model=rtl8139 -net user"

i386 is 32 bit emulation
-m is ram
-display shows the display output of the vm on the desktop
-boot makes sure it uses the c:\ partition (there are 2 partitions on the tpc ssd)
-vga uses standard vga virtual device
-enable-kvm allows qemu to use cpu virtualisation features
-cpu says to just pass through the host cpu info to the vm (I thought this was going to be problematic but it wasn't at all)
-net sets the virtual network device to be realtek rtl8139, which is the same type+model of network card on the real tpc.
-net user allows the tpc to connect to the outside world

Turns out that even though the virtual network card is the same as the nic on the real tpc, the drivers did not install automatically. I had to create an iso with a driver package that was added with the -cdrom parameter for XPe to access. After I installed the drivers, the tpc could communicate with an external network.

The next step is to make the host network interface transparent to the vm by bridging the two. My strategy will be to change /etc/network/interfaces to have the following:

auto lo
iface lo inet loopback

auto enp1s0
iface enp1s0 inet manual

auto br0
iface br0 inet dhcp
bridge_ports enp1s0
bridge_stp off
bridge_fd 0
bridge_maxwait 0

Run this command to allow qemu to access interface br0: echo "allow br0" | sudo tee /etc/qemu/bridge.conf
Then add these parameters to the qemu command to start the vm: -netdev bridge,br=br0,id=net0

If the bridging works, this is actually SUPER duable with a nuc with a 12th gen dual-core Celeron cpu or newer. I was watching htop while the vm was loading and holy crap this dual-core Celeron is handling it without even sweating. Like, you could probably mine some crypto at the same time and the vm would be almost completely unaffected/s. Things also load WAY faster in the vm than they do on the real tpc (makes sense), but until I can get the network bridge working I won't be able to tell if that will be a problem. The timing of the PIB and TPC booting shouldn't be super critical, but if the TPC is constanly booting way faster than the PIB, the vm could reach the timeout and show the error "PIB connection lost. Please power cycle projector", even though the PIB just takes a little longer to boot.

Also,

Diagnosing those boards without schematics is like trying to find a needle in a haystack, especially if nothing is obviously broken.

I've had a bad TPC in the part that was given up by the owner and replaced by a new one. I was able to revive it, by throwing the mainboard in a reflow oven.

I did some more testing with the real tpc main-board alternating between hot and cold on the board using a can of air turned upside down. I was careful to not power the board while there was visible condensation on it, but I definitely found an increase of the no POST problem when the board was cold/super cold. (I even put the thing in the freezer for 20 minutes and it would not turn on after several power-cycles until it was closer to room temperature again). I have a theory, and that is that from the ITEC chip either 1) it's not receiving the pulse on the PWRSW pin telling it to turn things on, or 2) the PWRFAIL state of the ITEC controller is not being cleared. The ITEC chip on this TPC looks like it got a bit toasty, and is one of the few larg chips that doesn't have thermal pads and some sort of sink-contact. There is also the fact that they use the really crappy lead-free solder that is known for becoming brittle and breaking connections (esp with smd chips). A reflow oven might fix a bad connection, but I don't have access to one. The cold making things not work sort of points to a solder issue. We are replacing a TPC at one of our other locations because of this issue, so I will soon have another to try to replicate some of these tests. I'm assuming it's a similar cause in most cases due to it's frequency.

[Marcel Birgelen]

@Ryan Gallagher, you could do that, but let's just hope they did not do that.

I've actually witnessed how a colleague of mine fixed his notebook by putting the mainboard into a regular oven, like the one in your kitchen... I'm not implying you should try this, as I'm afraid it's easy to 'cook' the mainboard to a state beyond repair, especially if there are already components on there that might have had to deal with their fair amount of heat. Maybe, if you can find a heatsink that fits on the ITEC IC, that might be worth a try.

As for the bridge, if I remember correctly, if you want to bridge the network in QEMU, you need to use tap devices. See a trial from my side below:

In order not to loose connectivity to the NUC, install a secondary ethernet adapter or enable wireless. Make sure the ethernet adapter on the NUC to be bridged is clean from other traffic.

I guess you have the bridge-utils installed already, but also make sure you have iproute2 installed. Assuming Debian 12 like you mentioned and sudo being installed and configured.

sudo apt install iproute2

Create the bridge;

sudo ip link add name br0 type bridge

Add your NUCs physical ethernet adapter to the bridge. This will kill connectivity if this adapter is still active. Also, make sure to set the device to UP, as this will not automatically happen if the device is unconfigured:

sudo ip link set dev enp1s0​ master br0
sudo ip link set dev enp1s0 up​

Disable STP to avoid any STP-related disasters down the liw

sudo ip link set dev br0 type bridge stp_state 0

Create the tap0 virtual ethernet device:

sudo ip tuntap add name tap0 mode tap

Add the tap0 device to the bridge and also make sure to set the tap0 device to up;
[CODE]sudo ip link set dev tap0 master br0
sudo ip link set dev tap0 up[/quote]

Fire up QEMU, based on your previous parameters:

qemu-system-i386 \
-hda tpc_t650H_ssd.qcow2 \
-m 2048 \
-display sdl,gl=off,show-cursor=on \
-vga std \
-boot c \
-cpu host \
-enable-kvm \
-netdev tap,id=tap0,ifname=tap0,script=no \
-device e1000,netdev=tap0​

[Ryan Gallagher]

This all seems extremely promising on the alternative control device front. Great work so far Caleb! Networking stuff is extremely fungible and should be solvable. And agree that thermal gradient in behavior of the TPC board does seem to indicate a cold solder joint somewhere... contracting and failing when cold from a shutdown, but functioning when warmed up after several boot attempts?

Our pedestal stays powered overnight and projector in standby, the only time we are asking it to boot is during a "reboot"... so it's already warm. Perhaps units that experience heavier thermal cycling are prone to this failure mode (or in cold storage for too long).

If you could identify the problem chip/board area a reflow oven is not really needed... just a hot air rework station for more surgical reflowing, but to replace with higher quality solder might require reballing depending on the issue package.

[Ryan Gallagher]

If a virtual TPC turns out to be stupid easy, it might actually be more reliable in several ways than an original. Back to that thread about automation sluggishness or non-responsiveness and having to trigger reboots to clear whatever issue is causing that.

[Marcel Birgelen]

Adding to my limited bridging fu... You may also need to give a "link up" to the br0 bridge... Linux has the irritating tendency of not setting ethernet or virtual ethernet thingies to "link up" if you don't associate an IP address... Without this "link up" the bridge may decide not to forward any traffic...

[Ryan Gallagher]

On that virtual TPC boot time versus PIB power up timing worry. Even if there was a timing mismatch, assuming the projector was powering the virtual host in a final "polished" implementation, you could just program in a wait time before the host fires up the VM. I would assume the PIB etc is fully awake before windows XPe finishes booting on the original slow TPC. So all it's checks are to a fully powered projector.

[Marcel Birgelen]

The NUC first needs to boot Linux and then it could auto-start the VM. Since it will now be booting on much faster hardware and from a proper SSD, it will probably still be started far earlier. If the TPC software cannot handle being "ready" before the projector is in a certain state, then you could, indeed implement an additional wait before you boot up the VM.

It would be pretty shoddy software engineering if it fails on this, but it would certainly not be the first race condition triggered by hardware speed increase.