Thoughts on this AP20 Automation Glitch?

[Ryan Gallagher]

I've mentioned this before, but as Datasat was unable to reproduce it with one of their units, perhaps someone here has further thoughts.

I hit our dreaded AP20 glitch again for one of our summer film bumper/trailers. Our unit, after a certain amount of uptime (seems to be about 1-2 months ish), will all the sudden be accepting fader automations, updating the fader level on it's LCD/VNC screen, but the mains output will not reflect the changes. Fader automations are via TCP/IP from a Doremi DCP2K4.

I had an automation "fade" take house music out, when it restored to 6.1 for the bumper, about half way through bumper I got a radio call that there was no audio in the house, it was still outputting 0.0 despite having 6.1 on the screen. Obviously the booth monitor doesn't follow the fader anyway so I had no way of knowing, everything appears/sounds normal from inside booth. The in the moment fix is just to bump the physical knob on the unit, that seems to always correct the output, but all automations are untrustworthy until next reboot and you have to do the same for every automated level change.

Every day I run a test SPL that has a bunch of fader automation and check it in the house, trying to catch this issue and reboot the AP20 if needed. But it tested good that day and failed within the show.

Datasat has been unable to reproduce this issue on their end. If we were a 1st run theatre with TMS and fully automated it would be a deal breaker. Since we used to run everything without automation, management's view is we can work around it. But it sure sucks when it catches you out.

This issue was first discovered about 2 years ago after we added fader automations and tried to use one during a crawl to fade out audio during a world premiere to start the Q&A. I even had studio techs in the booth with me. Was getting frantic coms that the music was still playing, we were all staring at the 0.0 on screen and scratching our heads. Had to just slap the mute button. The techs had never seen anything like that before. I suspect it was occurring at the start of the film too, because we got a flurry of unexpected volume requests from the director, it probably started with whatever volume the previous CPL was programmed for and didn't take the automation change in the house to the desired film level. Since then we've mostly been able to manage it by monitoring the AP20 behavior ahead of shows and rebooting if symptoms are starting to appear.

Needless to say, if I use fader automation on high profile things now, I always "nudge" the fader level right after each one to make extra sure it's in the house too. And don't trust my automated fades for premieres/festivals either. They will sound awful if it's starting to glitch out, or do nothing if it's fully glitched out.

We have reloaded our configs etc. But building configs from scratch is something we have not resorted to yet.

[Carsten Kurz]

'1-2 months uptime' - you don't powercycle it for as long as 1-2 months?

You can

[Ryan Gallagher]

'1-2 months uptime' - you don't powercycle it for as long as 1-2 months?

You can

Past practice was that it just stayed on. I was running an experiment to see how long before the glitch reappears trying to determine if it is uptime related or something else. If it is purely uptime then yes a policy to reboot it more frequently would be a solution. But so far the test is not conclusive that it is just uptime that causes it. (But knowing that a reboot fixes it for a time, certainly more frequent reboots would be viable, or as a policy before every show?)

[Carsten Kurz]

I would start a logbook on this. Assuming that it runs on current software, as you had contact with Datasat/ATI about it?

Our AP20 is in it's 12th year now and never showed that behaviour so far. It get's powered down daily (power cut). There is of course the soft power down executed on the front button. You should probably give both methods a try. I mean, once you know it can happen, the time to fix it is probably short. You could also simply try to add a quick Mute/Unmute before every show in your start playlist. Assuming that network command mute/unmute would cure the problem as well (but maybe also that is ignored). I guess when it happened in the past, you were not in the mood to test other options. Would be interesting to find out wether it's more network or more software related.
On a side note, we have configured both some GPO AND network automation from our servers towards the AP20. The GPO covers the bare minimum.

[Ioannis Syrogiannis]

If reboot is a solution, then I would guess that the whole thing has to do with memory, buffering and the likes.
Indeed, a monitor that can show the amplifiers' output worth its trouble. While solutions like that of Dolby, with the monitor on the processor is useful. Especially when one can listen to the input with the volume to the auditorium being 0, catching up troubles needs to go beyond that. The issue here, is that the person that checks the audio is more often in the auditorium than the booth (if any).

The first AP20 I connected to a Doremi ShowVault was on a one-screen setup with no control network switch. So, I went the other way and configured the serial connection. I could, then, use the automation buttons of the processor to send commands to the server. I was impressed and I definitely didn't see any practical reason why the AP20 was not as popular as the competition at the time.

In any case, I would guess that the problem is the classic IT one. (Does anyone remember the taped message on the "IT crowd" series? - watch?v=nn2FB1P_Mn8 on that... site)

[Harold Hallikainen]

I also think a regular reboot would solve the issue. I SUSPECT it is either a memory leak or a timer overflow. Library code I got for the microcontroller in several of our products used a 32 bit "tick timer." In various state machines, it would save the value of the tick timer plus some value, then when we got back to that statemachine, the current ticker was compared to the saved value (typically a timeout before moving on to the next state). But, if the system was up long enough, the 32 bit timer would overflow and it could be months before the saved value was ever reached.

Good luck!

Harold

[Ryan Gallagher]

Yeah datasat had us pull configs and send them over. Nothing spotted. They offered to arrange a new unit but we did not cross that bridge yet (or learn costs), I’ll reach out and find out.

I’m not sure any automation command is capable of resetting the issue, as it seems to be exclusive to the automation interface, but I do know it would be needed for every fader move even if mute/unmute did have the desired effect. Not ideal. Next time I have it doing it outside of a run I’ll try sending a mute/unmute from the doremi simply for diagnosis reasons, maybe the issue is limited to fader macros, or the way I wrote them.

[Ioannis Syrogiannis]

[...]from the doremi[...]

I am sure that someone would have figured checking that, up to now, but no harm by mentioning the idea here:
Doremis have one nice feature on their automation cues. One can stack different commands, and delay between them.
AP20s have a nice feature as well. Once can configure the tenths of seconds the volume fades out and back in on a format change.
Is there any chance that more than one automation cues from the server are sent at once?
And, besides that, is there any chance that the fade (in/out) times get in conflict with any automation cues?

(The odds are still in favor of flushing all temporary info by power circling.)

[Ryan Gallagher]

I am sure that someone would have figured checking that, up to now, but no harm by mentioning the idea here:
Doremis have one nice feature on their automation cues. One can stack different commands, and delay between them.
AP20s have a nice feature as well. Once can configure the tenths of seconds the volume fades out and back in on a format change.
Is there any chance that more than one automation cues from the server are sent at once?
And, besides that, is there any chance that the fade (in/out) times get in conflict with any automation cues?

(The odds are still in favor of flushing all temporary info by power circling.)

I've had all those thoughts and when my "scripted" fade out is executing (I handled this as an AP20 macro called by a single doremi command), it is quite possible that while it is running you try sending another command from Doremi that potentially conflicts. While that "possible" conflict complicates things, it definitely does it on it's own with single fader commands too. It will be fine one day and you won't have done anything crazy with it like scripted fades. You come in the next day, and our very simplistic test SPL fails.

The test just loads up center channel pink, and every two seconds sets the fader level a full integer lower. You can step into the house with the tablet and pull up AP20 VNC and watch that what you are hearing matches the triggered changes on screen.

When it is failing it will maybe do every third one in the house, or none at all, despite the screen following correctly.

It's second test is one of the scripted fades, and if you hear too much choppiness in the fade it has become "suspect" as well.

Hacking the format change features to get a timed fade out is another alternative to the scripted approach. But as the bug doesn't seem to be exclusive to the scripted cues, doesn't really solve anything, it needs to honor the regular fader cues reliably too.

[Marco Giustini]

Any Cinema support team would recommend to power cycle equipment every couple of weeks. I appreciate it's a machine and it shouldn't do that but I think the best solution would be to just power cycle the AP20 when the server/projector gets rebooted as well - because they get rebooted right? ?

What happens if you send another fader command when it's stuck to 0? Would it keep changing but with no effect on the output? Is touching the fader knob the only solution?

[Tony Bandiera Jr]

I've had all those thoughts and when my "scripted" fade out is executing (I handled this as an AP20 macro called by a single doremi command), it is quite possible that while it is running you try sending another command from Doremi that potentially conflicts. While that "possible" conflict complicates things, it definitely does it on it's own with single fader commands too. It will be fine one day and you won't have done anything crazy with it like scripted fades. You come in the next day, and our very simplistic test SPL fails.

The test just loads up center channel pink, and every two seconds sets the fader level a full integer lower. You can step into the house with the tablet and pull up AP20 VNC and watch that what you are hearing matches the triggered changes on screen.

When it is failing it will maybe do every third one in the house, or none at all, despite the screen following correctly.

It's second test is one of the scripted fades, and if you hear too much choppiness in the fade it has become "suspect" as well.

Hacking the format change features to get a timed fade out is another alternative to the scripted approach. But as the bug doesn't seem to be exclusive to the scripted cues, doesn't really solve anything, it needs to honor the regular fader cues reliably too.

OK, I'm gonna say it, and Ryan will hate me for it but....

Dude, you are dicking around far too much with a processor that is usually stone reliable, and by doing so you are causing the microcontroller to overload with silly and useless processes. In my 40+ years in the trade (plus 2 years of live sound) I have NEVER gone as crazy with "testing" as you have here. Nor have I ever needed to.

You are wasting time on that bolded test. (Which is the total opposite of "simplistic".) What exactly does it really accomplish? NOTHING. Again, I will be brutally direct..it is just a way to stroke your own ego. For real world testing that is simple and will accomplish the EXACT SAME THING, create a simple test macro with a single timed fade up to a preset, pause, timed fade down to another preset, pause, and do a mute/unmute. Over and done in less than 10 seconds.

Second, why are you leaving the processor on all the time? I agree with Carsten, Ioannis and Harold that you should power down daily, as a reboot clears all of the garbage that the controller stores so you get a fresh start.

With all of the digital controls and processors, you really need to KISS them..Keep It Simple Stupid. Stop doing that fader step test, make sure your fader macros are simple, with appropriate delays inserted between steps (Your audience will NEVER notice anything less than 2 seconds delay in execution, which is an eternity in the processor's world, which WILL allow plenty of time for processes to execute and buffers to clear. )

I really appreciate your passion and dedication to showmanship, a rare trait these days. But you are your own worst enemy by overthinking and over complicating things. Keep the passion, but simplify your methods way down and you will avoid a lot of the issues you have brought up. Your management will appreciate the time savings, and your health will benefit from the reduction in stress over things that you are burning unnecessary energy on. I've been there and learned all of this the hard way, no need for you to suffer through it too.

[Ryan Gallagher]

OK, I'm gonna say it, and Ryan will hate me for it but....

Dude, you are dicking around far too much with a processor that is usually stone reliable, and by doing so you are causing the microcontroller to overload with silly and useless processes. In my 40+ years in the trade (plus 2 years of live sound) I have NEVER gone as crazy with "testing" as you have here. Nor have I ever needed to.

You are wasting time on that bolded test. (Which is the total opposite of "simplistic".) What exactly does it really accomplish? NOTHING. Again, I will be brutally direct..it is just a way to stroke your own ego. For real world testing that is simple and will accomplish the EXACT SAME THING, create a simple test macro with a single timed fade up to a preset, pause, timed fade down to another preset, pause, and do a mute/unmute. Over and done in less than 10 seconds.

Second, why are you leaving the processor on all the time? I agree with Carsten, Ioannis and Harold that you should power down daily, as a reboot clears all of the garbage that the controller stores so you get a fresh start.

With all of the digital controls and processors, you really need to KISS them..Keep It Simple Stupid. Stop doing that fader step test, make sure your fader macros are simple, with appropriate delays inserted between steps (Your audience will NEVER notice anything less than 2 seconds delay in execution, which is an eternity in the processor's world, which WILL allow plenty of time for processes to execute and buffers to clear. )

I really appreciate your passion and dedication to showmanship, a rare trait these days. But you are your own worst enemy by overthinking and over complicating things. Keep the passion, but simplify your methods way down and you will avoid a lot of the issues you have brought up. Your management will appreciate the time savings, and your health will benefit from the reduction in stress over things that you are burning unnecessary energy on. I've been there and learned all of this the hard way, no need for you to suffer through it too.

Appreciate the sentiments Tony, there are definitely some simpler approaches than the one I took when created scripted options.

Going forward attempting to log this, perhaps I'll remove the scripted fade from the test CPL, or remove it altogether if I can make mute or a format change do the same thing, because you are right that running that more complex macro daily in test could be aggravating the microcontroller's memory situation and leading to getting the glitch on much simpler commands sooner.

That whole "equipment that is not supposed to be turned on and off constantly" thing is my whole reason for seeking alternative approaches/answers, cause we are totally capable of just turning it off after every show, but perhaps other changes will resolve our reliability problem too.

A bit of background on the rest:

The processor stays on cause this building's policy has been to leave that rack on and fully powered (because it also houses important booth networking). Obviously we can power down the AP20 but to do so "safely" in the traditionally audio sense involves a bit of faffing about with control software to remote mute the amps that are not located in the booth. It is not something all of our operators are expected to know how to do at this time. If someone here knows the AP20 to be perfectly safe on the outputs during soft or hard power down/up I can ignore that step going forward. Based on the pops I hear to speakers receiving the HI signal, at least those outputs are NOT completely safe. It should be noted that when in film setup all the amps are unmuted and stay on overnight, surrounds stay on unmuted 24/7 in the booth. Another change to consider?

Only the Christie and Doremi are put into standby/off... the rest of the DCI system just stays on unless we are striking out of film setup.

I think the work I did to create scripted fade out options is perhaps clouding the subject here. Although your instinct about their existence at all perhaps being part of the problem is a good one. They are not cross fade scripts for use on every CPL, just full fade out to zero, for when you are asked to interrupt audio on a running CPL in a "professional" manner".

This bug at issue occurs with nothing that complex being attempted/used. I just have regular fader level doremi one line macros like every other tech might implement for the AP20, it might only get two such commands through a whole SPL. They are placed before every CPL that needs a fader level change (preroll content into films etc), similar to how trailer volume might be automated. We space them a full second apart when needed, usually in blacks unless otherwise appropriate. And currently, if it's been too long since last AP20 reboot, these simple commands are liable fail in this particularly annoying way discussed. I was just asking because perhaps someone knows another avenue to investigate other than simply accepting that the AP20 needs to be rebooted more often without learning the specific cause. Obviously if rebooting is the ultimate solution totally willing to change policy. But your instincts about simplifying away from those complex fade macros and lean more heavily on the internal format/mute cross fades is probably the way to go.

The more complex scripted fades were a product of getting lots of requests during festivals/premieres and other shows with extra events... often they would want cut credits audio short (without knowing that plan ahead of rolling the SPL) and we have to do a manual knob fade on the other side of the booth away from the clearcom while potentially juggling other buttons too. We did not have mute setup to fade at that time. At least with a fade macro I could execute it "in the moment" from the macro execution utility on top of a running CPL and stay on coms. To be clear I did not create "fade" scripts between CPLs for their volume adjustments like you might be imagining, the 3 scripts are just for those cases where I need to roll a running CPL all the way down to zero and mute, and I gave myself 3 choices of how long it takes to do so. Implemented before I knew enough to see if the mute feature or a dummy format could behave the way I wanted with a roll off. I think it can and that should really be my solution here, but adopting simpler processor handled fades between formats or mutes doesn't necessarily make the bug go away, and I still have to test for it now, mostly for confidence reasons, unless we switch to rebooting daily (or turning it off between shows).

As for the test itself. We have to have content with audio rolling and be in the house to even notice the glitch has taken hold (booth monitors that follow output would be one improvement there), everything from the AP20's perspective is situation normal except the outputs themselves. No amount of testing format changes or firing mute macros and watching the screen will make the glitch detectable, you have to hear it in the house. What you are trying to avoid is having a pre-roll CPL at 5.1 and feature at 7.2 and have that simple macro fader change not take effect at the outputs... you might be half way into the film before anyone says anything to house management. Or even worse... preroll at 7.0, and feature at 5.6... and it just stays at 7.0 equivalent on outputs and blasts the crap out of everyone. I make a habit of stepping into the house to confirm or adjust, but not all our operators do.

Thanks for the useful criticism and encouraging refocusing on simpler ways to do the same thing.

I think I shied away from adding format fades at the configuration level because I was unclear how they behaved. I thought it would add a delay before it actually takes the format switch, but I think it is literally just a fader level cross fade, it takes the format immediately but then rolls between the different fader levels (if configured). I need to refresh my memory of those configuration options when I have time, if mute can have a fade out/up time on it that really helps simplify things. If it can't, maybe a dummy format with 0.0 configured can be made to do something similar.

[Ryan Gallagher]

Any Cinema support team would recommend to power cycle equipment every couple of weeks. I appreciate it's a machine and it shouldn't do that but I think the best solution would be to just power cycle the AP20 when the server/projector gets rebooted as well - because they get rebooted right? ?

What happens if you send another fader command when it's stuck to 0? Would it keep changing but with no effect on the output? Is touching the fader knob the only solution?

Appreciated. Christie and cinema server do get powered down or put into standby, but the rest was just staying on in practice. Before I started even the CP650 had instructions to be left on despite it running the SRD diodes 24/7. That is one people do tend to suggest leaving on but we need to wire in switches for our diodes first.

Once the glitch is occuring, it will honor all knob moves correctly, as well as VNC interface commands, but all automation moves on the output are DOA until reboot, despite screen thinking it took all of them and everything is fine.

[Ryan Gallagher]

Really what we need is an old school fader remote that can be placed near the operator location. Maybe there is a way to achieve that with the AP20, but the VNC interface is NOT it. ;-)

That or wireless coms, or replace our clearcom "biscuit" closest to the audio rack. It's quite vintage and the volume knob is missing. We've been overdue for updated coms for eons, but doubt wireless will be in the plans.

[Steve Guttag]

While I find Ryan to be overly eager on things, I need to remember I was once too (and still am, sometimes), I can certainly appreciate not accepting that things need to be power cycled.

I have clients that absolutely will not power cycle equipment, particularly sound equipment and feel that not thermal cycling or having capacitors and such go through cycles is absolutely better for the equipment. To their credit, I have equipment that has been powered for the better part of 3-deconds. So, how can I fault their logic?

Mind you, I have power cycled systems (95% of mine) that are power cycled and they are not having problems...except servers, particularly the IMS3000 and even the CP950 can wake up in a bad state. It isn't like it is regular...they might have a bad boot up a year or twice a year. I am absolutely ticked off that the world is in any way shape or form accepting this notion of "rebooting" as a "fix" for anything other than sloppy coding or poor thermal management.

Oddly, with Q-SYS, since it is often part of the control system for the theatres I put it in, it does not power cycle but it does put the amplifiers into "stand-by" But the Cores run 24/7 and are UPS backed up. About the only time they get any sort of cycling is when there is an update.

I have some DCP300s that can run 24/7 without complaints and some where they boot up in a bad state (much more rare).

The Eprad eCNA automations...all of them are 24/7 for me...by far, the most reliable thing in any theatre and they are often involved in waking a theatre up or putting it to bed.

I don't have any advice on the AP20 since I don't have any out there. I suspect though, since they don't know what the issue is...your choices are to power cycle daily/weekly or get an AP25 if you want to continue with the series. Or, possibly, source an AP20 used.