Thoughts on this AP20 Automation Glitch?

[Ryan Gallagher]

While I find Ryan to be overly eager on things, I need to remember I was once too (and still am, sometimes), I can certainly appreciate not accepting that things need to be power cycled.

I have clients that absolutely will not power cycle equipment, particularly sound equipment and feel that not thermal cycling or having capacitors and such go through cycles is absolutely better for the equipment. To their credit, I have equipment that has been powered for the better part of 3-deconds. So, how can I fault their logic?

Mind you, I have power cycled systems (95% of mine) that are power cycled and they are not having problems...except servers, particularly the IMS3000 and even the CP950 can wake up in a bad state. It isn't like it is regular...they might have a bad boot up a year or twice a year. I am absolutely ticked off that the world is in any way shape or form accepting this notion of "rebooting" as a "fix" for anything other than sloppy coding or poor thermal management.

Oddly, with Q-SYS, since it is often part of the control system for the theatres I put it in, it does not power cycle but it does put the amplifiers into "stand-by" But the Cores run 24/7 and are UPS backed up. About the only time they get any sort of cycling is when there is an update.

I have some DCP300s that can run 24/7 without complaints and some where they boot up in a bad state (much more rare).

The Eprad eCNA automations...all of them are 24/7 for me...by far, the most reliable thing in any theatre and they are often involved in waking a theatre up or putting it to bed.

I don't have any advice on the AP20 since I don't have any out there. I suspect though, since they don't know what the issue is...your choices are to power cycle daily/weekly or get an AP25 if you want to continue with the series. Or, possibly, source an AP20 used.

Haha. Definitely. It's a personality trait that works as a strength in some situations and a fault in others. Quite aware! Both a superpower and a kryptonite in potentially the same moment.

[Carsten Kurz]

The AP25 is so close to the AP20 that I wouldn't count on that bug to be gone with it.

Ryan did not say wether he performed a soft reboot (front button) or hard power cycle on the AP20. The soft reboot goes much faster. The reboot after a hard powercycle takes at least a minute.

If he uses automated volume ramps, I would look into their timing and try to change it. The AP20 shows all automation signals in it's logs, maybe that could help. The trouble, as often is, it happens so rarely that it's hard to catch it. If the automation signals show up on the display properly, it is obviously not a network/receiving issue, but something on a lower level where the level setting circuitry is addressed. Hard to understand why dialling the manual fader then solves it, but...

Honestly, I would always try to keep automation command density low. We know these devices with their simple port communication all have weaknesses.
So why not use the built-in fade in/out feature? That's why it's there.

[Ryan Gallagher]

The AP25 is so close to the AP20 that I wouldn't count on that bug to be gone with it.

Ryan did not say wether he performed a soft reboot (front button) or hard power cycle on the AP20. The soft reboot goes much faster. The Reboot after a hard powercycle takes at least a minute.

If he uses automated volume ramps, I would look into their timing and try to change it. The AP20 shows all automation signals in it's logs, maybe that could help. The trouble, as often is, it happens so rarely that it's hard to catch it. If the automation signals show up on the display properly, it is obviously not a network/receiving issue, but something on a lower level where the level setting circuitry is addressed. Hard to understand why dialling the manual fader then solves it, but...

Honestly, I would always try to keep automation command density low. We know these devices with their simple port communication all have weaknesses.
So why not use the built-in fade in/out feature? That's why it's there.

All our reboots have been with the soft (quicker) cycling. That unit remains powered. The quick version solves it.

I was really just trying to implement an elegant solution for that less common use case of fading out credits before a Q&A due to awkward equipment placement in booth. If a format switch or mute with built-in fade can get me there in that use case too I'm all for it. (Didn't have time to investigate yet, was busy with a 35mm show).

Edit: for those curious, my 3 "fade out" AP20 macros were implemented a block of 100 fader commands (moving .1 each) with a delay between each. That delay is what creates the overall fade time required to get to 0.0. 100 because I wanted them to be universal regardless of what fader level you were sitting on.

1. 50ms delay (approx 3.5 sec fade from 7.0)
2. 100ms delay (approx 7 sec fade from 7.0)
3. 200ms delay (approx 14 sec fade from 7.0)

The 100ms (7sec) one is the most useful in the credits to Q&A context. But I could see a 3 or 4 second internal AP20 fade also covering all situations, but still being fast enough not to be annoying when it is being automatically applied in other situations.

[Marco Giustini]

I have the same Kryptonite trait so I fully understand where you're coming from, Ryan - let's say that with age I've learnt that sometimes fighting with windmills is just a waste of time and it's better to focus my energies on wars I could win ?
But I do appreciate this type of approach to things.

[Carsten Kurz]

That is really A LOT of fader commands. I wouldn't be surprised if that actually triggers the problem. Also I guess that your testing scheme might actually provoke the problem more than it protects against it.

AP20/25 max format fade in/fade out time is 2s. I'd check if you can live with it.

[Ryan Gallagher]

That is really A LOT of fader commands. I wouldn't be surprised if that actually triggers the problem. Also I guess that your testing scheme might actually provoke the problem more than it protects against it.

AP20/25 max format fade in/fade out time is 2s. I'd check if you can live with it.

Agreed, that approach might just be more than it’s automation and cleanup routines can handle.

For the 2sec format option. What is the behavior? Does it accept your format change request, and then take 2sec to execute any crossfade, and only then change formats? Or does it do the reverse, take the new format first and then apply a fader transition to the programmed format level? Or does it do both and fade down to and up from 0.0 in 2sec on both sides of the format change? (Not at work to test).

Only the 1st or full fade out/in behavior could be useful to us. Unless mute can also be given a 2sec transition. If not it seems like adding a remote fader or improving our booth com locations is the way to go eventually.

[Marco Giustini]

The thing is that I doubt that Datasat ever tested their software with 100 fader commands spaced 100ms!
With D-Cinema equipment I've always played safe and left at least 1s between cues!

I haven't played with an AP for a while but I remember tinkering with the configuration files on my laptop - and then uploading it back - because the UI would not allow me to do some things (there were some length limitations which the file did not have).

Maybe you can change the fade in/out to 5s via configuration file and the software would accept it? Just mentioning it!

(PS: I remember mentioning Datasat about those length limitations and they told me mine was not a standard use of the unit and that they would not consider changing the current behaviour! ?. )

[Carsten Kurz]

The Format-Fade is not a crossfade, it's a fade-out to zero, and fade-in from zero. Both fade-out and fade-in time can be set individually per format. My AP20 manual quotes 2000ms for each value (makes 4s altogether). The AP25 manual quotes up to 5000ms each. Should be enough.

[Ryan Gallagher]

The Format-Fade is not a crossfade, it's a fade-out to zero, and fade-in from zero. Both fade-out and fade-in time can be set individually per format. My AP20 manual quotes 2000ms for each value (makes 4s altogether). The AP25 manual quotes up to 5000ms each. Should be enough.

Thanks Carsten. We’ll pick or make a format to host the 2sec fade out and use that as the simpler interrupt with fade approach. Then I’ll keep monitoring without hammering it in test. We don’t use format levels as is, it just inherits last level, so not a big change. Maybe our glitch will vanish.

If raw configs support editing that to longer than 2sec it is tempting, but will test without any funny business for a while.

[Ioannis Syrogiannis]

[...]
Edit: for those curious, my 3 "fade out" AP20 macros were implemented a block of 100 fader commands (moving .1 each) with a delay between each. That delay is what creates the overall fade time required to get to 0.0. 100 because I wanted them to be universal regardless of what fader level you were sitting on.

1. 50ms delay (approx 3.5 sec fade from 7.0)
2. 100ms delay (approx 7 sec fade from 7.0)
3. 200ms delay (approx 14 sec fade from 7.0)
[...]

If I was you, I would get rid of two thirds or three fourths of the cues, keeping the timing. Meaning, multiplying the delay between them. And that is because the 0.1 steps are inaudible. Indistinguishable, to be more precise. Therefore, they serve no purpose. I might even have used a final step (from x to 0.0, or back from 0.0 to x) that would be even higher, without keeping the timing as well.
Since the goal here is to have a smooth fade-out and fade-in transition for the audience, quantize accordingly.

All our reboots have been with the soft (quicker) cycling. That unit remains powered. The quick version solves it.
[...]

That would be less likely to introduce a thermal cycling* problem.

*Look! I wrote it correctly now!

[Ryan Gallagher]

If I was you, I would get rid of two thirds or three fourths of the cues, keeping the timing. Meaning, multiplying the delay between them. And that is because the 0.1 steps are inaudible. Indistinguishable, to be more precise. Therefore, they serve no purpose. I might even have used a final step (from x to 0.0, or back from 0.0 to x) that would be even higher, without keeping the timing as well.
Since the goal here is to have a smooth fade-out and fade-in transition for the audience, quantize accordingly.

Yeah greater than .1 in each step is an approach. It’s hard to implement a jump from say 2.0 to 0.0 specifically though, unless you got more involved and read the level first and used an appropriately sized loop. My prior method was not that “clever” as I was authoring it via the touch panel. I had to avoid hard coded fader levels because the start point is unknown, instead I relied on the relative fader commands, and just included enough of them to cover all situations.

But one could assume that by the time you have dropped it by 6.0 in steps that the rest is not needed, we rarely would have the fader above 8.0. That would save 40% of the commands, and the larger step size would cut that remaining number in half or greater.

I originally authored these as AP20 macros instead of attempting it on Doremi because, perhaps falsely, I thought such a command heavy script would be more reliable running on the AP20 hardware instead of over the network. It also made them server agnostic which is a plus.

I’m more inclined to try the internal methods for now rather than continue this fight. But good thoughts. I don’t think it could be thermal related, cause the abusive macro will often work as expected, subsequent simple fader commands too hours later, but then at some point down the road it gives up and the glitch appears.

[Carsten Kurz]

I can confirm that even the older AP20 supports up to 5s of separate fade-in and fade-out time per preset, accessible through the GUI, when running on recent software.

[Ryan Gallagher]

I can confirm that even the older AP20 supports up to 5s of separate fade-in and fade-out time per preset, accessible through the GUI, when running on recent software.

Perfect! That should do the trick.

[Ryan Gallagher]

Had time for a quick config change today. I implemented the format specific 5s fade-out out on my digital walkin music format as a test (that is used for nothing else). Works great. That saves one location where I am often tempted to fire the big fade out macro in a SPL. That said...

(Tony don't read this. LOL).

The AP20 mute however does not seem to be configurable with a fade (unless it's doable from the config files offline), will have to investigate, cause that would have been the silver bullet for the Q&A use case.

I wish the fade-out could be associated with the format you are switching to, not the format you are in when the change occurs. That would have permitted any "other" format to act as the fade out command. Alas not to be.

I can put similar fades on my film formats, but it's not quite a perfect solution. Most of the time minimal or no fade is preferred cause our show flow often has one of those formats in use for bumpers followed immediately by a live speaker for an intro. A 2s fade would probably execute before they get out there and starts talking, but anything more is maybe a bit too long, where as the longer variant is what I'd prefer for a credits roll-off option. One could make a pre-show format that is basically a copy of DCI 5.1 that lacks fades, but that feels in danger of adding more complexity, not less?

If long-ish fades exist on the DCI format options, I have to be a little more strategic about where the format switches occur if you otherwise don't want to hear them on DCI content. I suppose I could leave it alone 90% of the time and only configure a DCI format fade out when I know we are in Q&A potential land, could even technically make that adjustment after the feature is rolling. Of course being in setup menus after a feature is rolling is big no-no, not to mention hard to train other operators on. Begging for trouble on all fronts.

Also annoying is the UI gives you no confirmation a format fade is occurring or when it completes, so would really want the output monitor in the booth mentioned prior. At least my macro showed you the fader change progression, assuming it wasn't lying to you and leaving the outputs alone. LOL. Maybe there is some merit in keeping them but simplifying the number of commands sent, as Ioannis suggests.

Can't have everything it seems without QSYS. ;-) Yes all in the name of a tiny showmanship improvements (and not wanting to run across the booth without coms). Remote fader, com improvements, and booth output level monitoring should remain on our bucket list. Lots of options still to consider that could negate the sensation that automation is somehow a worthwhile solution.

[Carsten Kurz]

As the AP20 offers plenty of presets, I would just create a preset for 'mute' and name it like that. Switching to 'Mute' would then also indicate at least '0' on the display.
Having a long fade attached to Mute generically is probably not a good idea, as usually, when you want to mute, you really want it to mute quickly.