Q-SYS Corner

[Marcel Birgelen]

Yes, the larger "Cores" are indeed just DellEMC servers, but it's still a dedicated solution, as in, the machine just runs Q-Sys and nothing else. Like Sean mentioned, they're probably not entirely there yet. They probably don't have the licensing and deployment infrastructure yet and maybe they're holding back until they know how the "beast" behaves on hardware that isn't in their control.

What makes stuff a bit more complicated for virtualization is their dependency on real-time calculations. They're also not running a vanilla Linux kernel, but one with RTOS extensions. In a virtualized environment, it's hard to get 100% assured hardware resources all the time. While this usually isn't a big deal for your office application, this can be a major issue for sound processing. Even small hiccups will lead to audible artifacts, when streams suddenly get interrupted for a couple of milliseconds in a row.

Over the years, there have been some major improvements on most VM platforms though, where it was virtually (haha...) impossible to run something like a software PBX on a virtual platform, nowadays this is mostly a solved problem. Modern CPU extensions allow quasi real-time performance and dedicated resources for VMs. Getting it right is a learning process though, I can believe QSC is holding back until they have more insights in how it behaves on common virtualization platforms, including cloud platforms.

I'm not really a fan of QSC stating they see themselves as a software company in the future. Such statements have often lead to the demise of the company in question. You can (probably rightfully) critique Apple for their closed ecosystem, but Steve Jobs had reasons for it that did go further than just the profit motive. He's known for the quote "People who are really serious about software should make their own hardware." and while this wasn't even his own quote (which he admitted), I think he may be right to some extend. Hardware and software do go hand-in-hand, too often, software is considered to be this universal goo that strings everything together. It is often here where stuff starts to fall apart, unfortunately.

[Steve Guttag]

How many cinemas on a single CORE? Setting aside some philosophies on one CORE to multiple theatres versus one core per theatre, how many theatres can you put on a single CORE? The answer isn't just a function of the CORE's processing power. I'm finding the limiting factor is actually the input channel count. I just did a 10-plex design were 9 screens use a CORE510c (with a redundant CORE) and dual QLAN networks (each CORE gets its own UPS and transfer switch, as does each QLAN switch). Here is my "Check Design" info:



With just 9 screens, I've consumed 202 of 256 channels. That is the dominate criteria for cinema. Each screen has a DCIO-H and all inputs are wired in (16-channels of AES, stereo line, mono mic/line) so that is 19-channels/screen right there. I noticed that it was counting 22-channels/screen. I also have some A/V equipment (laptop ports, Blu-ray and Wireless Microphones). For the wireless Microphones, I used one of the amplifiers as an "on-ramp" so any of the theatres near the receivers (external antennas) can use any combination of the four microphones. That that is why the channel count isn't divisible by 9. Take those four off and you are left with 198-channels and divide that by 9 and you get 22/screen.

There are certainly ways to trim the channel count some. For instance, channels 9 and 10 on the AES input are for Left-Center/Right-Center. How many theatres have anything for them to play? If you don't have an ATMOS house, probably not so there are 2-channels/screen that can be freed up, which would save 18-channels in this plex for potentially something more useful. And if one were to do a larger plex, say 12-screens, that is 24-channels or one more theatre's worth of channels freed up. So dragging the DICO's output pins over to your next thing in the signal path may seem like the easy thing, but it isn't the most channel conscious thing.

One may think that you can also save channels by getting rid of channels 13 and 14 (and possibly 15 and 16) as they are often used as triggers now for things like 4DX and other non-audio applications. However, since these theatres have DCIO-H inputs and have HDMI capability, channels 9-16 have to be wired for 7.1 capability from HDMI sources.

Another thing to note is that channels don't count against you until you wire them! So if you have an amplifier with inputs (which all new systems now have as QSC has decided to eliminate the "Qn" versions, don't wire them unless you are using them.

This site has all SC-424 speakers for stage speakers and they are wired 4-way. Yet the channel count for outputs is significantly lower than that of the inputs! Much of that overhead is pushed off onto the amplifiers. It is just one channel that is going to the amplifier...the crossover and relating DSP is done in the amplifier despite it being a Q-SYS design.

Based on the above, I think, if you plan to support HDMI, you can safely handle 10-screens on a CORE510c, regardless of how fancy your alternative content audio may be. You can get up to 11-screens unless your alternative content gets out of hand and 12-screens, if you pay attention to what inputs you REALLY need. DSP wise, the CORE510c can handle that size plex unless you are just hammering it with talking to everything via plugins/user components on top of the audio load.

One more thing to note is the "Signal Processing by Category" meter above. See that little blue like at the far end? That is because I used a Feedback Suppressor component for the wireless microphones (multiple components, actually...one for each of the targeted auditoriums). It may seem small and insignificant but what it did was reserve the entire second half of the DSP for that sort of processing. What that means is that instead of only consuming about 25% of the CORE's DSP processing power, as the green part of the meter shows, I've actually consumed about 50% because once you use any category 2 processing, Category 1 (straight up audio) only gets to use the first half. Clearly, it isn't an issue here. But, if you are on a CORE110c, and doing Dolby ATMOS, it probably will be a limiting factor.

Here is the signal processing for the ATMOS house:

Screen Shot 2020-06-13 at 12.21.07 PM.png
I'm up to 62%. Note, it jumped from 50% to 61% so one apparently has to pay homage to the DSP gods when using second part of the DSP processing. So, in a sense, I'm just 2% over. This room has some A/V in it (laptop, Blu-ray, Mics). There is probably some non-essential processing I could have trimmed to pick up the CAT2 DSP but I'd still be right at the ceiling of CAT1.

For normal 7.1 theatres, the math says you should probably be able to get 5-screens on a single (redundant) CORE110c. I suspect that 4 would fit more comfortably, depending on your needs. If plain-jane 7.1 or 5.1 then 5 theatres probably works if you don't connect up what you don't really need.

[Marcel Birgelen]

Steve, the first image isn't loading correctly. But it's a nice exposition of how Q-Sys scales in some real-world applications. The system seems to reserve the resources for every component you put into it, whether it's being used at that time or not. I think, there's really no way around it if you want to play it safe, as the system itself cannot now how many of those channels will be in-use at the same time.

A feedback suppressor sounds harmless, but it's actually quite a complicated beast, as it includes stuff like harmonic analysis and adaptive filtering based on that analysis, so adding a few of those on your mic channels may fill up your DSP resources pretty quickly.

[Sean McKinnon]

That is a great analysis Steve!

[Steve Guttag]

Hmmm...the upper image was fine for me...until today. Here is the current version (which should be the same):

Screen Shot 2020-06-17 at 11.12.27 PM.png

Note, on the feedback suppressor. A poor mans way around using an active one is to use a real one during set up...find the frequencies that the room has an issue with, and then put in a PEQ that applies cuts to those frequencies likely to "run away" and result in feedback. It would also be possible to having that EQ be bypassed until either the overall level or the level at certain frequencies hit a certain level (use a notch filter on the suspect frequencies and a level comparator to un bypass the PEQ if a threshold is reached. Using suitable high pass and low pass filters for those frequencies outside of human speech also go a long way to preventing feedback. It won't be as good as a real one if the microphone may be moved about the auditorium (for a Q&A where the mic may move around or if it is placed in a poor spot, speaker wise). But it is still better than nothing and better than actual feedback.

Incidentally, this system is up now though the site hasn't opened yet (how many theatres are open now). The CORE510c and network is handling the 9 screens without incident. I'm using TP-Link Jet Stream series switches (T1500 at the screens and T1600 at the central audio) with the proper QoS settings and also IGMP query and snooping as per QSYS recommendations. It is an audio-only QLAN so IGMP setup isn't strictly required but recommended. Both local touchscreens and the WEB-UCI are up and fully functional. WEB-UCI comes with caveats now as they are continually improving it. It is working for me but there was one iteration where page-flips were not working well.

[Marcel Birgelen]

Replacing a "dynamic filter" that does a real-time harmonic analysis on your audio stream by a set of mostly static filters will probably free up a whole bunch of DSP resources.

But I was thinking about another solution, that may be more in-line with an interactive system like this. If you have a plex with 9 screens, how many screens could possibly require a microphone input all at the same time? Let it be three for example. A way to scale this, may be to set up an array of three "virtual feedback suppressors". In order to use them, you need to switch the auditorium in PA mode and then your microphone channel will be routed over one of those feedback suppressors.

It will introduce some potential complications though, that need to be handled somehow, like people forgetting to switch-off the "PA mode" after they're finished, thereby potentially blocking other auditoriums to go into "PA mode".

[Steve Guttag]

Marcel, the way QSYS uses resources, you pay a 50% DSP penalty to use the 1st Notch Feedback Suppressor. So use 1 or use 9, you've essentially consumed the same resources, effectively. That said, in this site, three auditoriums are set up to use any combination of 4 wireless microphones. They are, geographically, close enough that one set of external antennas work for all three screens. Though I've allowed for a separate feedback suppressor for each room (all three rooms are different, physically, so they can have different room modes).

[Marcel Birgelen]

I guess I have to read up about how Q-SYS reserves and allocates resources. I don't really understand how there can't be a resource difference between 1 or 9 feedback suppressors for example, since every one of them will have to use at least one thread on the systems CPU when being active.

[Steve Guttag]

I'd have to check to see just how much the resource meter moves if I were to add 6 more feedback suppressors but i didn't see it move between 1 and 3. It it would appear, to a degree, the first time you use a resource, you get a hunk of processing and you don't really consume much, if any more if you stay within that amount of processing. Sort of like when I went from 50-61% of CAT1 processing...that extra percentage really consumed 11%.

[Marcel Birgelen]

It's hard to find anything definitive on how "Signal Processing" resources are being calculated, but it has to be a combination of memory and processor usage. What could make sense is that a module like the feedback suppressor uses a large, very memory-intensive "hash table" that contains pre-calculated "hashes" for the harmonics analysis. This way, the actual DSP becomes way more "memory bound" than "CPU bound", since calculations are essentially being replaced by very fast table-lookups. Since you only need one instance of those tables for an "infinite" amount of instanced modules, it could explain why there isn't a noticeable increase in resource usage when you go from a single to three instances of the same module.

[Sean McKinnon]

Ther are level 1 and level 2 resources. If no level 2 resources are uses then all the cores processing power is available for level 1. As soon as you use any single Level 2 resource it no longer allows level 1 resources to use any of that space that is available to level 2 resources.

[Steve Guttag]

Something to consider when setting up COREs in a Primary/Backup fashion. Since the two COREs have the exact same settings, including the PTP configurations (have to be), the Grand Master clock will use the MAC address of the two COREs to determine which is Grand Master when both are on line. It isn't like the back up CORE is off line when it is in standby...it is fully operational so it can take over the moment the primary CORE goes off line so the PTP clock has to be arbitrated somehow. So, when selecting which box will be the Primary, select the one with the lowest MAC address or in the status, it will look funny because the backup CORE may be showing as the grand master clock.

[Steve Guttag]

Following up on the post above on the Grand Master clock and which core is the grand master. Many people use individual COREs for each screen...so in a 10-plex, 10-cores, which is very traditional for cinemas (one sound processor per cinema so a failure in screen 1 has no effect on screen 10...but is that true?).

Depending on your needs and how you set up your network(s), you may run into a problem with individual COREs, if you don't pay attention to your networks/Domains and PTP scheme. QSYS and the QLAN is a synchronized network and, believe it or not, the CLOCK is the highest priority, not the audio packets on that network (if the clocks are not in sync, the audio can't be). There can be only one grand master clock that the rest sync to.

The is set up (or forgotten about) is in the design properties. Please note, this box may look different, depending on which version of QSYS Designer you are running. Screen Shot 2020-07-19 at 1.16.06 PM.png

If you want to be able to stream audio between cores using QLAN Transmitters/Receivers or AES67 Transmitters/Receivers (e.g. synchronized), you will need to ensure that the two cores share the same domain and your network allows for them to see each other (likely on the same subnet as that is a best practice). That PTP Priority 1 is always set to 254 (and those that don't have that box...it too is set to 254). It doesn't have to be but that is the custom. PTP Priority 2 defaults to 100 and you can change that (like the domain and PTP priority 1) based on your needs. But what if you set up all of your cores without ever visiting the "Design Properties" area or do want to stream audio on the QLAN between COREs?

Then, when the various COREs see each other, they hold a quick meeting and elect a "Grand Master" (very democratic). The hierarchy is that if the COREs are on the same domain as each other (or any other PTP device capable of being the grand master...like an ATMOS processor), then they look to the PTP priority 1...the lowest number wins...since all COREs ship with PTP priority 1 at 254...they then look at PTP Priority 2 and since all COREs ship with that set to 100...we have a tie. But there can be only one Grand Master so the next thing they look for is the....MAC Address. The CORE with the lowest MAC address will be elected the "Grand Master."

This means that ALL COREs on that same network/domain will use that one CORE as their reference clock. There is no real problem with that as one clock is as good as the next (except in ATMOS applications...mostly because it comes/goes with the start/end of a movie).

The potential "gotcha" is if you need to reboot the CORE (or update it for design/firmware) when other COREs are actually playing audio. The moment the Grand Master clock goes off line...the rest of the cores are suddenly clockless! They will again, quickly hold a meeting to elect a new Grand Master and get on with their lives. This could (and will) cause a sound glitch so it is important to be cognizant that in a networked audio system with a synchronous clock just where that clock is and not to disturb it when others may be using it. There may also be something to actually setting up a hierarchy with the PTP priorities so you KNOW where the Grand Master is. It could be as simple as letting the PTP Priority 2 be the auditorium number or 10x the auditorium number so you can insert something if you need to later on without having to redo the scheme.

The other thing one can do is, if you are not going to (EVER) stream audio in a synchronous manner, make sure your COREs are on different Domains. This is also important if you are using Dolby ATMOS where you may choose to have a common subnet for support purposes. Dolby defaults to Domain 109 but you could, again, come up with a scheme like 100+screen number is the domain for ATMOS rooms to ensure that they never share the same domain yet can share the same network. It is just critical that whatever the Domain is that they match between the ATMOS processor and the CORE. It is also critical that the ATMOS processor is the higher priority/lower PTP priority number (it has to be the Grand Master)...this is normally accomplished by leaving PTP priority 1 at 254 so the ATMOS processor, that typically ships set to 128, so it will win the Grand Master election.

Just because you may choose to not have synchronous audio between COREs, that doesn't stop you from using the asynchronous streams like WAN and Media Stream for things like music.

[Marcel Birgelen]

If you run a QSys core or stack per auditorium, I would likely set-up a different time (PTP) domain for each of these, to avoid clock confusion. I've not yet tested this, but what I find annoying, in a case that you would have one large PTP domain and you do reset the grand-master-clock, there is no graceful handover between master clocks, without an audio glitch. It would be something different if a core drops out due to a malfunction or power-outage, but a planned reboot should, in my opinion, gracefully transfer any "master roles" to one of the remaining members in the pool.

B.T.W.: It looks like even in technical applications, we're no longer allowed to use terms like "master" and "slave", but should refer to it as "host" and "client"...

[Steve Guttag]

Nah...Master/Slave, Male/Female...still used. But more to the point, "Master" can be used in even non master/slave applications. It may be called "Global" in some application to denote it covers more than a local change.