Adjusting Doremi Server Time

[Ryan Gallagher]

So dolby provided me the IncreaseAdjustmentLimit DRM package for the one time change of up to 2 hours to correct our secure clock drift because we have never had an NTP configured.

They didn't link a guide (and google is not helping locate one) so just inquiring what to expect. Will it prompt me during the reboot install sequence for the GMT time or something?

Ours is ahead 58min right now, but it feels like it fluctuates depending on daylight savings time, which I would have assumed was compensated for.

Just don't want to reboot and then not know what the next step is. Thanks.

PS they provided our 20.05c dolphin 1.2 fimrware and got that sorted today. The time adjust was a bonus inquiry which they happily provided.

[Ryan Gallagher]

Actually, looking at the package it's just a .dlm inside... so I don't think it installs anything or launches a wizard, rather just gives me permission to do something.

So naturally, where do I set the time once the adjustment license is active? On the shell with root? Or within the time settings menus?

[James Gardiner]

From my understanding the DLM allows it time to be updated a certain amount. So once installed, just manually update..
Or if your running on NTP, the software will then be able to shift the time back to the position needed.

[Adrian Avram]

If there is an NTP server set up in Time adjust, it will automatically correct the time on the next reboot.
If you want to do it manually, after you ingested the dlm, just log in the Terminal woth root, and write
/doremi/sbin/ ntpwrap.sh and hit Enter. The clock should display the corrected time after a couple of seconds.

[Caleb Williams]

I would add to this if you're using the terminal, make sure to use the doremi script instead of just running the date command. If you don't use the web ui or the script Adrian mentioned you'd only be adjusting the server time and not the secure IMB time; that can cause issues.

[Leo Enticknap]

The bottom line is that there has to be a stratum 0 or 1 NTP server that the Doremi can see, the address of which is configured in the control panel. Even with the license file, you can't enter the time manually. In a multiplex, this is typically the TMS. In single screen theaters without a TMS, the SMS will need a gateway to the Internet on one of the NICs, and an external NTP server configured. I typically use one of the NIST ones (192.12.19.20 and 128.138.140.44 are the geographically closest to my neck of the woods and therefore most likely to offer a sync that the SMS accepts, but that might not be the case for Austin).

If, for security reasons, you don't want your SMS to have a permanent Internet gateway (and from IMS3000 malware infection incidents we know that this could be risky), you can always remove it after the reboot with the time adjust license loaded, and the secure clock has synced. However, with no NTP server available on the LAN, the clock will drift again over time, and eventually you'll need to rinse and repeat.

It is possible to make a regular Windows PC work as a NTP server. Depending on its Internet connection, sometimes it's good enough to qualify as stratum 1, and therefore be usable by a SMS to sync from. I have a cheat sheet for doing this. Ryan - please PM me if you'd like it (it's too big to attach here).

[Steve Guttag]

The Dolby DSS servers got me off of using NIST. Dolby servers put a 15-hop limit. I started using the NTP pool (us.pool.ntp.org). I recently ran into an issue where even that was an 18-hop journey. Going to time.google.com got the hop count down to below 10 (from the same site). When in doubt, do a run a tracert to see how many hops it takes to get to the NTP source.

Note, if a TMS isn't on site but a remote support PC is, that can become your NTP source so long as it has access to an NTP server...there are configurations to be done so that it will act as an NTP time source as well as opening its firewall on UTP port 123.

[Ryan Gallagher]

Thank you for the sage advice, asking first was the right instinct.

I was heading into it assuming a manual adjust was possible and most direct, but it sounds like Leo’s approach of temporary setup with a internet gateway and suitable ntp (checked for hops before hand) might be our best method.

I mentioned in my request to Dolby we had no NTP configured, they simply said they “advised” using an NTP, not that it might be required to do this. The manual adjust in the doremi UI might work, but ntp seems more fool proof, even if just temporary as it would be in our case.

Will take a gander at Leo’s guide doc too, thanks!

[Steve Guttag]

You can set the clock manually (After applying the DLM for the extended amount). Note, having an NTP source doesn't stop the secure clock from drifting. Essentially, when you set the secure clock, either manually or via NTP, you are applying an offset. Once your original 360 seconds are consumed, you can't offset anymore. The DLM doubles that (for 1-year...odds are you'll be back needing another DLM). I always set them via the WebUI. As others have indicated...the BIOS clock isn't the fussy one...it is the mediablock (Secure) clock that is of interest and has the 6m/yr restriction.

[Harold Hallikainen]

There are also GPS-based NTP servers such as https://www.aliexpress.us/item/3256808930759537.html , which I have not tried. Many sources are available.

[Ryan Gallagher]

Cheers all. I opted for the temporary NTP method, no snags. Sync'd immediately on next reboot with us.pool.ntp.org configured, temporary eth1 active with DHCP config.
3464s of 7200s of our offset quota utilized. SYNCRHONIZED!

I don't think this server stays in play here for longer than a year, so I may not have to experience what happens if that time extension was only valid for a calendar year.

No TMS here or Stratum 0 or 1 NTP, so back to our time-free wormhole after the one time NTP sync. But at least studios can't blame us now when their tech-check key is in danger of expiring because they are running late. (I've always avoided actual cluster-fs by cuing up their feature and with suitable pauses to prevent eject or advancement beyond it in a SPL, I'm sure I'll find at least one occasion where I regret not having the extra 30 min or more of key windows ahead of a need, but 58min was getting dangerous on the other end).

[Ryan Gallagher]

A couple points of clarification for anyone else needing extended time adjust in the future, I asked Gary at Dolby follow up questions since we don't have a permanent NTP configured.

The Extended Time DLM provided to us is permanently on the server, the quota still resets on Jan 1st but the quota amount is permanently increased to whatever the DLM provided (7200s in our case to overcome our initial adjustment).

The old 360s max per year adjust quota (or whatever a prior extended time DLM provided) is no longer in play, so if you are not on an NTP, catching adjustments before 360s is exceeded is not a thing anymore, it's whatever the new DLM sets as the limit.

So in our case, if NTP remains unconfigured (for now), we are free to repeat our adjustment procedure at any time we notice drift we wish to correct (provided usable quota is available). Setting up NTP would obviously be the most hands free painless version.

I expect our case is an outlier... it's probably pretty rare to have a server go entirely unadjusted for 9 years of it's installed life, so don't expect dolby to just hand out 2 hour adjustment DLMs willy nilly. But they did overshoot our adjustment need by 2x with the DLM, which I would not have expected.

EDIT: My TD tells me he corrected our DCP2K4 out of bounds one other time back when Doremi ran the company, and was a 1 time 1800s max adjustment for the need back then. He didn't think it reset annually, but his memory is fuzzy there. Scrolling back in my old photos, my snap of our diagnostics screen concurrent with our key extension shows 360s annual quota, so indeed the prior 1 shot DLM behaved different to the one we now have installed.

[Brandon Wood]

Our theater was closed for awhile before we officially purchased it. During the planning of reopening, I looked at the server (DSS200) and noticed the time in the UI was set to 2008! However, adjusting the BIOS clock to the current date and UTC time made it correct itself (although it's still ahead by about 3 mins). I have noticed it drifting (it's currently about 11 mins ahead), but rebooting and readjusting the BIOS puts it back to only being 3 mins ahead. It would be nice to truly fix it if there's a non-scary way (I assume it's replacing the battery, something I don't want to do but could maybe call our service tech for). It's our only server and if it fries we'd have to close down for awhile. Renovations are happening so I'm really just hoping to milk the server and projector (Christie CP2000-ZX) until we can install something new.

[Caleb Williams]

There are also GPS-based NTP servers such as https://www.aliexpress.us/item/3256808930759537.html , which I have not tried. Many sources are available.

When we were running DSS200 servers we ran a similar box on a public domain and pointed all of the LMS' to it. It generally worked well unless there were internet issues in the building where it was hosted. The IMS2-3000 servers don't require such a low stratem anymore so we just use the default time server on CentOS 7 w/Screenwriter ([0-4].centos.pool.ntp.org) and have mothballed the domain. I can't speak for Doremi or IMS1000 servers, but IMS2-3000 servers have much better internal clocks that the DSS200 servers did. I would only expect to see an adjustment of a second or two a day. Out of the 67 screens we have now, I have only had to apply a dlm to adjust the time twice in the last 5 years, and that was due to the GPS NTP server failing quietly and those servers being barely outside the adjustment limit. If you have to apply larger adjustments to the time over shorter periods that could be an indication of an issue with the server, or a component that lost the "silicon lottery". Dolby may not like issuing a lot of adjustment dlm's for a specific server without asking some questions

[Caleb Williams]

Our theater was closed for awhile before we officially purchased it. During the planning of reopening, I looked at the server (DSS200) and noticed the time in the UI was set to 2008! However, adjusting the BIOS clock to the current date and UTC time made it correct itself (although it's still ahead by about 3 mins). I have noticed it drifting (it's currently about 11 mins ahead), but rebooting and readjusting the BIOS puts it back to only being 3 mins ahead. It would be nice to truly fix it if there's a non-scary way (I assume it's replacing the battery, something I don't want to do but could maybe call our service tech for). It's our only server and if it fries we'd have to close down for awhile. Renovations are happening so I'm really just hoping to milk the server and projector (Christie CP2000-ZX) until we can install something new.

The BIOS battery is super easy to replace if you have a phillips screwdriver and something non-conductive to pry the battery out. You do have to lift the Cat. 862 out of the way to expose it but then it's just a CR2032 battery on the motherboard. You will have to adjust the time again after replacing it because bios settings will be lost. Replacing the battery will not help with long-term drift though, those servers just suck at keeping time. The ones we had would lose/gain several minutes per day