Film-Tech Cinema Systems
Film-Tech Forum ARCHIVE


  
my profile | my password | search | faq & rules | forum home
  next oldest topic   next newest topic
» Film-Tech Forum ARCHIVE   » Operations   » Ground Level   » Need another reason to watch your movies before showing them to customers? (Page 1)

 
This topic comprises 2 pages: 1  2 
 
Author Topic: Need another reason to watch your movies before showing them to customers?
Frank Cox
Film God

Posts: 2234
From: Melville Saskatchewan Canada
Registered: Apr 2011


 - posted 08-25-2017 06:57 PM      Profile for Frank Cox   Author's Homepage   Email Frank Cox   Send New Private Message       Edit/Delete Post 
Yesterday afternoon I was doing my regular pre-screening of tonight's movie and there was a very short (less than a second) stutter in the picture and sound at one point in the movie.

I paused the movie and replayed that portion a number of times but couldn't make it happen again. The server control panel still says "Raid online", too.

After sending the logs to GDC for evaluation it's been determined that one of the hard drives needs replacement.

And it occurs to me that a very short-duration issue that apparently cleared itself up right away would never cause one of my customers to come out of the auditorium and tell me about it. Had I not been watching that movie myself I would never have known there was any problem. And I would really like to know about these things before it turns into a show-stopper.

So here's yet another reason to watch your movies before showing them to the public. You'll find out about issues that the nobody in the audience would bother to tell you about.

 |  IP: Logged

Justin Hamaker
Film God

Posts: 2253
From: Lakeport, CA USA
Registered: Jan 2004


 - posted 08-25-2017 07:26 PM      Profile for Justin Hamaker   Author's Homepage   Email Justin Hamaker   Send New Private Message       Edit/Delete Post 
I completely agree that it's good to sit and watch a movie on each screen from time to time. It allows you to check to see if screen files need tweaking, see if you have a speaker going bad, or any of a variety of other issues. But it doesn't necessarily do anything for a RAID issue. Those issues can come up at any time, so just because you've pre-screened the movie, doesn't mean you are safe from those issue when customers are present.

 |  IP: Logged

Frank Cox
Film God

Posts: 2234
From: Melville Saskatchewan Canada
Registered: Apr 2011


 - posted 08-25-2017 07:33 PM      Profile for Frank Cox   Author's Homepage   Email Frank Cox   Send New Private Message       Edit/Delete Post 
That's true, but this may or may not have been an "unreported issue" for the past week. I know that it wasn't a problem a week ago when I last watched the movie from beginning to end.

My point is that I'll find out about any unreported issues on a weekly basis. That may not be hard drive issues; perhaps an intermittent issue with the sound could manifest itself like that too. Doing a full inspection of the movie every week gives me a chance to find out about any issues that may not have been reported by the public over the previous week.

 |  IP: Logged

Mike Blakesley
Film God

Posts: 12767
From: Forsyth, Montana
Registered: Jun 99


 - posted 08-25-2017 11:53 PM      Profile for Mike Blakesley   Author's Homepage   Email Mike Blakesley   Send New Private Message       Edit/Delete Post 
Those RAID problems do indeed crop up anytime without warning. It's happened to me 3x now in the 7 years since we went to digital. Since I can hear the sound from the office, my ears are pretty tuned-in to anything going wrong with the sound. The RAID problem always shows up with one little two-second (or less) glitch. It may or may not repeat in the same spot. If you're lucky, it'll happen in a trailer.

When that happens, might as well go ahead and replace the hard drives because even if the problem doesn't recur right away, it WILL come back and will just get worse, until (at least in our case) within 2 or 3 days it will make your movie unwatchable. No amount of rebooting, reloading content or anything else will fix it, in my experience. And most maddening, the system will still show the drives as healthy!

Best advice is to keep a spare set of drives on hand.

 |  IP: Logged

Stephan Shelley
Jedi Master Film Handler

Posts: 854
From: castro valley, CA, usa
Registered: Nov 2014


 - posted 08-31-2017 08:55 PM      Profile for Stephan Shelley   Email Stephan Shelley   Send New Private Message       Edit/Delete Post 
Good luck doing that with the rerelease of Close Encounters of the 3rd Kind. 7pm show keys unlocked 1 hour before and we were on screen with another feature till 6:30pm.

 |  IP: Logged

Steve Guttag
We forgot the crackers Gromit!!!

Posts: 12814
From: Annapolis, MD
Registered: Dec 1999


 - posted 09-01-2017 06:16 AM      Profile for Steve Guttag   Email Steve Guttag   Send New Private Message       Edit/Delete Post 
Yes, because a 40-year old movie has to be kept under lock and key! I wonder how it might end this time! [Roll Eyes]

 |  IP: Logged

Stephan Shelley
Jedi Master Film Handler

Posts: 854
From: castro valley, CA, usa
Registered: Nov 2014


 - posted 09-01-2017 08:01 PM      Profile for Stephan Shelley   Email Stephan Shelley   Send New Private Message       Edit/Delete Post 
Must be for that preshow stuff they added. Lots of secrets told.

 |  IP: Logged

Marcel Birgelen
Film God

Posts: 3357
From: Maastricht, Limburg, Netherlands
Registered: Feb 2012


 - posted 09-04-2017 07:59 AM      Profile for Marcel Birgelen   Email Marcel Birgelen   Send New Private Message       Edit/Delete Post 
Hollywood just has no clue about day to day operations. You can't just dump some content somewhere and expect it to play flawlessly the first run.

Regarding those RAID glitches. It is pretty unfortunate they still need to happen. Seemingly only the most expensive storage systems know how to handle those kind of situations. In case a RAID array with parity, instead of waiting for the lagging disk to return the data, the controller or software RAID should immediately reconstruct it from parity data and flag the affected sector for further inspection during a background operation. Once too many of those time-outs on a single disk occur, it should flag that disk for replacement, before permanent failure occurs. That way, the playback can continue uninterrupted and the operator gets informed about possible issues with a disk.

Like I indicated, those systems do exist, but are usually reserved for enterprise grade storage appliances. Apparently, cinema equipment isn't deemed to be "enterprise grade".

 |  IP: Logged

Mike Blakesley
Film God

Posts: 12767
From: Forsyth, Montana
Registered: Jun 99


 - posted 09-04-2017 03:31 PM      Profile for Mike Blakesley   Author's Homepage   Email Mike Blakesley   Send New Private Message       Edit/Delete Post 
quote: Marcel Birgelen
In case a RAID array with parity, instead of waiting for the lagging disk to return the data, the controller or software RAID should immediately reconstruct it from parity data and flag the affected sector for further inspection during a background operation. Once too many of those time-outs on a single disk occur, it should flag that disk for replacement, before permanent failure occurs. That way, the playback can continue uninterrupted and the operator gets informed about possible issues with a disk.
I thought that was the whole POINT of a RAID...?

Apparently having a RAID only protects from catastrophic disk failures.

 |  IP: Logged

Frank Cox
Film God

Posts: 2234
From: Melville Saskatchewan Canada
Registered: Apr 2011


 - posted 09-04-2017 04:38 PM      Profile for Frank Cox   Author's Homepage   Email Frank Cox   Send New Private Message       Edit/Delete Post 
As far as I know, if a hard drive is dying in a GDC server the only way to know that is when the picture and sound start to stutter, and the only way to find out which drive is at fault is to send the logs to GDC for inspection.

 |  IP: Logged

Marcel Birgelen
Film God

Posts: 3357
From: Maastricht, Limburg, Netherlands
Registered: Feb 2012


 - posted 09-04-2017 05:29 PM      Profile for Marcel Birgelen   Email Marcel Birgelen   Send New Private Message       Edit/Delete Post 
The problem here is that off-the-shelf RAID systems are usually more tuned to data consistency than to continuous throughput.

For normal business and database applications, you don't want to give up on a disk, just because of an occasional timeout. A timeout could be caused by the disk remapping a bad sector to a spare one, for example.

If this would've been a normal business application, the impact would be rather minimal, maybe you'd experience a few seconds of non-responsiveness. But for real-time applications, like the streaming of a movie for playout, those hiccups can be killing.

To fix this, the manufacturers like Dolby/GDC/etc. would need to alter the RAID implementation or replace those off-the-shelf solutions with one of those expensive storage manufacturers. Obviously they're not really keen on doing that, it's also not really their core business anyway.

You can already see where it's heading though. Integrated Media Servers are becoming the norm now for new installs, they usually don't have a lot of on-board storage. So, handling storage the "correct" way will become the responsibility of your NAS supplier.

 |  IP: Logged

Mike Blakesley
Film God

Posts: 12767
From: Forsyth, Montana
Registered: Jun 99


 - posted 09-04-2017 07:42 PM      Profile for Mike Blakesley   Author's Homepage   Email Mike Blakesley   Send New Private Message       Edit/Delete Post 
quote: Frank Cox
As far as I know, if a hard drive is dying in a GDC server the only way to know that is when the picture and sound start to stutter, and the only way to find out which drive is at fault is to send the logs to GDC for inspection.
You're right except that last part...even after it starts to stutter the logs will show everything as being "A-OK." The on-screen report will show the drives as "healthy" while your picture and sound are hiccuping like a drunk on a Saturday night.

The last time it happened here, they thought the trouble was in the RAID box (not the drives.) So we swapped out the box, the trouble continued. I had a spare set of drives on hand and swapped those in, problem solved.

 |  IP: Logged

Steve Guttag
We forgot the crackers Gromit!!!

Posts: 12814
From: Annapolis, MD
Registered: Dec 1999


 - posted 09-05-2017 07:03 AM      Profile for Steve Guttag   Email Steve Guttag   Send New Private Message       Edit/Delete Post 
Generally, when I see hiccups on screen I look to the SMART data on the RAID. You can do this on the GDC if you have Administrative privileges (you get there from the Control Panel). Check each drive and look at item 5, reallocated sectors and for the number of errors showing on that drive. It isn't a guarantee but if you see double-digit reallocated sectors, odds are, there have been visible issues on screen due to that drive. If you do have some reallocated sectors, keep an eye on it and see how fast they are rising. More than one a month, I'd yank the drive.

 |  IP: Logged

Scott Jentsch
Phenomenal Film Handler

Posts: 1061
From: New Berlin, WI, USA
Registered: Apr 2003


 - posted 09-07-2017 10:16 AM      Profile for Scott Jentsch   Author's Homepage   Email Scott Jentsch   Send New Private Message       Edit/Delete Post 
What kind of drives are going into these RAID systems? Are they high-grade NAS drives, mid-grade NAS drives, or just consumer desktop drives?

 |  IP: Logged

Steve Guttag
We forgot the crackers Gromit!!!

Posts: 12814
From: Annapolis, MD
Registered: Dec 1999


 - posted 09-07-2017 10:41 AM      Profile for Steve Guttag   Email Steve Guttag   Send New Private Message       Edit/Delete Post 
All of the server companies have their "approved" drives. They are all going to be "Enterprise" grade drives (desktop drives can be a disaster in a RAID because they will try to re-read things figuring they are the only means of getting the data). By and large, HGST (Hitachi) seem to be the favored drives in most of the applications but definitely not all. Often the manufacturers will also list those by Western Digital and Seagate.

On the 2.5" drives (used by the IMS1000/2000, GDC's PSD system and Barco's ICMP) they are even more fussy about the drives since often 2.5" drives are not used in arrays but by laptop type PCs.

 |  IP: Logged



All times are Central (GMT -6:00)
This topic comprises 2 pages: 1  2 
 
   Close Topic    Move Topic    Delete Topic    next oldest topic   next newest topic
 - Printer-friendly view of this topic
Hop To:



Powered by Infopop Corporation
UBB.classicTM 6.3.1.2

The Film-Tech Forums are designed for various members related to the cinema industry to express their opinions, viewpoints and testimonials on various products, services and events based upon speculation, personal knowledge and factual information through use, therefore all views represented here allow no liability upon the publishers of this web site and the owners of said views assume no liability for any ill will resulting from these postings. The posts made here are for educational as well as entertainment purposes and as such anyone viewing this portion of the website must accept these views as statements of the author of that opinion and agrees to release the authors from any and all liability.

© 1999-2020 Film-Tech Cinema Systems, LLC. All rights reserved.