Ingestion Indigestion

The Relentless Struggle
by Mark C. Petersen, Loch Ness Productions
A presentation for the IMERSA Summit,
Denver, Colorado, March 2014

© 2014, Loch Ness Productions

Actually, until Matt Mascheri said he wanted me to talk about how Loch Ness Productions INGESTS all the fulldome files and such we get from other producers, I'd never used the term INGESTION. But I guess it's appropriate enough. After all, the gyrations I have to go through whenever someone sends me a hard drive are enough to give anyone ulcers!

When I get a drive delivered to me, I want to see specific things on it, and there are things I DON'T want to see. I'm going to show you the way we prefer to work at Loch Ness Productions. If you follow our methods, I'll be happy, because you'll have done things correctly. If someone else wants to do things differently — well, they're wrong :-)

So you're going to send out a hard drive with your creation, your baby on it.

Step #1 — Format it first!

For one thing, that will remove all the unwanted crapware the drive manufacturer puts on there, including all the hidden autorun files that plant icons and logos and create virtual drives and install drivers you don't want.

Here's what I'm talking about. This one I get to blame on Robin Sip. He sent me a drive, and the first time I plugged it into my workstation, it installed something that messed up Windows royally. Look at the folder tree on the left. I have a C drive, a D Drive, an E Drive, an F drive... and a duplicate E drive, with identical contents to the first one — but this one's labeled "Mirage3D". Now Windows is screwed up, it thinks it has 2 E drives. Every time I plug anything in that gets assigned E drive, I get this doubled entry, and I see "Mirage3D" show up in my folder tree, even though that's bogus. I've tried deleting the just second entry, and it sure enough deletes them both. I've scanned the Registry. I can't find what happened. But it's really annoying and four years later, I still haven't figured it out.

I've also had a horrible experience with an autorun rootkit worm, on a drive from another of our trusted distribution partners. I didn't practice safe computer sex; when I got it, I plugged it directly into our server. Instantly I started getting popup windows to porn sites, and Internet Explorer came up (which I never even use on our server) with its homepage commandeered, trying to log in to more foreign Web sites. I quickly shut down, but the damage had been done. I had to wipe the drives, reinstall and reconfigure the RAID, which I hadn't done the first time, since I bought it that way from Dell. And that's when I discovered that the stock version of Windows Server from the CDs is not the one Dell had installed, so I had to sleuth out all the mods they'd put on. The server is still not back to the way it was originally. And then I discovered the worm had spread to 4 other machines on our LAN that happened to be powered up at the time. They all had to be wiped clean and new reinstalls of Windows put on. I lost a week trying to recover.

We now keep a laptop, isolated from the network, for the sole purpose of ingesting incoming drives. Now when I get a drive in, first thing I do is plug it into the isolated laptop and run a virus scan on it. I learned the hard way.

When formatting, use NTFS, so we don't run into the 4GB filesize limitation. You probably make movies larger than 4GB; we certainly do. Everyone can read NTFS. Macs can't write to them easily — but you don't want them to be writing to your drive anyway.

When you format the drive (or afterward), give it a meaningful name, so that I can see that in the Explorer folder tree when I plug it in, so I know whose drives I've got connected. Keep it short. With NTFS, you're not limited to eleven characters like you used to be, but I still keep to the old ways.

So what do we want to see on that drive when we get it? Simplicity. No ambiguity. I'm going to show you the directory structure to use. It's what we do. It's the Holy Grail. Do it this way.

Your drive has a volume label. At the root level, there will be one folder, with the name of your show. If you have more than one show on the drive, they each get their own root level folder. That's all. Nothing else. There can only be one folder to look in, for any show's content.

Do not use spaces or punctuation in the folder name. This will screw up any DOS-level programs you may want to run, and will confuse many renaming utilities. You can use CamelCaps, or underscores. They told me at Evans & Sutherland they use only CamelCaps for folder names, and underscores in file names. I hadn't been that fussy before — but in any case, no spaces!

When I open the root level folder, this is what I want to see: Subfolders for "frames", "audio", "video", "promo", "script", "license", etc. One word names, no spaces. They're short, they can only mean one thing. Where's the audio for the show? It can only be in one place, the audio folder.

If you have a "readme.txt" file, this is the place for it, in the show folder. Name it using the title of the show, so if it's accidentally moved, you know from the filename what show it's for. And they can't accidentally get overwritten, whereas they could if they were all generically named "readme.txt".

Open the frames folder, and we want to see a folder named for the pixel diameter of the dome masters. In this example, we have 4k frames for the show and trailer, and we have 1k frames for the trailer as well.

Now this is critical. Do not put all the show frames in one folder. Break them up into groups of 10,000, and put them in sequentially numbered folders, as shown above.

Why? If you've ever clicked on a folder in Explorer with 90,000 frames in it, you know. It takes Windows forever to build the directory. Earlier versions of After Effects will simply crash with frame sequences longer than 10,000. With them logically ordered and rationed, things are easier to work with.

Numbering schemes start at 0. Not 1. No arguments.

I've actually seen folder names like the purple line. Do not do this. I can tell they're JPG files, there are 10,000 examples to look at. I already know they're frames, they're in the "frames" folder. No spaces. No commas. No dashes. No octothorpes. Just — No.

Ideally, those 2-digit subfolder names wouldn't even need the leading zero in them, as most shows don't have more than 100,000 frames. But if you're running at 60 frames per second, you might. At any rate, you want to be sure Explorer keeps your folders in numerical, not alphabetical order — you don't want 11 to follow right after 1. So that's why the leading zero is there; otherwise you can ignore it. In the real world, nobody makes fulldome shows longer than an hour anyway. And nobody makes trailers longer than 10,000 frames either.

The filenames for the frames are as indicated. The title (you can abbreviate it; we usually use a 3-letter acronym), an underscore, the word "show" or "trailer", an underscore, the frame number, the dot, and the extension. No spaces. Only one dot.

Moving on to audio — two folders, named "stereo" and "surround". Not 5.1, not 7.1 — those have dots in them that can confuse those rename utilities. Stereo, surround. That's all we'll ever run into.

Again with the filenames: the title, an underscore, whether it's the show or trailer, a dot, and the extension. If there's something special, like a different language version, that goes after the show/trailer word and underscore, before the dot.

The accepted convention for surround channels is as shown. There's no such thing as Left Front or Right Front; your left and right channels ARE in the front. Indicate them "_L" and "_R". Your Low Frequency/Effects channel is "_LFE" in all caps, not "Lf". Ideally, you'd indicate the surround channels with a subscript "s", but you can't do a subscript in ASCII, so you use a lower-case s instead: "_Ls" and "_Rs". Like that. No exceptions.

If you're generous enough to provide a small reference video that shows how the frames and audio are supposed to sync together, that goes in the "video" folder. Be sure the fisheye movie is printed from the actual frames you use; include the corner frame numbers if you want, even though they're often hard to read when you scale them down.

If you provide flatscreen versions, to be put on a Web site or something, these too go in the video folder. They're video, after all.

If you provide promo materials — graphics for posters, a presskit, like that — they go in the "promo" folder. Again, the title, an underscore, what it is, one dot, and the extension.

Same goes for the "script" folder. You should consider being good enough to provide both the fully annotated script — with timings, visuals and narration text, line numbers, etc. — and also just the narration in plain ASCII text, for use in captioning devices.

In the immortal words of Lufthansa: "There's no better way..." Follow these rules, and all will be right with the world.

---

If you find this information useful, you can help support our efforts. Please go here or here or here.