Disk storage - a terabyte beauty is born

WIRED: The challenge with surfing your business model on the waves of technological advance is to notice the day that your wave…

WIRED:The challenge with surfing your business model on the waves of technological advance is to notice the day that your wave has been pulled from underneath you, and it's time to throw yourself down on another. In the 1990s, Windows leaped to prominence the moment Microsoft's team realised that PCs would soon grow powerful enough to run several of the old Microsoft DOS programs side-by-side (rather than one program at a time, as was then the norm).

Microsoft's competitors were outfoxed because they didn't see the change coming, and were trapped in the old DOS world that Microsoft escaped.

The PC world may be about to experience another one of these flips - but it's not in computing power, it's in disk storage. And Microsoft may be one of the losers.

This week, Apple announced the first consumer terabyte server. Called "Time Capsule", it will sell for $499 (or €499, at Apple's conversion rate), and it stores more than 1,000 gigabytes of data.

READ MORE

In the next few years, our drives will be dealing with dozens of terabytes - and Apple, Microsoft and Linux software will be struggling to keep up.

The problem isn't what to put on the drives: whether it's backups of our other data or movies or some new data-intensive application we have yet to dream of, we'll find a way to fill that gaping space. It's whether the filing system of these operating systems can keep up with our plans.

Filing systems are the maps and rules by which computers throw down data onto permanent storage. Designing a filing system is like trying to devise the perfect way of storing inventory in the back of a shop: you want to be able to save goods quickly, but find it just as fast. You want flexibility to remove as well as pile stuff back there, but you don't want a lot of unusable space when you've pulled out lots of tiny stuff from different parts of the storage space. And you definitely don't want to lose anything, ever.

Modern filing systems are a mix of trade-offs, and work very well. They're part of the most tweaked, optimised and reliable pieces of software, which is hardly surprising given how constantly we save to a hard drive, and how precious the data we store can be.

But there are some buried presumptions in filing systems that can be hard to undo. Remember the strangely curtailed BIGWORD.XXX filenames of old Windows files? That came from the restrictions of the DOS filing system that lay under it.

NTFS, Windows' current filing system, doesn't have those kind of hardwired limits, but it does come with some presumptions. It assumes that setting up a new drive is something you rarely want to do, and that repairing bad data on a drive is a rare act too. So it doesn't bother too much about making those actions slow and cumbersome. The same is true for Linux's filing system, ext3.

Both (hopefully) will stay true - but with multiple terabyte drives, the penalties when you do have to repair or create drives are far bigger. It can take hours to reformat a hard drive. When Linux needs to peer closely over a super-large drive that it suspects of having bad data, it can take days.

The answer is a new generation of filing systems, developed by companies that have already dealt with these problems. ZFS, by Sun, is recognised as one of the more forward-thinking of filing systems. Before NTFS, Windows could only cope with 8GB drives. Building a drive big enough to fox ZFS would involve, famously, "boiling the oceans" (somebody did the mathematics).

More importantly, it can cope with repairing and reconfiguring drives in a matter of seconds.

Apple has already marked its intention to adopt ZFS, at least unofficially. It was expected to appear in their latest operating system upgrade, but was released instead as an experimental add-on.

Linux has an interesting problem adopting ZFS - while the design of ZFS is open source (why Apple can use it without permission from Sun), it's not quite under the same kind of open source licence as Linux itself. Legally that means the two cannot be wired together to the same cosy extent as filing systems and operating systems usually are. It turns out that in this case, companies like Apple have a head-start on Linux, because they are able to adopt ZFS far more easily.

Which leaves Microsoft. Microsoft's problem is in the other direction: while, technically, it could adopt ZFS, it would appear a particular defeat for the company. Microsoft's last attempt to produce a sequel to NTFS, WinFS, was abandoned in the race to get Vista out of the door, and the ultra-competitive company seems unlikely to adopt an alternative created by a competitor.

The incredibly ambitious WinFS, as far as anyone knows, is still under development in Microsoft's Redmond campus. But the clock is ticking. Will Microsoft - and Linux - find themselves left behind by the leap in hard drive size? Or will it be, to mirror Norma Desmond, that it wasn't the drives that got big, it was their filing systems that stayed small?