The DR-DOS/OpenDOS Enhancement Project


FAT+ alternative
Post Reply   Forum
Posted by: NanoTech
06/14/2010, 14:12:50


I've been recently thinking about FAT-32 4 GB limit, which is a major problem on USB sticks and embedded flash disks (and DOS/Windows 9x).
We need a good backwards compatible, universal, open (unlike exFAT which is patent-encumbered) simple file system.

FAT+ is nice but is not so good at R/W compatibility.
I found another way to break the 4 GB limit with better compatibility.
It would work at a higher level.

Large (>=4 GB) files would be cut down as several files (I'll call them "file parts"). All of them would be 4 GB-1 byte except the last one.
An index file, at the directory root, with a special name unlikely to clash with normal files (e.g. large_files.{D18F60CC-E40E-402B-857E-0D03933BAF99}), would contain a list of large files and for each of them, the list of file parts making the large file.
File parts would virtually be concatenated.

At first, this may look very inefficient when dealing with a large number of large files, but actually there can not be many of them. There are at most 8191 large files (32 TiB / 4 GB) and at most 16382 file parts.

A large file aware OS would update this index file only when it's modified.
1) When a file becomes large or a large file size gains or loose a 4 GB part.
2) When a large file is deleted, moved or renamed.

I wonder whether this file should be a text file
e.g.

\dir1\dir2\bigfile.dsk
bigfile.dsk.0
bigfile.dsk.1
bigfile.dsk.2

Or a binary file with a format making insertion/removal efficient (e.g. like a FAT directory).

Compatibility:
This would be 100% compatible with fs checking and defrag utilities.
On unaware OS, data would still easily be available (cat'ing data before piping or copying to a fs with large file support).
The format could be simple enough to make users be able to manually create large files on an unaware OS by cutting the file with any file cutting utility and writing down file names in the index file.

Unaware systems, writing or removing file, could hardly harm:
1) Moving/renaming/deleting a file part would not update the index file, but when the fs is mounted again on a large file aware system, it could easily remove these entries from the index file.
2) Deleting file parts and replacing them with normal small files would not harm that much as a large file aware system could detect that the first file part is smaller then 4 GB-1 byte, except in the rare case where the new small file has this size.

On one hand the file system driver would be more complicated than FAT+ driver, on the other hand, it could be implemented as a TSR working with any logical drive, including CD-ROM and network drives.

I'm open to positive and negative comments.
I plan to implement it on Linux, DOS and Windows.



Post Reply View All  
Followups