Page 1 of 6

issue-#138 Support for files larger than 2Gb

Posted: Wed Oct 19, 2016 1:55 am
by DGDanforth
In HostFiles we find

Code: Select all

		File = POINTER TO RECORD (Files.File)
			state: INTEGER;
			name: FullName;
			ref: WinApi.HANDLE;
			loc: Locator;
			swapper: INTEGER;	(* index into file table / next buffer to swap *)
			len: INTEGER;
			bufs: ARRAY nofbufs OF Buffer;
			t: LONGINT	(* time stamp of last file operation *)
		END;
The len: INTEGER restricts files to be less than 2,147,483,648 bytes.
My mail inbox is 2,865,152,000 bytes. That causes BB to trap when attempting
to open the file for the purpose of copying it to backup.

I request that we make len: LONGINT

Are there any negative consequences for doing that?
-Doug

Re: Max file size

Posted: Wed Oct 19, 2016 1:57 am
by Ivan Denisov
This should be fixed! However that is not a bug. I have to move your request to features section.

Re: Max file size

Posted: Wed Oct 19, 2016 2:02 am
by Ivan Denisov
Target version for this issue can be 1.8 because it will change many interfaces: Length() SetPos() etc.

Re: Max file size

Posted: Wed Oct 19, 2016 2:32 am
by DGDanforth
Ivan Denisov wrote:Target version for this issue can be 1.8 because it will change many interfaces: Length() SetPos() etc.
Right.
For example

Code: Select all

		FileInfo = POINTER TO RECORD 
			next: FileInfo;
			name: Name;
			length: INTEGER;
			type: Type;
			modified: RECORD 
				year, month, day, hour, minute, second: INTEGER
			END;
			attr: SET
		END;

Re: Max file size

Posted: Wed Oct 19, 2016 4:56 am
by Josef Templ
Is the TRAP at a reasonable place?
I mean, is it possible to see from the TRAP that there is an overflow of the file length?

If not, then we should add a check in HostFiles.Old, where the length is initialized.

Code: Select all

				f.len := WinApi.GetFileSize(ref, j);
				ASSERT(j = 0);   (* >>> add this line *)
This would then be a bug fix again, not a feature.

Changing Files.File.Length) to LONGINT is not an option within 1.7.x.
It may be done in 1.8 but it will have some performance impact since LONGINT
on a 32-bit platform is slower than INTEGER.
It would be a natural choice on a 64-bit platform, of course.

What is more realistic, at least in 1.8, is a change of Files.FileInfo.length to LONGINT.
This is not performance critical and would allow one to check if the file length
is valid before opening the file.

A very practical option would be to add 64-bit modules, say LFiles and HostLFiles (or Files64 and HostFiles64).
For the purpose of file operations such as copy this would be sufficient. The rest of the system
would still use the 32-bit Files module. This could be done in 1.7.1, I guess.
It is very much like Math and SMath.

- Josef

Re: Max file size

Posted: Wed Oct 19, 2016 5:32 am
by DGDanforth
Is the TRAP at a reasonable place?
I mean, is it possible to see from the TRAP that there is an overflow of the file length?
One sees that the file length is negative.

Re: Max file size

Posted: Wed Oct 19, 2016 9:50 am
by Josef Templ
DGDanforth wrote:
Is the TRAP at a reasonable place?
I mean, is it possible to see from the TRAP that there is an overflow of the file length?
One sees that the file length is negative.
Well, you see this only if the length is between 2 and 4 GB.
With 5 GB it would be positiv again, but truncated to the low-order 32 bit.
This should really be checked in Old.

The more I think about it the more I like the variant with adding a Files64 (alias LFiles) module.
Actually, this is the only possibility for now to provide support for large files.
It would be straightforward to do, I believe, and we could then claim to support large files.

- Josef

Re: Max file size

Posted: Wed Oct 19, 2016 10:53 am
by Robert
Josef Templ wrote:The more I think about it the more I like the variant with adding a Files64 (alias LFiles) module.
Actually, this is the only possibility for now to provide support for large files.
It would be straightforward to do, I believe, and we could then claim to support large files.
A long time ago - long before BlackBox became open source, we commissioned Oms to provide big file support.
We needed to read / write third party .bin files; we didn't need full BlackBox View support for big files.

I am not in a position to publish the code, but the interfaces may be of interest here. There are no compatibility problems with the 32-bit file size code.

Code: Select all

DEFINITION ToolsBigFiles;

	IMPORT Files;

	CONST
		archive = 3;
		ask = TRUE;
		dontAsk = FALSE;
		exclusive = FALSE;
		hidden = 1;
		readOnly = 0;
		shared = TRUE;
		stationery = 4;
		system = 2;

	TYPE
		Directory = POINTER TO ABSTRACT RECORD 
			(d: Directory) Delete (loc: Files.Locator; name: Files.Name), NEW, ABSTRACT;
			(d: Directory) FileList (loc: Files.Locator): FileInfo, NEW, ABSTRACT;
			(d: Directory) GetFileName (name: Files.Name; type: Files.Type; OUT filename: Files.Name), NEW, ABSTRACT;
			(d: Directory) LocList (loc: Files.Locator): Files.LocInfo, NEW, ABSTRACT;
			(d: Directory) New (loc: Files.Locator; ask: BOOLEAN): File, NEW, ABSTRACT;
			(d: Directory) Old (loc: Files.Locator; name: Files.Name; shared: BOOLEAN): File, NEW, ABSTRACT;
			(d: Directory) Rename (loc: Files.Locator; old, new: Files.Name; ask: BOOLEAN), NEW, ABSTRACT;
			(d: Directory) SameFile (loc0: Files.Locator; name0: Files.Name; loc1: Files.Locator; name1: Files.Name): BOOLEAN, NEW, ABSTRACT;
			(d: Directory) Temp (): File, NEW, ABSTRACT;
			(d: Directory) This (IN path: ARRAY OF CHAR): Files.Locator, NEW, ABSTRACT
		END;

		File = POINTER TO ABSTRACT RECORD 
			type-: Files.Type;
			(f: File) Close, NEW, ABSTRACT;
			(f: File) Flush, NEW, ABSTRACT;
			(f: File) InitType (type: Files.Type), NEW;
			(f: File) Length (): LONGINT, NEW, ABSTRACT;
			(f: File) NewReader (old: Reader): Reader, NEW, ABSTRACT;
			(f: File) NewWriter (old: Writer): Writer, NEW, ABSTRACT;
			(f: File) Register (name: Files.Name; type: Files.Type; ask: BOOLEAN; OUT res: INTEGER), NEW, ABSTRACT
		END;

		FileInfo = POINTER TO RECORD 
			next: FileInfo;
			name: Files.Name;
			length: LONGINT;
			type: Files.Type;
			modified: RECORD 
				year, month, day, hour, minute, second: INTEGER
			END;
			attr: SET
		END;

		LocInfo = Files.LocInfo;

		Locator = Files.Locator;

		Reader = POINTER TO ABSTRACT RECORD 
			eof: BOOLEAN;
			(r: Reader) Base (): File, NEW, ABSTRACT;
			(r: Reader) Pos (): LONGINT, NEW, ABSTRACT;
			(r: Reader) ReadByte (OUT x: BYTE), NEW, ABSTRACT;
			(r: Reader) ReadBytes (VAR x: ARRAY OF BYTE; beg, len: INTEGER), NEW, ABSTRACT;
			(r: Reader) SetPos (pos: LONGINT), NEW, ABSTRACT
		END;

		Writer = POINTER TO ABSTRACT RECORD 
			(w: Writer) Base (): File, NEW, ABSTRACT;
			(w: Writer) Pos (): LONGINT, NEW, ABSTRACT;
			(w: Writer) SetPos (pos: LONGINT), NEW, ABSTRACT;
			(w: Writer) WriteByte (x: BYTE), NEW, ABSTRACT;
			(w: Writer) WriteBytes (IN x: ARRAY OF BYTE; beg, len: INTEGER), NEW, ABSTRACT
		END;

		Name = Files.Name;

		Type = Files.Type;

	VAR
		dir-: Directory;
		docType-: Files.Type;
		objType-: Files.Type;
		stdDir-: Directory;
		symType-: Files.Type;

	PROCEDURE SetDir (d: Directory);

END ToolsBigFiles.

Code: Select all

DEFINITION ToolsBigFMappers;

	IMPORT ToolsBigFiles;

	TYPE
		Reader = RECORD 
			rider-: ToolsBigFiles.Reader;
			(VAR rd: Reader) ConnectTo (f: ToolsBigFiles.File), NEW;
			(VAR rd: Reader) Pos (): LONGINT, NEW;
			(VAR rd: Reader) ReadBool (OUT x: BOOLEAN), NEW;
			(VAR rd: Reader) ReadByte (OUT x: BYTE), NEW;
			(VAR rd: Reader) ReadChar (OUT x: CHAR), NEW;
			(VAR rd: Reader) ReadInt (OUT x: INTEGER), NEW;
			(VAR rd: Reader) ReadLong (OUT x: LONGINT), NEW;
			(VAR rd: Reader) ReadReal (OUT x: REAL), NEW;
			(VAR rd: Reader) ReadSChar (OUT x: SHORTCHAR), NEW;
			(VAR rd: Reader) ReadSInt (OUT x: SHORTINT), NEW;
			(VAR rd: Reader) ReadSReal (OUT x: SHORTREAL), NEW;
			(VAR rd: Reader) ReadSString (OUT x: ARRAY OF SHORTCHAR), NEW;
			(VAR rd: Reader) ReadSet (OUT x: SET), NEW;
			(VAR rd: Reader) ReadString (OUT x: ARRAY OF CHAR), NEW;
			(VAR rd: Reader) ReadXChar (OUT x: CHAR), NEW;
			(VAR rd: Reader) ReadXInt (OUT x: INTEGER), NEW;
			(VAR rd: Reader) ReadXReal (OUT x: REAL), NEW;
			(VAR rd: Reader) ReadXString (OUT x: ARRAY OF CHAR), NEW;
			(VAR rd: Reader) SetPos (pos: LONGINT), NEW
		END;

		Writer = RECORD 
			rider-: ToolsBigFiles.Writer;
			(VAR wr: Writer) ConnectTo (f: ToolsBigFiles.File), NEW;
			(VAR wr: Writer) Pos (): LONGINT, NEW;
			(VAR wr: Writer) SetPos (pos: LONGINT), NEW;
			(VAR wr: Writer) WriteBool (x: BOOLEAN), NEW;
			(VAR wr: Writer) WriteByte (x: BYTE), NEW;
			(VAR wr: Writer) WriteChar (x: CHAR), NEW;
			(VAR wr: Writer) WriteInt (x: INTEGER), NEW;
			(VAR wr: Writer) WriteLong (x: LONGINT), NEW;
			(VAR wr: Writer) WriteReal (x: REAL), NEW;
			(VAR wr: Writer) WriteSChar (x: SHORTCHAR), NEW;
			(VAR wr: Writer) WriteSInt (x: SHORTINT), NEW;
			(VAR wr: Writer) WriteSReal (x: SHORTREAL), NEW;
			(VAR wr: Writer) WriteSString (IN x: ARRAY OF SHORTCHAR), NEW;
			(VAR wr: Writer) WriteSet (x: SET), NEW;
			(VAR wr: Writer) WriteString (IN x: ARRAY OF CHAR), NEW;
			(VAR wr: Writer) WriteXChar (x: CHAR), NEW;
			(VAR wr: Writer) WriteXInt (x: INTEGER), NEW;
			(VAR wr: Writer) WriteXReal (x: REAL), NEW;
			(VAR wr: Writer) WriteXString (IN x: ARRAY OF CHAR), NEW
		END;

END ToolsBigFMappers.

Re: Max file size

Posted: Wed Oct 19, 2016 8:43 pm
by cfbsoftware
Robert wrote: We needed to read / write third party .bin files; we didn't need full BlackBox View support for big files.

I am not in a position to publish the code, but the interfaces may be of interest here. There are no compatibility problems with the 32-bit file size code.
That seems like a really neat and sensible solution to me.

Re: issue-#138 Support for files larger than 2Gb

Posted: Wed Oct 19, 2016 9:07 pm
by DGDanforth
My mail inbox is 2,865,152,000 bytes. That causes BB to trap when attempting
to open the file for the purpose of copying it to backup.
I have a temporary fix that allows me to continue to use my incremental backup program.
I moved my mail folder to C:\Mail which is not in my backup path.
When I desire to backup the mail I will do that as a separate drag and drop
to my backup device.

-Doug