issue-#138 Support for files larger than 2Gb

Merged to the master branch
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

issue-#138 Support for files larger than 2Gb

Post by DGDanforth »

In HostFiles we find

Code: Select all

		File = POINTER TO RECORD (Files.File)
			state: INTEGER;
			name: FullName;
			ref: WinApi.HANDLE;
			loc: Locator;
			swapper: INTEGER;	(* index into file table / next buffer to swap *)
			len: INTEGER;
			bufs: ARRAY nofbufs OF Buffer;
			t: LONGINT	(* time stamp of last file operation *)
		END;
The len: INTEGER restricts files to be less than 2,147,483,648 bytes.
My mail inbox is 2,865,152,000 bytes. That causes BB to trap when attempting
to open the file for the purpose of copying it to backup.

I request that we make len: LONGINT

Are there any negative consequences for doing that?
-Doug
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: Max file size

Post by Ivan Denisov »

This should be fixed! However that is not a bug. I have to move your request to features section.
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: Max file size

Post by Ivan Denisov »

Target version for this issue can be 1.8 because it will change many interfaces: Length() SetPos() etc.
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

Re: Max file size

Post by DGDanforth »

Ivan Denisov wrote:Target version for this issue can be 1.8 because it will change many interfaces: Length() SetPos() etc.
Right.
For example

Code: Select all

		FileInfo = POINTER TO RECORD 
			next: FileInfo;
			name: Name;
			length: INTEGER;
			type: Type;
			modified: RECORD 
				year, month, day, hour, minute, second: INTEGER
			END;
			attr: SET
		END;
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: Max file size

Post by Josef Templ »

Is the TRAP at a reasonable place?
I mean, is it possible to see from the TRAP that there is an overflow of the file length?

If not, then we should add a check in HostFiles.Old, where the length is initialized.

Code: Select all

				f.len := WinApi.GetFileSize(ref, j);
				ASSERT(j = 0);   (* >>> add this line *)
This would then be a bug fix again, not a feature.

Changing Files.File.Length) to LONGINT is not an option within 1.7.x.
It may be done in 1.8 but it will have some performance impact since LONGINT
on a 32-bit platform is slower than INTEGER.
It would be a natural choice on a 64-bit platform, of course.

What is more realistic, at least in 1.8, is a change of Files.FileInfo.length to LONGINT.
This is not performance critical and would allow one to check if the file length
is valid before opening the file.

A very practical option would be to add 64-bit modules, say LFiles and HostLFiles (or Files64 and HostFiles64).
For the purpose of file operations such as copy this would be sufficient. The rest of the system
would still use the 32-bit Files module. This could be done in 1.7.1, I guess.
It is very much like Math and SMath.

- Josef
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

Re: Max file size

Post by DGDanforth »

Is the TRAP at a reasonable place?
I mean, is it possible to see from the TRAP that there is an overflow of the file length?
One sees that the file length is negative.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: Max file size

Post by Josef Templ »

DGDanforth wrote:
Is the TRAP at a reasonable place?
I mean, is it possible to see from the TRAP that there is an overflow of the file length?
One sees that the file length is negative.
Well, you see this only if the length is between 2 and 4 GB.
With 5 GB it would be positiv again, but truncated to the low-order 32 bit.
This should really be checked in Old.

The more I think about it the more I like the variant with adding a Files64 (alias LFiles) module.
Actually, this is the only possibility for now to provide support for large files.
It would be straightforward to do, I believe, and we could then claim to support large files.

- Josef
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: Max file size

Post by Robert »

Josef Templ wrote:The more I think about it the more I like the variant with adding a Files64 (alias LFiles) module.
Actually, this is the only possibility for now to provide support for large files.
It would be straightforward to do, I believe, and we could then claim to support large files.
A long time ago - long before BlackBox became open source, we commissioned Oms to provide big file support.
We needed to read / write third party .bin files; we didn't need full BlackBox View support for big files.

I am not in a position to publish the code, but the interfaces may be of interest here. There are no compatibility problems with the 32-bit file size code.

Code: Select all

DEFINITION ToolsBigFiles;

	IMPORT Files;

	CONST
		archive = 3;
		ask = TRUE;
		dontAsk = FALSE;
		exclusive = FALSE;
		hidden = 1;
		readOnly = 0;
		shared = TRUE;
		stationery = 4;
		system = 2;

	TYPE
		Directory = POINTER TO ABSTRACT RECORD 
			(d: Directory) Delete (loc: Files.Locator; name: Files.Name), NEW, ABSTRACT;
			(d: Directory) FileList (loc: Files.Locator): FileInfo, NEW, ABSTRACT;
			(d: Directory) GetFileName (name: Files.Name; type: Files.Type; OUT filename: Files.Name), NEW, ABSTRACT;
			(d: Directory) LocList (loc: Files.Locator): Files.LocInfo, NEW, ABSTRACT;
			(d: Directory) New (loc: Files.Locator; ask: BOOLEAN): File, NEW, ABSTRACT;
			(d: Directory) Old (loc: Files.Locator; name: Files.Name; shared: BOOLEAN): File, NEW, ABSTRACT;
			(d: Directory) Rename (loc: Files.Locator; old, new: Files.Name; ask: BOOLEAN), NEW, ABSTRACT;
			(d: Directory) SameFile (loc0: Files.Locator; name0: Files.Name; loc1: Files.Locator; name1: Files.Name): BOOLEAN, NEW, ABSTRACT;
			(d: Directory) Temp (): File, NEW, ABSTRACT;
			(d: Directory) This (IN path: ARRAY OF CHAR): Files.Locator, NEW, ABSTRACT
		END;

		File = POINTER TO ABSTRACT RECORD 
			type-: Files.Type;
			(f: File) Close, NEW, ABSTRACT;
			(f: File) Flush, NEW, ABSTRACT;
			(f: File) InitType (type: Files.Type), NEW;
			(f: File) Length (): LONGINT, NEW, ABSTRACT;
			(f: File) NewReader (old: Reader): Reader, NEW, ABSTRACT;
			(f: File) NewWriter (old: Writer): Writer, NEW, ABSTRACT;
			(f: File) Register (name: Files.Name; type: Files.Type; ask: BOOLEAN; OUT res: INTEGER), NEW, ABSTRACT
		END;

		FileInfo = POINTER TO RECORD 
			next: FileInfo;
			name: Files.Name;
			length: LONGINT;
			type: Files.Type;
			modified: RECORD 
				year, month, day, hour, minute, second: INTEGER
			END;
			attr: SET
		END;

		LocInfo = Files.LocInfo;

		Locator = Files.Locator;

		Reader = POINTER TO ABSTRACT RECORD 
			eof: BOOLEAN;
			(r: Reader) Base (): File, NEW, ABSTRACT;
			(r: Reader) Pos (): LONGINT, NEW, ABSTRACT;
			(r: Reader) ReadByte (OUT x: BYTE), NEW, ABSTRACT;
			(r: Reader) ReadBytes (VAR x: ARRAY OF BYTE; beg, len: INTEGER), NEW, ABSTRACT;
			(r: Reader) SetPos (pos: LONGINT), NEW, ABSTRACT
		END;

		Writer = POINTER TO ABSTRACT RECORD 
			(w: Writer) Base (): File, NEW, ABSTRACT;
			(w: Writer) Pos (): LONGINT, NEW, ABSTRACT;
			(w: Writer) SetPos (pos: LONGINT), NEW, ABSTRACT;
			(w: Writer) WriteByte (x: BYTE), NEW, ABSTRACT;
			(w: Writer) WriteBytes (IN x: ARRAY OF BYTE; beg, len: INTEGER), NEW, ABSTRACT
		END;

		Name = Files.Name;

		Type = Files.Type;

	VAR
		dir-: Directory;
		docType-: Files.Type;
		objType-: Files.Type;
		stdDir-: Directory;
		symType-: Files.Type;

	PROCEDURE SetDir (d: Directory);

END ToolsBigFiles.

Code: Select all

DEFINITION ToolsBigFMappers;

	IMPORT ToolsBigFiles;

	TYPE
		Reader = RECORD 
			rider-: ToolsBigFiles.Reader;
			(VAR rd: Reader) ConnectTo (f: ToolsBigFiles.File), NEW;
			(VAR rd: Reader) Pos (): LONGINT, NEW;
			(VAR rd: Reader) ReadBool (OUT x: BOOLEAN), NEW;
			(VAR rd: Reader) ReadByte (OUT x: BYTE), NEW;
			(VAR rd: Reader) ReadChar (OUT x: CHAR), NEW;
			(VAR rd: Reader) ReadInt (OUT x: INTEGER), NEW;
			(VAR rd: Reader) ReadLong (OUT x: LONGINT), NEW;
			(VAR rd: Reader) ReadReal (OUT x: REAL), NEW;
			(VAR rd: Reader) ReadSChar (OUT x: SHORTCHAR), NEW;
			(VAR rd: Reader) ReadSInt (OUT x: SHORTINT), NEW;
			(VAR rd: Reader) ReadSReal (OUT x: SHORTREAL), NEW;
			(VAR rd: Reader) ReadSString (OUT x: ARRAY OF SHORTCHAR), NEW;
			(VAR rd: Reader) ReadSet (OUT x: SET), NEW;
			(VAR rd: Reader) ReadString (OUT x: ARRAY OF CHAR), NEW;
			(VAR rd: Reader) ReadXChar (OUT x: CHAR), NEW;
			(VAR rd: Reader) ReadXInt (OUT x: INTEGER), NEW;
			(VAR rd: Reader) ReadXReal (OUT x: REAL), NEW;
			(VAR rd: Reader) ReadXString (OUT x: ARRAY OF CHAR), NEW;
			(VAR rd: Reader) SetPos (pos: LONGINT), NEW
		END;

		Writer = RECORD 
			rider-: ToolsBigFiles.Writer;
			(VAR wr: Writer) ConnectTo (f: ToolsBigFiles.File), NEW;
			(VAR wr: Writer) Pos (): LONGINT, NEW;
			(VAR wr: Writer) SetPos (pos: LONGINT), NEW;
			(VAR wr: Writer) WriteBool (x: BOOLEAN), NEW;
			(VAR wr: Writer) WriteByte (x: BYTE), NEW;
			(VAR wr: Writer) WriteChar (x: CHAR), NEW;
			(VAR wr: Writer) WriteInt (x: INTEGER), NEW;
			(VAR wr: Writer) WriteLong (x: LONGINT), NEW;
			(VAR wr: Writer) WriteReal (x: REAL), NEW;
			(VAR wr: Writer) WriteSChar (x: SHORTCHAR), NEW;
			(VAR wr: Writer) WriteSInt (x: SHORTINT), NEW;
			(VAR wr: Writer) WriteSReal (x: SHORTREAL), NEW;
			(VAR wr: Writer) WriteSString (IN x: ARRAY OF SHORTCHAR), NEW;
			(VAR wr: Writer) WriteSet (x: SET), NEW;
			(VAR wr: Writer) WriteString (IN x: ARRAY OF CHAR), NEW;
			(VAR wr: Writer) WriteXChar (x: CHAR), NEW;
			(VAR wr: Writer) WriteXInt (x: INTEGER), NEW;
			(VAR wr: Writer) WriteXReal (x: REAL), NEW;
			(VAR wr: Writer) WriteXString (IN x: ARRAY OF CHAR), NEW
		END;

END ToolsBigFMappers.
cfbsoftware
Posts: 204
Joined: Wed Sep 18, 2013 10:06 pm
Contact:

Re: Max file size

Post by cfbsoftware »

Robert wrote: We needed to read / write third party .bin files; we didn't need full BlackBox View support for big files.

I am not in a position to publish the code, but the interfaces may be of interest here. There are no compatibility problems with the 32-bit file size code.
That seems like a really neat and sensible solution to me.
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

Re: issue-#138 Support for files larger than 2Gb

Post by DGDanforth »

My mail inbox is 2,865,152,000 bytes. That causes BB to trap when attempting
to open the file for the purpose of copying it to backup.
I have a temporary fix that allows me to continue to use my incremental backup program.
I moved my mail folder to C:\Mail which is not in my backup path.
When I desire to backup the mail I will do that as a separate drag and drop
to my backup device.

-Doug
Post Reply