issue-#138 Support for files larger than 2Gb

Merged to the master branch

issue-#138 Support for files larger than 2Gb

Postby DGDanforth » Wed Oct 19, 2016 1:55 am

In HostFiles we find
Code: Select all
      File = POINTER TO RECORD (Files.File)
         state: INTEGER;
         name: FullName;
         ref: WinApi.HANDLE;
         loc: Locator;
         swapper: INTEGER;   (* index into file table / next buffer to swap *)
         len: INTEGER;
         bufs: ARRAY nofbufs OF Buffer;
         t: LONGINT   (* time stamp of last file operation *)
      END;

The len: INTEGER restricts files to be less than 2,147,483,648 bytes.
My mail inbox is 2,865,152,000 bytes. That causes BB to trap when attempting
to open the file for the purpose of copying it to backup.

I request that we make len: LONGINT

Are there any negative consequences for doing that?
-Doug
User avatar
DGDanforth
 
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA

Re: Max file size

Postby Ivan Denisov » Wed Oct 19, 2016 1:57 am

This should be fixed! However that is not a bug. I have to move your request to features section.
Ivan Denisov
 
Posts: 1698
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: Max file size

Postby Ivan Denisov » Wed Oct 19, 2016 2:02 am

Target version for this issue can be 1.8 because it will change many interfaces: Length() SetPos() etc.
Ivan Denisov
 
Posts: 1698
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: Max file size

Postby DGDanforth » Wed Oct 19, 2016 2:32 am

Ivan Denisov wrote:Target version for this issue can be 1.8 because it will change many interfaces: Length() SetPos() etc.

Right.
For example
Code: Select all
      FileInfo = POINTER TO RECORD
         next: FileInfo;
         name: Name;
         length: INTEGER;
         type: Type;
         modified: RECORD
            year, month, day, hour, minute, second: INTEGER
         END;
         attr: SET
      END;
User avatar
DGDanforth
 
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA

Re: Max file size

Postby Josef Templ » Wed Oct 19, 2016 4:56 am

Is the TRAP at a reasonable place?
I mean, is it possible to see from the TRAP that there is an overflow of the file length?

If not, then we should add a check in HostFiles.Old, where the length is initialized.
Code: Select all
            f.len := WinApi.GetFileSize(ref, j);
            ASSERT(j = 0);   (* >>> add this line *)

This would then be a bug fix again, not a feature.

Changing Files.File.Length) to LONGINT is not an option within 1.7.x.
It may be done in 1.8 but it will have some performance impact since LONGINT
on a 32-bit platform is slower than INTEGER.
It would be a natural choice on a 64-bit platform, of course.

What is more realistic, at least in 1.8, is a change of Files.FileInfo.length to LONGINT.
This is not performance critical and would allow one to check if the file length
is valid before opening the file.

A very practical option would be to add 64-bit modules, say LFiles and HostLFiles (or Files64 and HostFiles64).
For the purpose of file operations such as copy this would be sufficient. The rest of the system
would still use the 32-bit Files module. This could be done in 1.7.1, I guess.
It is very much like Math and SMath.

- Josef
User avatar
Josef Templ
 
Posts: 2038
Joined: Tue Sep 17, 2013 6:50 am

Re: Max file size

Postby DGDanforth » Wed Oct 19, 2016 5:32 am

Is the TRAP at a reasonable place?
I mean, is it possible to see from the TRAP that there is an overflow of the file length?

One sees that the file length is negative.
User avatar
DGDanforth
 
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA

Re: Max file size

Postby Josef Templ » Wed Oct 19, 2016 9:50 am

DGDanforth wrote:
Is the TRAP at a reasonable place?
I mean, is it possible to see from the TRAP that there is an overflow of the file length?

One sees that the file length is negative.


Well, you see this only if the length is between 2 and 4 GB.
With 5 GB it would be positiv again, but truncated to the low-order 32 bit.
This should really be checked in Old.

The more I think about it the more I like the variant with adding a Files64 (alias LFiles) module.
Actually, this is the only possibility for now to provide support for large files.
It would be straightforward to do, I believe, and we could then claim to support large files.

- Josef
User avatar
Josef Templ
 
Posts: 2038
Joined: Tue Sep 17, 2013 6:50 am

Re: Max file size

Postby Robert » Wed Oct 19, 2016 10:53 am

Josef Templ wrote:The more I think about it the more I like the variant with adding a Files64 (alias LFiles) module.
Actually, this is the only possibility for now to provide support for large files.
It would be straightforward to do, I believe, and we could then claim to support large files.

A long time ago - long before BlackBox became open source, we commissioned Oms to provide big file support.
We needed to read / write third party .bin files; we didn't need full BlackBox View support for big files.

I am not in a position to publish the code, but the interfaces may be of interest here. There are no compatibility problems with the 32-bit file size code.
Code: Select all
DEFINITION ToolsBigFiles;

   IMPORT Files;

   CONST
      archive = 3;
      ask = TRUE;
      dontAsk = FALSE;
      exclusive = FALSE;
      hidden = 1;
      readOnly = 0;
      shared = TRUE;
      stationery = 4;
      system = 2;

   TYPE
      Directory = POINTER TO ABSTRACT RECORD
         (d: Directory) Delete (loc: Files.Locator; name: Files.Name), NEW, ABSTRACT;
         (d: Directory) FileList (loc: Files.Locator): FileInfo, NEW, ABSTRACT;
         (d: Directory) GetFileName (name: Files.Name; type: Files.Type; OUT filename: Files.Name), NEW, ABSTRACT;
         (d: Directory) LocList (loc: Files.Locator): Files.LocInfo, NEW, ABSTRACT;
         (d: Directory) New (loc: Files.Locator; ask: BOOLEAN): File, NEW, ABSTRACT;
         (d: Directory) Old (loc: Files.Locator; name: Files.Name; shared: BOOLEAN): File, NEW, ABSTRACT;
         (d: Directory) Rename (loc: Files.Locator; old, new: Files.Name; ask: BOOLEAN), NEW, ABSTRACT;
         (d: Directory) SameFile (loc0: Files.Locator; name0: Files.Name; loc1: Files.Locator; name1: Files.Name): BOOLEAN, NEW, ABSTRACT;
         (d: Directory) Temp (): File, NEW, ABSTRACT;
         (d: Directory) This (IN path: ARRAY OF CHAR): Files.Locator, NEW, ABSTRACT
      END;

      File = POINTER TO ABSTRACT RECORD
         type-: Files.Type;
         (f: File) Close, NEW, ABSTRACT;
         (f: File) Flush, NEW, ABSTRACT;
         (f: File) InitType (type: Files.Type), NEW;
         (f: File) Length (): LONGINT, NEW, ABSTRACT;
         (f: File) NewReader (old: Reader): Reader, NEW, ABSTRACT;
         (f: File) NewWriter (old: Writer): Writer, NEW, ABSTRACT;
         (f: File) Register (name: Files.Name; type: Files.Type; ask: BOOLEAN; OUT res: INTEGER), NEW, ABSTRACT
      END;

      FileInfo = POINTER TO RECORD
         next: FileInfo;
         name: Files.Name;
         length: LONGINT;
         type: Files.Type;
         modified: RECORD
            year, month, day, hour, minute, second: INTEGER
         END;
         attr: SET
      END;

      LocInfo = Files.LocInfo;

      Locator = Files.Locator;

      Reader = POINTER TO ABSTRACT RECORD
         eof: BOOLEAN;
         (r: Reader) Base (): File, NEW, ABSTRACT;
         (r: Reader) Pos (): LONGINT, NEW, ABSTRACT;
         (r: Reader) ReadByte (OUT x: BYTE), NEW, ABSTRACT;
         (r: Reader) ReadBytes (VAR x: ARRAY OF BYTE; beg, len: INTEGER), NEW, ABSTRACT;
         (r: Reader) SetPos (pos: LONGINT), NEW, ABSTRACT
      END;

      Writer = POINTER TO ABSTRACT RECORD
         (w: Writer) Base (): File, NEW, ABSTRACT;
         (w: Writer) Pos (): LONGINT, NEW, ABSTRACT;
         (w: Writer) SetPos (pos: LONGINT), NEW, ABSTRACT;
         (w: Writer) WriteByte (x: BYTE), NEW, ABSTRACT;
         (w: Writer) WriteBytes (IN x: ARRAY OF BYTE; beg, len: INTEGER), NEW, ABSTRACT
      END;

      Name = Files.Name;

      Type = Files.Type;

   VAR
      dir-: Directory;
      docType-: Files.Type;
      objType-: Files.Type;
      stdDir-: Directory;
      symType-: Files.Type;

   PROCEDURE SetDir (d: Directory);

END ToolsBigFiles.

Code: Select all
DEFINITION ToolsBigFMappers;

   IMPORT ToolsBigFiles;

   TYPE
      Reader = RECORD
         rider-: ToolsBigFiles.Reader;
         (VAR rd: Reader) ConnectTo (f: ToolsBigFiles.File), NEW;
         (VAR rd: Reader) Pos (): LONGINT, NEW;
         (VAR rd: Reader) ReadBool (OUT x: BOOLEAN), NEW;
         (VAR rd: Reader) ReadByte (OUT x: BYTE), NEW;
         (VAR rd: Reader) ReadChar (OUT x: CHAR), NEW;
         (VAR rd: Reader) ReadInt (OUT x: INTEGER), NEW;
         (VAR rd: Reader) ReadLong (OUT x: LONGINT), NEW;
         (VAR rd: Reader) ReadReal (OUT x: REAL), NEW;
         (VAR rd: Reader) ReadSChar (OUT x: SHORTCHAR), NEW;
         (VAR rd: Reader) ReadSInt (OUT x: SHORTINT), NEW;
         (VAR rd: Reader) ReadSReal (OUT x: SHORTREAL), NEW;
         (VAR rd: Reader) ReadSString (OUT x: ARRAY OF SHORTCHAR), NEW;
         (VAR rd: Reader) ReadSet (OUT x: SET), NEW;
         (VAR rd: Reader) ReadString (OUT x: ARRAY OF CHAR), NEW;
         (VAR rd: Reader) ReadXChar (OUT x: CHAR), NEW;
         (VAR rd: Reader) ReadXInt (OUT x: INTEGER), NEW;
         (VAR rd: Reader) ReadXReal (OUT x: REAL), NEW;
         (VAR rd: Reader) ReadXString (OUT x: ARRAY OF CHAR), NEW;
         (VAR rd: Reader) SetPos (pos: LONGINT), NEW
      END;

      Writer = RECORD
         rider-: ToolsBigFiles.Writer;
         (VAR wr: Writer) ConnectTo (f: ToolsBigFiles.File), NEW;
         (VAR wr: Writer) Pos (): LONGINT, NEW;
         (VAR wr: Writer) SetPos (pos: LONGINT), NEW;
         (VAR wr: Writer) WriteBool (x: BOOLEAN), NEW;
         (VAR wr: Writer) WriteByte (x: BYTE), NEW;
         (VAR wr: Writer) WriteChar (x: CHAR), NEW;
         (VAR wr: Writer) WriteInt (x: INTEGER), NEW;
         (VAR wr: Writer) WriteLong (x: LONGINT), NEW;
         (VAR wr: Writer) WriteReal (x: REAL), NEW;
         (VAR wr: Writer) WriteSChar (x: SHORTCHAR), NEW;
         (VAR wr: Writer) WriteSInt (x: SHORTINT), NEW;
         (VAR wr: Writer) WriteSReal (x: SHORTREAL), NEW;
         (VAR wr: Writer) WriteSString (IN x: ARRAY OF SHORTCHAR), NEW;
         (VAR wr: Writer) WriteSet (x: SET), NEW;
         (VAR wr: Writer) WriteString (IN x: ARRAY OF CHAR), NEW;
         (VAR wr: Writer) WriteXChar (x: CHAR), NEW;
         (VAR wr: Writer) WriteXInt (x: INTEGER), NEW;
         (VAR wr: Writer) WriteXReal (x: REAL), NEW;
         (VAR wr: Writer) WriteXString (IN x: ARRAY OF CHAR), NEW
      END;

END ToolsBigFMappers.
User avatar
Robert
 
Posts: 1023
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: Max file size

Postby cfbsoftware » Wed Oct 19, 2016 8:43 pm

Robert wrote:We needed to read / write third party .bin files; we didn't need full BlackBox View support for big files.

I am not in a position to publish the code, but the interfaces may be of interest here. There are no compatibility problems with the 32-bit file size code.

That seems like a really neat and sensible solution to me.
cfbsoftware
 
Posts: 204
Joined: Wed Sep 18, 2013 10:06 pm

Re: issue-#138 Support for files larger than 2Gb

Postby DGDanforth » Wed Oct 19, 2016 9:07 pm

My mail inbox is 2,865,152,000 bytes. That causes BB to trap when attempting
to open the file for the purpose of copying it to backup.


I have a temporary fix that allows me to continue to use my incremental backup program.
I moved my mail folder to C:\Mail which is not in my backup path.
When I desire to backup the mail I will do that as a separate drag and drop
to my backup device.

-Doug
User avatar
DGDanforth
 
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA

Next

Return to Resolved (Features)

Who is online

Users browsing this forum: No registered users and 1 guest

cron