issue-#138 Support for files larger than 2Gb

Merged to the master branch
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: Max file size

Post by Josef Templ »

cfbsoftware wrote:
Robert wrote: We needed to read / write third party .bin files; we didn't need full BlackBox View support for big files.

I am not in a position to publish the code, but the interfaces may be of interest here. There are no compatibility problems with the 32-bit file size code.
That seems like a really neat and sensible solution to me.
This would be very easy to do because the Win32 API supports 64 bit file length.
The high part is currently ignored. This is the only change, I think.

Together with a length check in Files.Old this would be an improvement.

- Josef
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

Re: issue-#138 Support for files larger than 2Gb

Post by DGDanforth »

Good. Agreed.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

A test version can be downloaded from http://blackboxframework.org/unstable/i ... a1.715.zip.

It has a new module Files64.
A length check has been added to Files.Directory.Old and when creating a FileInfo.

See the diffs at http://redmine.blackboxframework.org/pr ... 729f9239c9.

- Josef
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

Re: issue-#138 Support for files larger than 2Gb

Post by DGDanforth »

Problem

Code: Select all

		Views.Register(v, Views.dontAsk, loc
type is not identical with that of formal VAR-,IN-, or OUT-parameter

Code: Select all

, name, conv, res);

That means if one uses an alias
Files:=Files64
then ANY module in the framework that IMPORTs Files will not work.

That means code for Files64 and Files must be kept separate.

-Doug
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Josef Templ wrote:See the diffs at ...
1 - A trivial comment: I think we should omit copying the historical comments at the top that do not actually refer to the new file.
2 - A less trivial comment: Do we intend to also produce versions of the Stores.Reader & Writer objects that work with the new file type, similar to the second interface I published a few days ago?

I think there is a need to read/write structured types, such as REALs, even if there is no need to support general BlackBox Views of over 2 GBytes. With this limitation writing 64-bit sized file compatible Readers & Writers really is a simple cut-and-paste job.
That means code for Files64 and Files must be kept separate.
Yes.
luowy
Posts: 234
Joined: Mon Oct 20, 2014 12:52 pm

Re: issue-#138 Support for files larger than 2Gb

Post by luowy »

Robert,
can you post your ToolsBigFiles.Mod and ToolsBigMappers.Mod module files to public?
it is neat and suitable to read/write bigfile exactly,as Chris said.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

LocInfo, Locator, Name, Type need to be merged with Files, of course.
I should have read Robert's interface more carefully.

I need to think about the Reader/Writer issue.

- Josef
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

The Files64 types LocInfo, Locator, Name, and Type are now declared equal to the related types from Files.
See the diffs at http://redmine.blackboxframework.org/pr ... 579008183c.
Download test version from http://blackboxframework.org/unstable/i ... a1.716.zip.

ad Stores.Reader/Writer:
There are multiple possibilities and it is the question where to stop with the duplication of functionality.
Here is a choice of alternatives:
- a subset of Stores64 covering only simple Reader and Writer procedures (seems to be simple)
- a complete Stores64 (don't know if it is complicated)
- more methods put into Files64.Reader/Writer
- move it all into an extension package (CPC, AppStore) instead of the standard distribution.

There is also the aspect that Files64 must be considered a temporary
solution for 64-bit files support in 1.7.x.
In 1.8 there may (or should) be 64-bit support in Files directly,
which will make Files64 redundant.
Maybe it is a good idea to express this in the docu of Files64,
unless everything is moved to an extension package anyway.

- Josef
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

Re: issue-#138 Support for files larger than 2Gb

Post by DGDanforth »

Josef,
Much better!
Now I can do

Files:=Files64

Backup program works just fine.
(Yet to be tested on >2GB)

-Doug
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Doug - What are you doing with multi-GByte files - are you just cataloguing, copying, moving, deleting them, or are you reading & writing their contents: individual bytes, strings, numbers, Views?

Josef - I hope to have time today to download, inspect, and comment on build 716. Does this build also contain the latest proposal on hierarchical menus?
Post Reply