Page 3 of 6
Re: issue-#138 Support for files larger than 2Gb
Posted: Mon Oct 24, 2016 10:16 am
by Robert
Josef Templ wrote:Download test version from ...716.zip.
1 - There seems to be a lot of
unnecessary duplication in HostFiles64 - for example could PROCEDURE Error simply be exported from HostFiles?
2 - There seems to be some
undesirable duplication in HostFiles64. Why do we want two, possibly different ? - concurrent values for the variables MapParamString & ignoreAsk. Less confusing, I think, to only have / define them in HostFiles, but use them in both.
Re: issue-#138 Support for files larger than 2Gb
Posted: Mon Oct 24, 2016 8:45 pm
by DGDanforth
Robert wrote:Doug - What are you doing with multi-GByte files - are you just cataloguing, copying, moving, deleting them, or are you reading & writing their contents: individual bytes, strings, numbers, Views?
Josef - I hope to have time today to download, inspect, and comment on build 716. Does this build also contain the latest proposal on hierarchical menus?
Robert,
I have created an incremental backup program. I specify a root directory on C:\root and then conditionally copy those files (of any type) to the
backup directory E:\root. Conditional on whether the source file is more recent than the destination file. I do a BYTE by BYTE copy.
If the destination folder don't exist, I create it.
The only file larger than 2GB was my mail. I have since compressed that file and it is now less than 2GB.
I will have to create an artificial file > 2GB to test the copy process with Files64.
-Doug
Re: issue-#138 Support for files larger than 2Gb
Posted: Mon Oct 24, 2016 11:33 pm
by DGDanforth
I just copied file (XCode) of 4+GB with no problem.
Re: issue-#138 Support for files larger than 2Gb
Posted: Tue Oct 25, 2016 9:24 am
by Josef Templ
Robert wrote:
Josef - I hope to have time today to download, inspect, and comment on build 716. Does this build also contain the latest proposal on hierarchical menus?
If you look at
http://blackboxframework.org/unstable/, you see the builds
for the various branches. 716 is related to issue-#138. It does not contain the submenus.
The branches are independent from each other. The merge is only in master.
It depends on the time when a branch has been created what version of master
was used as the starting point.
- Josef
Re: issue-#138 Support for files larger than 2Gb
Posted: Tue Oct 25, 2016 10:42 am
by Robert
Josef Templ wrote: ad Stores.Reader/Writer:
There are multiple possibilities and it is the question where to stop with the duplication of functionality.
Here is a choice of alternatives:
- a subset of Stores64 covering only simple Reader and Writer procedures (seems to be simple)
- a complete Stores64 (don't know if it is complicated)
- more methods put into Files64.Reader/Writer
- move it all into an extension package (CPC, AppStore) instead of the standard distribution.
I suspect that Doug in not
interpreting the contents of his multi-GByte files, that Files64 is overkill for his needs, and that something like just using WinApi.CopyFile or WinApi.MoveFile is all he needs?
On the other hand, if we do go to the trouble to provide Files64 it is of very limited general use unless we also provide a compatible "Stores64".Reader & Writer.
I think your first two options are the only good ones.
Why do we need a
complete new Stores64; surely the only parts that we want are a new Reader & Writer?
A
complete new Reader & Writer might be the
best solution, but a partial Reader and Writer, as I illustrated earlier, may be a good compromise between utility and implementation effort. We can always add support for Multi-GByte Stores later when a real need for this functionality arises?
Re: issue-#138 Support for files larger than 2Gb
Posted: Tue Oct 25, 2016 12:14 pm
by Ivan Denisov
Let's make the new branch for BlackBox x64... it will be still x86 compiler, but with smooth transition to the 64bit pointers. The first step will be to change INTEGER type to LONGINT for the Files and Stores interface. For x64 BlackBox INTEGER still should be 32-bit. I found this plan better than to fit this Files64 and HostFiles64 in the 1.7.1 version.
SYSTEM.ADR(var) will return LONGINT and so on.
Re: issue-#138 Support for files larger than 2Gb
Posted: Tue Oct 25, 2016 12:27 pm
by Robert
Its a matter of timescale. I am not against having a 64-bit BlackBox version, but I just don't believe a complete, reliable, and tested version is going to be available very quickly.
In the meantime people may wish to access large files.
Re: issue-#138 Support for files larger than 2Gb
Posted: Tue Oct 25, 2016 1:56 pm
by Josef Templ
Robert wrote:Its a matter of timescale. I am not against having a 64-bit BlackBox version, but I just don't believe a complete, reliable, and tested version is going to be available very quickly.
In the meantime people may wish to access large files.
I agree with Robert. This is the pragmatic point of view.
64 bit file length support is not directly related with 64-bit BlackBox because it is supported by the Win32 API.
For 64-bit BlackBox we would need the Win64 API, wouldn't we? This is a much larger step
than supporting large files and supporting large files is already a large step if put into Files eventually.
- Josef
Re: issue-#138 Support for files larger than 2Gb
Posted: Tue Oct 25, 2016 6:02 pm
by Ivan Denisov
I prepared the demo version:
http://blackboxframework.org/unstable/m ... a1.723.zip
http://redmine.blackboxframework.org/pr ... b7dfe816ed
Josef Templ wrote:64 bit file length support is not directly related with 64-bit BlackBox because it is supported by the Win32 API.
For 64-bit BlackBox we would need the Win64 API, wouldn't we? This is a much larger step
than supporting large files and supporting large files is already a large step if put into Files eventually.
Yes, but this Files issue can be some step forward to change interfaces for 64-bit version.
Re: issue-#138 Support for files larger than 2Gb
Posted: Tue Oct 25, 2016 6:23 pm
by Ivan Denisov
I run this TestBig for the demo and it makes 3.4 Gb file.
Code: Select all
MODULE TestBig;
IMPORT Files;
PROCEDURE New*;
VAR f: Files.File; wr: Files.Writer; res: INTEGER; i, j, m: INTEGER;
BEGIN
f := Files.dir.New (Files.dir.This(""), FALSE);
wr := f.NewWriter(NIL);
FOR m := 0 TO 3200 DO
FOR j := 0 TO 1023 DO
FOR i := 0 TO 1022 DO
wr.WriteByte(67);
END;
wr.WriteByte(10);
END;
END;
f.Register("big", "txt", FALSE, res);
f.Close;
END New;
END TestBig.New