issue-#138 Support for files larger than 2Gb

Merged to the master branch
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Josef Templ wrote:Download test version from ...716.zip.
1 - There seems to be a lot of unnecessary duplication in HostFiles64 - for example could PROCEDURE Error simply be exported from HostFiles?
2 - There seems to be some undesirable duplication in HostFiles64. Why do we want two, possibly different ? - concurrent values for the variables MapParamString & ignoreAsk. Less confusing, I think, to only have / define them in HostFiles, but use them in both.
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

Re: issue-#138 Support for files larger than 2Gb

Post by DGDanforth »

Robert wrote:Doug - What are you doing with multi-GByte files - are you just cataloguing, copying, moving, deleting them, or are you reading & writing their contents: individual bytes, strings, numbers, Views?

Josef - I hope to have time today to download, inspect, and comment on build 716. Does this build also contain the latest proposal on hierarchical menus?
Robert,
I have created an incremental backup program. I specify a root directory on C:\root and then conditionally copy those files (of any type) to the
backup directory E:\root. Conditional on whether the source file is more recent than the destination file. I do a BYTE by BYTE copy.
If the destination folder don't exist, I create it.

The only file larger than 2GB was my mail. I have since compressed that file and it is now less than 2GB.
I will have to create an artificial file > 2GB to test the copy process with Files64.

-Doug
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

Re: issue-#138 Support for files larger than 2Gb

Post by DGDanforth »

I just copied file (XCode) of 4+GB with no problem.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

Robert wrote: Josef - I hope to have time today to download, inspect, and comment on build 716. Does this build also contain the latest proposal on hierarchical menus?
If you look at http://blackboxframework.org/unstable/, you see the builds
for the various branches. 716 is related to issue-#138. It does not contain the submenus.
The branches are independent from each other. The merge is only in master.
It depends on the time when a branch has been created what version of master
was used as the starting point.

- Josef
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Josef Templ wrote: ad Stores.Reader/Writer:
There are multiple possibilities and it is the question where to stop with the duplication of functionality.
Here is a choice of alternatives:
- a subset of Stores64 covering only simple Reader and Writer procedures (seems to be simple)
- a complete Stores64 (don't know if it is complicated)
- more methods put into Files64.Reader/Writer
- move it all into an extension package (CPC, AppStore) instead of the standard distribution.
I suspect that Doug in not interpreting the contents of his multi-GByte files, that Files64 is overkill for his needs, and that something like just using WinApi.CopyFile or WinApi.MoveFile is all he needs?

On the other hand, if we do go to the trouble to provide Files64 it is of very limited general use unless we also provide a compatible "Stores64".Reader & Writer.

I think your first two options are the only good ones.

Why do we need a complete new Stores64; surely the only parts that we want are a new Reader & Writer?

A complete new Reader & Writer might be the best solution, but a partial Reader and Writer, as I illustrated earlier, may be a good compromise between utility and implementation effort. We can always add support for Multi-GByte Stores later when a real need for this functionality arises?
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: issue-#138 Support for files larger than 2Gb

Post by Ivan Denisov »

Let's make the new branch for BlackBox x64... it will be still x86 compiler, but with smooth transition to the 64bit pointers. The first step will be to change INTEGER type to LONGINT for the Files and Stores interface. For x64 BlackBox INTEGER still should be 32-bit. I found this plan better than to fit this Files64 and HostFiles64 in the 1.7.1 version.
SYSTEM.ADR(var) will return LONGINT and so on.
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Its a matter of timescale. I am not against having a 64-bit BlackBox version, but I just don't believe a complete, reliable, and tested version is going to be available very quickly.

In the meantime people may wish to access large files.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

Robert wrote:Its a matter of timescale. I am not against having a 64-bit BlackBox version, but I just don't believe a complete, reliable, and tested version is going to be available very quickly.

In the meantime people may wish to access large files.
I agree with Robert. This is the pragmatic point of view.

64 bit file length support is not directly related with 64-bit BlackBox because it is supported by the Win32 API.
For 64-bit BlackBox we would need the Win64 API, wouldn't we? This is a much larger step
than supporting large files and supporting large files is already a large step if put into Files eventually.

- Josef
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: issue-#138 Support for files larger than 2Gb

Post by Ivan Denisov »

I prepared the demo version:
http://blackboxframework.org/unstable/m ... a1.723.zip

http://redmine.blackboxframework.org/pr ... b7dfe816ed

Josef Templ wrote:64 bit file length support is not directly related with 64-bit BlackBox because it is supported by the Win32 API.
For 64-bit BlackBox we would need the Win64 API, wouldn't we? This is a much larger step
than supporting large files and supporting large files is already a large step if put into Files eventually.
Yes, but this Files issue can be some step forward to change interfaces for 64-bit version.
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: issue-#138 Support for files larger than 2Gb

Post by Ivan Denisov »

I run this TestBig for the demo and it makes 3.4 Gb file.

Code: Select all

MODULE TestBig;

IMPORT Files;

PROCEDURE New*;
VAR f: Files.File; wr: Files.Writer; res: INTEGER; i, j, m: INTEGER;
BEGIN
	f := Files.dir.New (Files.dir.This(""), FALSE);
	wr := f.NewWriter(NIL);
	FOR m := 0 TO 3200 DO
		FOR j := 0 TO 1023 DO
			FOR i := 0 TO 1022 DO
				wr.WriteByte(67);
			END;
			wr.WriteByte(10);
		END;
	END;
	f.Register("big", "txt", FALSE, res);
	f.Close;
END New;


END TestBig.New
Post Reply