issue-#138 Support for files larger than 2Gb

Merged to the master branch
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

Re: issue-#138 Support for files larger than 2Gb

Post by DGDanforth »

I am now using Master64 which is very nice but wonder how that will be integrated into
our next release (1.71 or 1.8)?
-Doug
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: issue-#138 Support for files larger than 2Gb

Post by Ivan Denisov »

It can not be included in 1.7.1 because it is binary incompatible with 1.7. For 1.8 it is debatable to include this. There are some pluses and minuses.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

Robert wrote:
Josef Templ wrote:Download test version from ...716.zip.
1 - There seems to be a lot of unnecessary duplication in HostFiles64 - for example could PROCEDURE Error simply be exported from HostFiles?
2 - There seems to be some undesirable duplication in HostFiles64. Why do we want two, possibly different ? - concurrent values for the variables MapParamString & ignoreAsk. Less confusing, I think, to only have / define them in HostFiles, but use them in both.
It is 99.x% code duplication, some can be removed easily some would require to export more stuff from HostFiles, etc.
I didn't want to change the HostFiles interface only for supporting a more or less temporary solution.
And this file is normally not loaded, so there is no overhead in most cases.

MapParamString is interesting. It is currently not set and if it is to be supported without a change in Dialog
it must be done by importing HostFiles from HostFiles64. And then it would also be possible to reuse ignoreAsk, Locator, etc.
It is also possible to remove appName, which simplifies Init.

This reduces the code duplication from 99% to about 90%.
The changes for 64-bit file length support are now marked with the comment (*64-bit*) because it is
no longer possible to see them from a text diff with HostFiles.

See diffs at http://redmine.blackboxframework.org/pr ... 775ffa538b.

- Josef
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Josef Templ wrote:This reduces the code duplication from 99% to about 90%.
Looks like a worthwhile simplification. I would probably go further (89 % ?). Why export, or even define, pathLen, FullName & Locator? Why not simply use the Files version directly?

Actually this represents a rather more than 1 % reduction of the Files64 interface.


And, of course, the big unanswered question regarding Stores Readers & Writers?
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: issue-#138 Support for files larger than 2Gb

Post by Ivan Denisov »

I do not like the idea to include this Files64 and HostFiles64 in 1.7.1... It can be an experimental extension distributed separately.
We can suggest an 64-bit oriented version of BlackBox with all the intrefaces (Stores, Files etc.) with LONGINT. Assuming, that some day it will be 64bit pointers also.
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

The diff for the Docu file says:
Module Files64 is identical to Files except ...
Fine, but it would be helpful to make "Files" an active link to \Docu\Files.odc.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

Robert wrote:
Josef Templ wrote:This reduces the code duplication from 99% to about 90%.
Looks like a worthwhile simplification. I would probably go further (89 % ?). Why export, or even define, pathLen, FullName & Locator? Why not simply use the Files version directly?

Actually this represents a rather more than 1 % reduction of the Files64 interface.


And, of course, the big unanswered question regarding Stores Readers & Writers?
It should be possible to switch from Files to Files64 by simply using an alias import of the form

IMPORT Files := Files64;

Therefore everything defined in Files must also be defined in Files64.
This is the reason why Files64 is as complete as Files.

ad Stores Readers & Writers:
This is currently not used by anybody. If it is required it can be added
or, maybe better, it can be copied from Stores on demand into your code and then you
would only copy the parts that you really need and you have full control over the binary file format of
the basic data types in your application.
This is basically a trivial copy-and-paste of some lines of code.
Since the rest of the system is mostly incompatible with Files64 it would not add much.
To summarize, for the time being, I would ignore this.

- Josef
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Josef Templ wrote:It should be possible to switch from Files to Files64 by simply using an alias import of the form
IMPORT Files := Files64
I found myself disagreeing strongly with everything in the previous post!

After thinking about it for a while my theory is that we have very different approaches: Josef thinks of Files64 as an alternative to Files, whereas I think of it as a complement to Files?

For brevity, two points:

1 - I think the suggested IMPORT alias is horrible. A human seeing "Files.###' will, incorrectly, think it is the Module Files. Whereas if the alias was "Kg69h := Files64", or anything, a human would think "I don't recognise that, I'll check the IMPORT list". (I know that this alias was suggested by someone else first!).

2 - We must have a Files64 compatible Stores. There is almost no use for any kind of files if you don't either read stuff from them, or write stuff to them, and Stores is the means of writing INTEGERs, REALs, SETs, strings, etc.

I agree with Ivan that the current Files64 suggestions are, in some ways, quite undesirable. I don't necessarily agree with his conclusion that we shouldn't include it, because all the alternatives I can think of also have undesirable aspects. But one alternative that maybe we should consider is to extend the current Files interface with two new methods: Pos64 & SetPos64, and to modify the existing Stores to use them. Existing code that uses Pos & SetPos will still work fine on small files as now, but code that simply uses Stores will have extended functionality with no (source code) changes.
Actually SetPos64 is unnecessary; changing the TYPE of SetPos's parameter to LONGINT would not require any client source code changes.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

Robert wrote:But one alternative that maybe we should consider is to extend the current Files interface with two new methods: Pos64 & SetPos64, and to modify the existing Stores to use them. Existing code that uses Pos & SetPos will still work fine on small files as now, but code that simply uses Stores will have extended functionality with no (source code) changes.
Actually SetPos64 is unnecessary; changing the TYPE of SetPos's parameter to LONGINT would not require any client source code changes.
Interesting approach, can you work out the details?
In order to preserve the current state of the issue, I would recommend to
create an alternative issue branch, for example issue-#138b.

- Josef
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Josef Templ wrote:Interesting approach, can you work out the details?
I could possibly try, but it will be a slow process. I won't start until I have some level of proficiency in using the development tools. I am making progress, but expect that will be several (many ?) weeks away.

Actually your response was not what I expected. I carefully said I thought this approach would maintain client source code compatibility, but expected the reaction that it was a bad / unacceptable idea due to binary code compatibility issues. This topic should be thought about, and the approach agreed, before much detailed implementation work is undertaken.
Post Reply