issue-#138 Support for files larger than 2Gb

Merged to the master branch
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

Didn't you also say "Existing code that uses Pos & SetPos will still work fine on small files as now"?

This implies binary compatibility, doesn't it?
I was expecting a solution that maintains binary compatibility with existing code.
If does not maintain binary compatibility it is not an interesting approach.

- Josef
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: issue-#138 Support for files larger than 2Gb

Post by Ivan Denisov »

This will brake binary compatibility, so there should be separate project. I understand Roberts idea well and sharing it. I am already started this way with master64. I have to change Stores according that concept this weekend.
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Josef Templ wrote:Didn't you also say "Existing code that uses Pos & SetPos will still work fine on small files as now"?
I twice said "source code", including once in the very sentence you have partially quoted.

I don't fully understand the binary code issues. I suspect that this proposal may require other modules to be recompiled. I suspect that this has version numbering implications due to our version numbering conventions.

That is why I said the idea needs to be considered.

From implementation and interface viewpoints I think it is much nicer than the Files64 suggestion. But if it has version number implications that probably leads to timescale implications?
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

Robert wrote: I don't fully understand the binary code issues. I suspect that this proposal may require other modules to be recompiled. I suspect that this has version numbering implications due to our version numbering conventions.

...I twice said "source code", including once in the very sentence you have partially quoted.
This is why I said: "can you work out the details."

If we are talking about 1.7.1 we stay binary compatible. That's why Files64 exists as an add-on.
If we talk about 1.8 we can change the length in Files to LONGINT directly.
Then we don't need to think about tricky constructs to maintain
some level of compatibility. This is very simple.

Well, for me "existing code" is not the same as "existing source code".
It can also mean "existing binary code". And in the context of 1.7.1 it must
mean "existing binary code".

- Josef
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

Ivan Denisov wrote:This will brake binary compatibility, so there should be separate project. I understand Roberts idea well and sharing it. I am already started this way with master64. I have to change Stores according that concept this weekend.
What is master64 intended to show?
Do you want to show that the file length can be changed to LONGINT?
This is clear anyway but it cannot be done before 1.8.
So what is the point here?

- Josef
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Josef Templ wrote:If we are talking about 1.7.1 we stay binary compatible. ...
If we talk about 1.8 we can change the length in Files to LONGINT directly.
From a user's perspective there are three situations:
- Nothing is necessary
- Recompilation (only) is necessary
- (Minor ?) source code changes and recompilation are necessary.

It seems like we ignore the middle option, but I think it is a useful option as we want to make the last option a very rare event.
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: issue-#138 Support for files larger than 2Gb

Post by Ivan Denisov »

Josef Templ wrote:
Ivan Denisov wrote:This will brake binary compatibility, so there should be separate project. I understand Roberts idea well and sharing it. I am already started this way with master64. I have to change Stores according that concept this weekend.
What is master64 intended to show?
Do you want to show that the file length can be changed to LONGINT?
This is clear anyway but it cannot be done before 1.8.
So what is the point here?
I think, that there should be two master branches. One intended for 32-bit operations and another is intended for LONGINT as a preparation step for x64 platform.

So I suggest not include in 1.7.1 or 1.8 these Files64 and HostFiles64, but make special version of BlackBox for all the tasks related with large addresses.
This is demonstrated with master64: http://blackboxframework.org/unstable/master64
With git it will be easy to keep master and master64 synced, I can manage this synchronization.
Josef, now it includes you proposal for large files operations and test program demonstrates that it works well. Next step is to changes Stores operations.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#138 Support for files larger than 2Gb

Post by Josef Templ »

If I remember correctly the starting point was that Doug had a file backup
utility that didn't work with large files.
This is a rare but perfectly justified use case.

For this kind of utility it is sufficient to replace Files by Files64
in the utility source code and then it works almost unchanged.
The incompatibilities are marked by the compiler.
It is not important if his utility uses an alias import or not, by the way.
What I think is important is that he didn't have to learn anything
about Files64 or its interaction with Files but
he could reuse all the knowledge he had about Files and
apply it to Files64. And he succeeded in a very short amount of time.
This is a success story, I believe.

There was no need to read/write base types.
Everything is at the level of bytes and byte blocks.
For this kind of requirements Files64 can serve as a workaround of
the length limitation imposed by Files until BB 1.8.

For the regular user of BlackBox there is absolutely no need to
use large files. On the other side, if you want to create for
example a database server you will use your own
binary format for base types anyway. Strings will
be stored differently, integers may be big endian, etc, etc.
Every support beyond the byte level includes some speculation
about the intended usage and should be postponed for now.

The advantage of adding Files64 to 1.7.1 is that Doug can
implement his backup utility without the need to install an extension package
first. Extension packages are inconvenient, must be reinstalled
when you install a new BB version, depend on someone that
maintains them properly, may be out of sync with fixes we do in Files,
are not under version control, may disappear, etc.

I still regard it as the best pragmatic compromise.
Moving it to CPC or app store would simplify the distribution (slightly) but complicate it
for the user (largely). Ivan seems to search for examples to put in his app store.
But the app store, as it is now, only simplifies the uploading of extensions,
not the downloading. If he cared for the user experience, he would prefer to add it to 1.7.1.

- Josef
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#138 Support for files larger than 2Gb

Post by Robert »

Josef Templ wrote:I still regard it as the best pragmatic compromise.
I regard the current proposal as a rather unhappy compromise.

- Ivan does not want Files64 or Stores64 in 1.7.1.
- I think Files64 is pretty useless without Stores64 (or something that does the same job of reading and writing content) in 1.7.1.
- A simpler solution for Doug's need is a Copy or Move function. This might just be a 1 line wrapper around some WinApi call.

So this compromise serves little purpose for anyone.

I don't have a personal position here as I, and my colleagues, already have access to both Files64 & Stores64 equivalents, written by Oms. We have used them
on a daily basis for over 10 years. We commissioned them because we needed them both.

The detailed comments above about why Stores64 is unnecessary are (in our experience) simply wrong.
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: issue-#138 Support for files larger than 2Gb

Post by Ivan Denisov »

Josef Templ wrote:Ivan seems to search for examples to put in his app store.
Not in this case. I do not care where this can be published. We can just keep this two files here in the board or at the Community board.
Josef Templ wrote:But the app store, as it is now, only simplifies the uploading of extensions,
not the downloading. If he cared for the user experience, he would prefer to add it to 1.7.1.
I do not care about user experience for this case. I am caring about keeping BlackBox clean.
The reasons to add something to basic distribution should be more serious than just some use case. There should be some prospective thinking...

When you suggest to add disassembler I was thinking that it will be some rare used stuff. Then you explained that you have some foresight about development of 64bit version. This disassembler will help to develop new compilers. So that is serious argument to include something in basic distribution.

Here I am using the same arguments. If you want to solve one problem this can be separate extension. However we can solve more general problem and start switching all BlackBox to LONGINT.
Post Reply