BlackBox Framework Center

After finding the trick how to read/write normal Stores.Store objects from/to a Files64 file
I have added Stores64 now.
Note: It does not only support reading/writing of base types but also of Stores
and it does not introduce any incompatibilities.
However, due to the file format of Stores a single Store cannot exceed 2GB.
Nevertheless it is possible to create for example a database of texts with more than 2GB total size.

See diffs at https://redmine.blackboxframework.org/p ... cb362970b2.

- Josef

Josef Templ wrote:After finding the trick how to read/write normal Stores.Store objects from/to a Files64 file
I have added Stores64 now.
Note: It does not only support reading/writing of base types but also of Stores
and it does not introduce any incompatibilities.
However, due to the file format of Stores a single Store cannot exceed 2GB.
Nevertheless it is possible to create for example a database of texts with more than 2GB total size.

See diffs at https://redmine.blackboxframework.org/p ... cb362970b2.

- Josef

Good job.
More and more work is being done on large data sets (specifically Artificial Intelligence) and so
64bit processing is becoming more important.

This issue is ready for voting.

Summary: this issue adds module Files64 for large files and it adds
module Stores64 for reading and writing large files in the binary Stores format.
Stores64 also allows one to read and write a Stores.Store from/to a large file.
If I remember correctly, there was a consensus that Files64 is required.
There was an open discussion about adding Stores Reader/Writer operations to Files64.
The present solution provides the Stores Reader/Writer operations but not inside Files64 but
in a separate module Stores64.
Since module Files deals only with bytes and byte blocks Stores Reader/Writer operations do not belong
into Files64. Since reading and writing a Stores.Store is now also supported it is quite natural to
put them into a module named Stores64. Having the operations separated gives also sense if one considers
typical use cases such as serving large files in an http server or copying large files in a file backup program.
These deal with bytes and byte blocks only and Files64 provides everything required.
If Stores Reader/Writer operations are to be used one must be aware of the fact that they use a specific file
format for things like integers, strings, etc and this is expressed by importing Stores/Stores64.

- Josef

Josef Templ wrote:This issue is ready for voting.

A couple of minor comments:
1 - There seem to be two relevant diffs: Stores64 posted 3-May-2017 & Files64 posted 1-Nov-2016? The latter diff is confusing; the Mod is Host but the Docu is System.
Can we have a simpler diff like the Stores64 one?

2 - The Files64 Docu says "refer to Files", or words to that effect. Ok for the detailed descriptions, but I think it would be helpful to include the interface at the beginning as in most Docu files. Often the interface is adequate "reminder" documentation, and this would avoid a two-stage lookup.

As with Files there is an additional module HostFiles64 that provides the low-level implementation of Files64.
That's why there is a module Files64 in System and in Host.
Stores64 does not need such a split (same as for Stores).

The module interface has been added to the Files64 docu.

The complete diff is here: https://redmine.blackboxframework.org/p ... 729f9239c9.

- Josef