issue-#156 adding Coroutines to BlackBox

Merged to the master branch
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#156 adding Coroutines to BlackBox

Post by Josef Templ »

I am now asking for a vote about including the coroutine support as in
issue branch issue-#156.

For the diffs see ... oops "An error occurred when trying to access the repository: Cannot allocate memory - 'git' --version --no-color"

Chairman, feel free to split the vote into any number of sub votes if you think
it would help finding a decision.

Let me summarize the reason and rationale for the proposal.

The short form:
If you don't know what a coroutine is, I can only ask you to trust me.
This is a starting point for a new feature that fits well with the BlackBox design.
It can be refined later if experience shows new requirements or problems.
Without starting it, we will never get experience for refining it.
There are no incompatibilities whatsoever.
This is a pure add-on that is not activated if not used.
Kernel size is increased from 31001 to 32902 bytes, i.e. by 1901 bytes.

The long form:
There seems to be a need by the community for supporting coroutines in
BlackBox as can be seen by the existence of the CPC package named Co_.
I can only confirm the need for coroutine support by my own experience.
It is certainly not used every day but sometimes it can be very helpful, for example
when dealing with long running background tasks or with discrete event simulation.

It is not possible to add coroutine support without Kernel modifications.
Therefore Kernel support for coroutines has been added.
Coroutine support has been added such that it cannot introduce any instabilities
because the new code is only activated when coroutines are really used.
This trades a small amount of code replication for guaranteed stability.
Coroutine support is encapsulated in Kernel because Kernel maintains the core
runtime data structures including the heap, the main stack, the trap handling, the stack overflow detection,
the garbage collection etc. and coroutines are part of that. Any split would break that abstraction
and introduce more complications than simplifications. BlackBox has been designed by ominc
from the beginning as a portable system with support for Windows and MacOS.
Some base modules have been split and implemented in a separate Host module.
Kernel is not among them. I trust the design of ominc in this respect and don't see any
chance and need to 'improve' this.
(personal estimation: with the amount of time spent into looking at alternative approaches,
coroutines could have been ported to 3 platforms including maintenance for 10 years.)

The Kernel's coroutine support makes it possible to implement the Co_ package
without crashing BlackBox occasionally. There have been doubts that this would be possible
but it has been shown that it works by simply replacing the WinApi fiber calls by
the corresponding Kernel calls 1:1.

The Co_ package is designed with specific kinds of applications in mind. It is not
a general purpose coroutine library. Therefore a general purpose module named 'Coroutines'
has been added. In addition to basic general purpose coroutines it supports
two special coroutine patterns named Task and Iterator. These are common patterns that
can appear in many applications. They are special in the sense that they have
additional methods (Transfer wrappers) with checked pre- and post conditions that
restrict their usage to the intended purpose.
Further application specific patterns can be implemented on top of Coroutines.
It would for example be possible to create a library specifically
targeted at discrete event simulation or similar.

ObxCoroutines has been added for showing by example how plain coroutines,
Tasks and Iterators can be used.

- Josef
Zinn
Posts: 476
Joined: Tue Mar 25, 2014 5:56 pm
Location: Frankfurt am Main
Contact:

Re: issue-#156 adding Coroutines to BlackBox

Post by Zinn »

Josef Templ wrote: For the diffs see ... oops "An error occurred when trying to access the repository: Cannot allocate memory - 'git' --version --no-color"
There is a file blackbox-1.7.1-a1.856-buildlog.htlm but no 856.zip and no 856-setup.exe file in the unstable/issue#156 directory.
-Helmut

P.S. Thank you for your great work.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#156 adding Coroutines to BlackBox

Post by Josef Templ »

Zinn wrote: There is a file blackbox-1.7.1-a1.856-buildlog.htlm but no 856.zip and no 856-setup.exe file in the unstable/issue#156 directory.
-Helmut
Something seems to be wrong with the Edis host.
May be there is a problem with the X-Server or xvfb command.
Also in Redmine I get an "out of memory" error when I try to open the Repository.

Ivan, can you please check this?

Thanks

- Josef
Ivan Denisov
Posts: 1700
Joined: Tue Sep 17, 2013 12:21 am
Location: Russia

Re: issue-#156 adding Coroutines to BlackBox

Post by Ivan Denisov »

We need to make preliminary voting to choose realization composition first.

I will check what's happening with the server after finish my work.
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#156 adding Coroutines to BlackBox

Post by Josef Templ »

The complete diffs for this issue are now available, see https://redmine.blackboxframework.org/p ... 97faa7e691.

The latest build is http://blackboxframework.org/unstable/i ... a1.857.zip.

Also the Redmine Repository function works again.
Thanks to Ivan for fixing the Edis host problems.

- Josef
User avatar
DGDanforth
Posts: 1061
Joined: Tue Sep 17, 2013 1:16 am
Location: Palo Alto, California, USA
Contact:

Re: issue-#156 adding Coroutines to BlackBox

Post by DGDanforth »

Question:
How fast can a Coroutine run?
By that I mean that Service comes up for air 20 times a second.
How often can a Coroutine come up for air?
Is the granularity the same as for Services or is it finer?
-Doug
User avatar
Robert
Posts: 1024
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: issue-#156 adding Coroutines to BlackBox

Post by Robert »

Josef Templ wrote:The solution is based on Windows fibers...
The Wikipedia coroutines page (https://en.wikipedia.org/wiki/Coroutine) says:
Consequently, the use of the fiber API to switch tasks is currently not a viable option in the .NET Framework.
Is there an issue here?
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#156 adding Coroutines to BlackBox

Post by Josef Templ »

DGDanforth wrote:Question:
How fast can a Coroutine run?
By that I mean that Service comes up for air 20 times a second.
How often can a Coroutine come up for air?
Is the granularity the same as for Services or is it finer?
-Doug
Plain coroutines are completely independent from Services.Action.
A Transfer takes about 200 - 300 nanoseconds, depending on the hardware.

Tasks use the Services.Action infrastructure and are bound to the same behavior
except for one optimization: Task.Sleep(0) is transferring to main only occasionally,
i.e. when a certain amount of time has passed. Otherwise it is a dummy call
and the coroutine resumes immediately. This delivers close to 100% cpu utilization
and is much faster as can be seen in the Primes example.

The minimum cycle time for a Services.Action is about 62 milliseconds in normal mode and
16 milliseconds in server mode. This is a limitation imposed by the so-called
Windows idle period, which is affected by switching to server mode.
The granularity of Services.Ticks() is always 16 milliseconds, at least on my PC.

- Josef
User avatar
Josef Templ
Posts: 2047
Joined: Tue Sep 17, 2013 6:50 am

Re: issue-#156 adding Coroutines to BlackBox

Post by Josef Templ »

Robert wrote:
Josef Templ wrote:The solution is based on Windows fibers...
The Wikipedia coroutines page (https://en.wikipedia.org/wiki/Coroutine) says:
Consequently, the use of the fiber API to switch tasks is currently not a viable option in the .NET Framework.
Is there an issue here?
BlackBox has nothing in common with the .NET CLR infrastructure.

- Josef
luowy
Posts: 234
Joined: Mon Oct 20, 2014 12:52 pm

Re: issue-#156 adding Coroutines to BlackBox

Post by luowy »

Josef,
1, shared stack coroutine is more valuable than independent stack one;
2,sperate scheduler from coroutine is needed,
Q: where the trap handle code for coroutine? what happen when coroutine traped?
Post Reply