Metadata-Version: 2.1
Name: extrainterpreters
Version: 0.2a1
Summary: Utilities for concurrent code using subinterpreters
Home-page: https://github.com/jsbueno/extrainterpreters
Author: João S. O. Bueno
Author-email: gwidion@gmail.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Requires-Python: >=3.12-alpha
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: wheel

# extrainterpreters

Utilities to make use of new 
"interpreters"  module in Python 3.12

Status: alpha

Usage: for now one must build cpython 3.12 alpha from the main cpython branch


then, just import "extrainterpreters" and use the `Interpreter` class
as a wrapper to the subinterpreter represented by a single integer
handle created by the new `interpreters` module (that should be 
in Python stdlib as soon as Eric Snow polishes it)

with an `extrainterpreter.Interpreter` instance, one can call:
`inter.run(func_name, *args, **kwargs})` to run code there in the same
thread: the return value is just returned as long as it is pickleable.
The `inter.run_in_thread` with the same signature, will start a fresh
thead and call the target function there - the Interpreter instance  then should
be pooled in its "done()" method, and once done, the return value
will be available by calling  "result".

Also working is a threading like interface:

```
from extrainterpreters import Interpreter

...

    interp = Interpreter(target=myfunc, args=(...,))
    interp.start()
    ...
    interp.join()

    # bonus: we have return values!

    print(interp.result()
```

It will even work for methods defined on the REPL, so, just
give it a try.

There is a new class in the works which will be able to
run code in the sub-interpreter without depending of the
PEP554 "run_string" (once the setup is done). All processing in the sub-
interpreter will automatically take place in another thread,
and the roadmap is towards having a Future object and
an  `InterpreterPoolExecutor` compatible with the
existing executors in `concurrent.Futures`

A lot of things here are subject to change.
For now, all data exchange takes place in a custom memory
buffer with 10MB (it is a bytearray on  the parent interpreter,
and a memoryview for that on the target)

Data is passed to and back the interpreter using pickle - so,
no black magic trying to "kill an object here" and "make it be born there"
(not even if I intended to, as ctypes is not currently importable on 
sub-interpreters).

The good news is that `pickle` will take care of importing whatever is
needed on the other side, so that a function can run. 

# Roadmap

I plan to get this compatible with concurrent.futures.Executor, and have
an easy way to schedule a subinterpreter function execution as an async task.

Also, I should come up with a Queue object to pass data back and forth.
As for the data passing mechanism: we will use pickle, no doubt.

## When is Beta?

As soon as I get to a well thought way of creating the initial buffer
for comunication with a sub-interpreter.

# Architecture
The initial implementation used a pre-allocated mmaped file
to send/get data from other interpreters, until I found out the
mmap call in the sub-interpreter would build a separate mmap
object in a different memory region, and a lot of data copying
was taking place.

Currently, two C methods retrieve the address of an object implementing
the buffer interface, and create a "stand alone" memoryview instance
from the address alone. This address is passed to the subinterpreter
as a string during the setup stage,

