Metadata-Version: 2.5
Name: qamule-scholar
Version: 0.1.0
Summary: macOS-only local SenseVoice model bootstrap for QAMule.
Classifier: Operating System :: MacOS :: MacOS X
Requires-Python: >=3.11
Requires-Dist: adbutils>=2.12.0
Requires-Dist: modelscope>=1.30.0
Requires-Dist: numpy>=2.3.0
Requires-Dist: sherpa-onnx-core==1.13.6
Requires-Dist: sherpa-onnx>=1.13.6
Requires-Dist: sounddevice>=0.5.6
Requires-Dist: uiautomator2>=3.7.0
Description-Content-Type: text/markdown

# qamule-scholar

`qamule-scholar` prepares the local SenseVoice model and starts QAMule
Scholar's local Android projection workspace.

## Platform

This package is supported only on macOS. Its build hook rejects non-macOS
builds and its published wheel is tagged for macOS. It requires Python 3.11 or
newer.

## Install

Install it from the workspace during development:

```sh
uv sync --package qamule-scholar
```

## Local model

Run the workspace with:

```sh
uv run qamule-scholar
```

The command checks the default ModelScope cache before starting the local web
service. If the supported int8 SenseVoice ONNX snapshot is absent, it asks for a
`y` or `n` confirmation, downloads `poloniumrock/SenseVoiceSmallOnnx` through
ModelScope, validates `model.int8.onnx` and `tokens.txt`, then opens the browser
workspace after the service is listening.

To start the local service without opening a browser, use `-q`:

```sh
uv run qamule-scholar -q
```

## Android projection

Scholar includes a local Android screen projection workspace. It packages the
device-side DEX JAR and exposes the workspace through a loopback web page. The
default URL is `http://127.0.0.1:26828/`. Use `--port` to choose another local
port:

```sh
uv run qamule-scholar --port 28080
```

By default, Scholar does not pass a device server port to `uiautomator2`. Declare
one explicitly when needed:

```sh
uv run qamule-scholar --device-port 26831
```

The page lists ADB-authorized devices as direct-select cards; click a card to start
projection. Use the `+` entry to add more device streams, and disconnect each
stream from its own card. Device selection, connection, and disconnection are
not available as CLI commands. The browser must support WebCodecs H.264
decoding, such as a current Chrome or Edge build.

While a device is projected, browser screen gestures and the current-device
controls act directly on that device before recording starts. These direct
actions do not create recording artifacts; the device card and its current
controls remain locked until each action completes.

## Recording

Select **Start Recording** in the sidebar to create a local session. Scholar
starts microphone capture from the system-default input and transcribes detected
speech with the local SenseVoice model. The most recent recognized sentences
appear in the floating voice panel and are also written locally. If macOS denies
microphone access, device recording continues and the panel reports the reason.
Before the first recording, allow the application used to start Scholar in
**System Settings > Privacy & Security > Microphone**. This is usually
**Terminal**, **Visual Studio Code**, or the terminal application in which
`uv run qamule-scholar` was started. Also select a usable input in **System
Settings > Sound > Input**, then start a new recording.

For every device action while a session is active, Scholar saves a JPEG
screenshot, raw UI hierarchy XML, and a simplified UI hierarchy text file
*before* sending the action to the device. The text file follows `u2cli
dump-hierarchy` output: invisible and zero-area nodes are omitted, and trivial
single-child wrapper nodes are collapsed. It does not automatically capture a
second device state after that action. Actions performed outside an active
recording remain direct controls and create no artifacts.

Sessions are created below `./qamule-scholar` from the directory where the
command was run:

```text
qamule-scholar/
    recording-<UTC timestamp>-<id>/
        recording.json
        transcript.txt
        step_000/
            screenshot.jpg
            hierarchy.xml
            hierarchy.txt
        step_001/
            screenshot.jpg
            hierarchy.xml
            hierarchy.txt
```

`recording.json` is the complete session manifest, including devices, recorded
actions, and SenseVoice segments in one ordered `events` list. Events are
ordered using internal capture and recognition times before serialization, so
actions during or after a segment follow that voice event. `transcript.txt`
contains the same recognized speech in a concise line-oriented format. Each
action event records both hierarchy artifact paths as `hierarchy` and
`hierarchy_text`.

Each recorded action identifies its source device with `serial`; device labels
and the most recently observed `window_size` appear once in the top-level
`devices` list. Tap and swipe coordinates remain normalized in the inclusive
range `0` through `1`.

### Inspect a recorded tap

To find the smallest hierarchy node containing a recorded tap without opening
the full XML, run:

```sh
uv run qamule-scholar tap-node qamule-scholar/recording-<id> --step 0
```

`--step 0` selects `step_000`; omit it to inspect every recorded tap. The
command converts the recorded normalized coordinates to device pixels and
prints only concise `u2cli dump-hierarchy`-style node summaries, one per line.
By default it returns up to five nodes. Use `--depth N` to choose a different
limit: the smallest bounds match is first, then its ancestors. Trivial
single-child wrapper nodes are collapsed like `u2cli`; if the root is reached
first, it returns every available node:

```sh
uv run qamule-scholar tap-node qamule-scholar/recording-<id> --step 0 --depth 5
```

The bridge is bound to loopback. At startup it temporarily pushes
`agt-screen-server.jar`, creates an ADB forward to its Android abstract socket,
and runs the DEX with `app_process`. On shutdown it closes the local forward,
stops the device-side process, and removes the temporary JAR. Screen capture
uses `adbutils` against the local ADB server for this transport; device
interactions remain the responsibility of Scholar's `U2DeviceGateway`, which
delegates to the reusable `u2cli` service.
