Metadata-Version: 2.4
Name: pytango-db
Version: 0.7.0
Summary: Fork of PyTango's internal sqlite based database device.
Author-email: MAX IV KITS Software <kits-sw@maxiv.lu.se>
License-Expression: LGPL-3.0-or-later
Project-URL: Repository, https://gitlab.com/tango-controls/incubator/pytango-db
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: pytango
Provides-Extra: tests
Requires-Dist: pytest; extra == "tests"
Dynamic: license-file

# PyTango DatabaseDS

This is a "fork" of some internal code from pytango, that runs a Tango database device server. This
code was originally written by Tiago Coutinho. It uses sqlite for storing data and therefore
doesn't have any dependencies beyond python and PyTango.

It's currently under heavy development, but is pretty usable. Please report any problems!

The repo also contains (the beginnings of) a test suite that runs on both this device and a C++ database, for compatibility checking.

For discussion of the future of this device, see https://gitlab.com/tango-controls/pytango/-/issues/626


## Usage

Create a python environment and:

    $ pip install pytango-db
    $ TANGO_HOST=localhost:11000 PyDatabaseds 2

To work from source, check out this repo, create a python environment, and

    $ pip install -e .
    $ TANGO_HOST=localhost:11000 PyDatabaseds 2

Now you should be able to use any normal Tango stuff like Jive etc, as long as you set `TANGO_HOST=localhost:11000`. You can pick any port, but 10000 might be taken if you
already have a Tango installation locally.

The sqlite database is saved in the current working directory. It's possible to have
any number of databases running on the same host, with different ports, but be careful
about sharing the same database file, it will likely cause problems. You can configure
where it is stored by setting the environment variable `PYTANGO_DATABASE_NAME`.
For a pure in-memory db, use `:memory:`.

For debugging, you can enable logging:

    $ TANGO_HOST=localhost:11000 PyDatabaseds --logging_level=2 2

## pytest fixture

This project also includes two pytest fixtures to start a database:

- `pytango_db`, a function-scoped fixture (database is started and stopped after each test)
- `session_pytango_db`, a session-scoped fixture (database is started only once and stopped at the end of the test session)

To use one of this fixture in your project, add `pytango-db` to your tests requirements.

These fixtures will start a database on a random port and return the used `TANGO_HOST` variable,
that has been set in the environment.
The database runs in memory.

WARNING: While you can mix both fixtures in your tests, be careful if you create a `DeviceProxy` as cpptango
is caching the database connection. You might point to the wrong database. Pass the full TRL including the `tango_host`
returned by the fixture.

### Custom server fixtures

It's common to use `pytango-db` to run your Tango devices in a realistic way for tests.

`run_device_server` is a helper function that provides a running device server with configuration.
It's intended to be used for creating custom fixtures, like this:

```python
from databaseds.fixtures import run_device_server

@pytest.fixture
def my_device(pytango_db):
    yield from run_device_server(
        server_instance="MyDevice/1",
        device_classes={
            MyDevice: {
                "test/mydevice/1": {
                    "properties": {"SomeProperty": ["hello"]}
                }
            }
        })
```
Tests using the `my_device` fixture will now always have the device running, and `my_device`
will contain a proxy to it.

Note that `MyDevice` here is the actual device class. When specified this way, the server will
be created from the class(es) specified. The class can also be specified as a string, in which
case we'll start the server by running `MyDevice`, which must be in the `$PATH`.

If your device is not installed or doesn't have an entry point corresponding to the server name,
you can supply the `commandline` argument to tell the function how to start it:

```python
@pytest.fixture
def my_device(pytango_db):
    yield from run_device_server(
        server_instance="MyDevice/1",
        device_classes={
            "MyDevice": {
                "test/mydevice/1": {}
            }
        },
        commandline="python mydevice.py 1 -v4",
    )
```

It's possible to use this helper directly, for example:
```python
for (proxy,) in run_device_server("MyDevice/1", {RixsAq: {"test/mydevice/1": {}}}):
    proxy.State()
```
For more documentation see the docstring and tests.

## Use cases

- Local Tango development. Easy setup, possibility of having several databases and switching between them, etc.
- Continuous integration pipelines. Simpler setup, no service depencencies.
- Small lab setups?


## Drawbacks

- Still buggy
- Likely lower performance than the C++ device
- No concurrency (for now)


## Tests

The test suite is implemented as two very simple tests, one runs agains the pytango database device (provided by a fixture) and the other assumes there is a C++ database server running at the configured TANGO_HOST.

The test are run multiple times, with data provided by a JSON "spec". The spec contains separate definitions for each test run, e.g.

```
    {
        "name": "DbAddDevice",
        "check": [
            {
                "command": "DbAddDevice",
                "argument": [
                    "Dummy/1",
                    "test/dummy/2",
                    "Dummy"
                ]
            },
            {
                "command": "DbGetDeviceInfo",
                "argument": "test/dummy/2",
                "result": [
                    [
                        0,
                        0
                    ],
                    [
                        "test/dummy/2",
                        "nada",
                        "0",
                        "Dummy/1",
                        "nada",
                        "?",
                        "?",
                        "Dummy"
                    ]
                ]
            }
        ],
        "teardown": [
            {
                "command": "DbDeleteDevice",
                "argument": "test/dummy/2"
            }
        ]
    },

```
Here, the test is named after "DbAddDevice" since this is the command that is being tested.

The "check" section contains several command runs that will be executed on the database device under test. "command" and "argument" describe what to run, and the optional "result" describes the expected outcome. It will be compared against the actual result, to determine if the test was successful. Multiple commands are run in sequence.

The "teardown" section is optional, and will be run after the "check" section. If any modification is made to the database, this should be implemented in order to restore the database as far as possible, so that the test doesn't interfere with any other tests. At least in the case of the C++ database, we can't wipe it completely between tests so make sure to remove anything that was added by the test.

**Protip**: getting the "teardown" right and not forgetting something in the database can be tricky. However you can check this by setting the env var `PYTANGO_DATABASE_NAME=/tmp/test.sqlite` when running the tests. That will cause the test database to be stored on disk instead of memory, and allows you to inspect the contents of the database afterwards, e.g. using `sqlite3`. Note that some tables such as `*_history` aren't possible to clean up via Tango.

There is another optional section, "setup" that is run before "check" and can be used in order to create prerequisites for the checks.

The test may also contain a "comment" field for extra information, and a boolean "skip" field which if `true` will cause the test to be skipped. At the time of writing, many command tests are still empty "stubs", set to be skipped.

If a test is failing on one of the implementations, it can be marked as `xfail` by adding the field `expected_to_fail_on`.
This field shall be a list where each entry is either:

- a backend string: `cppdb` or `pydb`
- a dict with: `backend` set to `cppdb` or `pydb`, and an optional boolean `strict`

Examples:

```json
"expected_to_fail_on": ["cppdb"]
```

```json
"expected_to_fail_on": [
    {
        "backend": "cppdb",
        "strict": false
    }
]
```

If `strict` is omitted, it defaults to `true`.
Don't forget to add a comment explaining why this test is expected to fail and/or a link to an issue.


### Check

The check section also has a few more features:

`result` is a list or a string, depending on the type returned by the command. Items in the list are usually strings, but sometimes it can be useful to match them in a less exact way, when we only care about the "format" of the item:

* `null` means we don't care what the item is, it matches anything.
* `{"regexp": "a.*"}` is an example of a *regular expression* matching any string starting with "a".
* `{"timestamp": "%Y-%m-%d"}` matches any date in ISO format e.g. "2024-10-01". The matcher uses ordinary *strptime* syntax.

Ordinary string values may contain special *template* syntax e.g. `{name}`. before matching, it will be substituted with the name of the current test. This is useful in order to name things uniquely, which simplifies tracking down cases of stuff being left in the database by mistake (which should be cleaned up in the "teardown" step). For now, only "name" is available.

`result_slice` takes a list of arguments, corresponding to the python `slice` builtin; "start", "stop" and "step". Each is an integer or `None`. This can be used to get only part of the result, in case it contains things we aren't interested in.
