Metadata-Version: 2.4
Name: opensampl
Version: 1.2.0
Summary: Python tools for adding clock data to a timescale db.
Project-URL: Homepage, https://github.com/ORNL/OpenSAMPL
Project-URL: Repository, https://github.com/ORNL/OpenSAMPL
Project-URL: Documentation, https://ornl.github.io/OpenSAMPL
Project-URL: Changelog, https://github.com/ORNL/OpenSAMPL/blob/main/CHANGELOG.md
Author-email: Midgie MacFarland <macfarlandmj@ornl.gov>, Cory Watson <watsoncl1@ornl.gov>, Joshua Grant <grantjn@ornl.gov>
License: MIT License
        
        Copyright (c) 2025 Oak Ridge National Laboratory
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Framework :: Pydantic
Classifier: Framework :: Pydantic :: 2
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Telecommunications Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Communications
Classifier: Topic :: Database
Classifier: Topic :: Database :: Database Engines/Servers
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: Scientific/Engineering :: Visualization
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Networking :: Monitoring
Requires-Python: <3.15,>=3.10
Requires-Dist: allantools
Requires-Dist: astor
Requires-Dist: click<9,>=8.0.0
Requires-Dist: geoalchemy2==0.16.0
Requires-Dist: jinja2>=3.1.6
Requires-Dist: libcst
Requires-Dist: loguru<0.8,>=0.7.0
Requires-Dist: numpy<2,>=1.26.4
Requires-Dist: pandas<3,>=2.2.1
Requires-Dist: psycopg2-binary<3,>=2.9.0
Requires-Dist: pydanclick
Requires-Dist: pydantic-settings>=2.9.0
Requires-Dist: pydantic<3,>=2.10.3
Requires-Dist: python-dotenv
Requires-Dist: python-multipart<0.0.27,>=0.0.26
Requires-Dist: pytz~=2024.1
Requires-Dist: pyyaml<7,>=6.0.0
Requires-Dist: requests<3,>=2.31.0
Requires-Dist: sqlalchemy<3,>=2.0.39
Requires-Dist: tabulate
Requires-Dist: tqdm<5,>=4.66.2
Provides-Extra: backend
Requires-Dist: fastapi; extra == 'backend'
Requires-Dist: prometheus-client; extra == 'backend'
Requires-Dist: uvicorn; extra == 'backend'
Provides-Extra: collect
Requires-Dist: ntplib<0.5,>=0.4.0; extra == 'collect'
Requires-Dist: telnetlib3==2.0.4; extra == 'collect'
Provides-Extra: migrations
Requires-Dist: alembic; extra == 'migrations'
Description-Content-Type: text/markdown

# OpenSAMPL

<div align="center">
<!-- PyPI → version -->
<a href="https://pypi.org/project/opensampl/"><img src="https://img.shields.io/pypi/v/opensampl?logo=pypi" alt="PyPI"></a>
<!-- MIT licence -->
<a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-lightgrey.svg" alt="MIT licence"></a>
<!-- Supported Python versions (will show “missing” until you add the trove classifiers) -->
<a href="https://pypi.org/project/opensampl/"><img src="https://img.shields.io/pypi/pyversions/opensampl?logo=python" alt="python versions"></a>
<!-- Universal wheel? -->
<a href="https://pypi.org/project/opensampl/"><img src="https://img.shields.io/pypi/wheel/opensampl" alt="wheel"></a>
<!-- Monthly downloads -->
<a href="https://pypistats.org/packages/opensampl"><img src="https://img.shields.io/pypi/dm/opensampl?label=downloads%20%28month%29" alt="downloads per month"></a>
<!-- GitHub Actions CI -->
<a href="https://github.com/ORNL/OpenSAMPL/actions/workflows/publish.yml"><img src="https://github.com/ORNL/OpenSAMPL/actions/workflows/publish.yml/badge.svg" alt="PyPi Publishing"></a>
<a href="https://github.com/ORNL/OpenSAMPL/actions/workflows/lint.yml"><img src="https://github.com/ORNL/OpenSAMPL/actions/workflows/lint.yml/badge.svg" alt="ruff Formating and Linting"></a>
<a href="https://github.com/ORNL/OpenSAMPL/actions/workflows/tests.yml"><img src="https://github.com/ORNL/OpenSAMPL/actions/workflows/tests.yml/badge.svg" alt="PyTest Testing"></a>
<!-- Docs on GitHub Pages -->
<a href="https://ornl.github.io/OpenSAMPL/"><img src="https://img.shields.io/website?url=https%3A%2F%2Fornl.github.io%2FOpenSAMPL%2F&label=docs&logo=github" alt="docs"></a>
</div>


OpenSAMPL provides Python tools for collecting, loading, and visualizing clock data in a
TimescaleDB-backed synchronization analytics stack.
This project came out of [**CAST**](https://cast.ornl.gov), the **C**enter for
**A**lternative **S**ynchronization and **T**iming at Oak Ridge National Laboratory (ORNL).
The name OpenSAMPL stands for **O**pen **S**ynchronization **A**nalytics and
**M**onitoring **PL**atform.

The current codebase supports loading and analysis workflows for ADVA, Microchip TWST,
Microchip TP4100, and NTP-derived probe data. Visualization is provided through
[Grafana](https://grafana.com/), and the data is stored in
[TimescaleDB](https://www.timescale.com/), which is built on PostgreSQL.


### (**O**pen **S**ynchronization **A**nalytics and **M**onitoring **PL**atform)

Python tools for adding clock and timing data to a TimescaleDB database.

## Installation

1. Ensure you have Python 3.10 or higher installed.
2. Install the latest release:

```bash
pip install opensampl
```

### Development Setup

```bash
uv venv
uv sync --all-extras --dev
source .venv/bin/activate
```
This creates a virtual environment and installs the development dependencies.

### Environment Setup

The CLI reads configuration from environment variables or a local `.env` file.

When routing through a backend service:
```bash
ROUTE_TO_BACKEND=true
BACKEND_URL=http://localhost:8000

ARCHIVE_PATH=/path/to/archive
```

When connecting directly to PostgreSQL / TimescaleDB:
```bash
DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<database>
ARCHIVE_PATH=/path/to/archive
```

Use `opensampl config show` to inspect the current resolved configuration.

## CLI

The main CLI exposes `collect`, `config`, `create`, `init`, and `load`.
Use `opensampl --help` and `opensampl <command> --help` for current options.

If you plan to use the NTP, Microchip TWST, or Microchip TP4100 collectors, install the optional collection dependencies:

```bash
pip install "opensampl[collect]"
```

### Load Probe Data

Load data with the probe type name directly:

```bash
opensampl load ADVA path/to/file.txt.gz
opensampl load ADVA path/to/directory/
```

ADVA files bundle metadata and time-series data in a single file, so the split flags are
usually not needed.

```bash
opensampl load MicrochipTWST path/to/twst-output
opensampl load MicrochipTP4100 path/to/tp4100-output
```

NTP data is collected first and then loaded from the output directory:

```bash
opensampl collect ntp --mode remote --server pool.ntp.org --output-path ./ntp-out
opensampl load NTP ./ntp-out
```

Load options:

- `--metadata` / `-m`: load only probe metadata
- `--time-data` / `-t`: load only time-series data
- `--no-archive` / `-n`: skip archiving processed files
- `--archive-path` / `-a`: override the archive directory
- `--max-workers` / `-w`: set the worker count
- `--chunk-size` / `-c`: set the batch size for time-series inserts

### Load Direct Table Data

Load YAML or JSON directly into a table:

```bash
opensampl load table locations updated_location.yaml
```

Conflict handling is controlled by `--if-exists`:

- `update`: fill null fields in an existing row
- `error`: raise if the row exists
- `replace`: replace non-primary-key values
- `ignore`: skip existing rows

Example input:

```yaml
name: EPB Chattanooga
lat: 35.9311256
lon: -84.3292469
```

### View Configuration

```bash
opensampl config show
opensampl config show --explain
opensampl config show --var DATABASE_URL
```

### Set Configuration

```bash
opensampl config set VARIABLE_NAME value
```

## File Format Support

The loaders currently support:

- ADVA probe files named like
  `<ip_address>CLOCK_PROBE-<probe_id>-YYYY-MM-DD-HH-MM-SS.txt.gz>`
- Microchip TWST and TP4100 output produced by the collector tooling
- NTP snapshot output produced by `opensampl collect ntp`

Example ADVA file:
`10.0.0.121CLOCK_PROBE-1-1-2024-01-02-18-24-56.txt.gz`

# Contributing
We welcome contributions! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to get started.

# CAST Database Schema Documentation

## castdb.locations
Stores geographic locations with their coordinates and metadata. Supports both 2D and 3D point geometries.

```yaml
name: "Lab A"  # Unique name for the location
lat: 35.93  # Latitude coordinate
lon: -84.31  # Longitude coordinate
z: 100  # Optional elevation in meters
projection: 4326  # Optional SRID/projection (defaults to 4326/WGS84)
public: true  # Optional boolean for public visibility
```

## castdb.test_metadata
Tracks testing periods and experiments with start and end timestamps.

```yaml
name: "Holdover Test 1"  # Unique name for the test
start_date: "2024-01-01T00:00:00"  # Test start timestamp
end_date: "2024-01-07T00:00:00"  # Test end timestamp
```

## castdb.probe_metadata
Contains information about timing probes, including their network location and associated metadata. Insertion handled by `opensampl load probe`.

```yaml
probe_id: "1-1"  # Probe identifier
ip_address: "10.0.0.121"  # IP address of the probe
vendor: "ADVA"  # Vendor type
model: "OSA 5422"  # Model number
name: "GMC1"  # Human-readable name
public: true  # Optional boolean for public visibility
location_uiid: "123e4567-e89b-12d3-a456-426614174000"  # Optional reference to location
test_uiid: "123e4567-e89b-12d3-a456-426614174001"  # Optional reference to test
```

## castdb.probe_data
Time series data from probes, storing timestamps and measured values. Insertion handled by `opensampl load probe`.
```yaml
time: "2024-01-01T00:00:00"  # Timestamp of measurement
probe_uuid: "123e4567-e89b-12d3-a456-426614174000"  # Reference to probe
value: 1.234e-09  # Measured value
```

## castdb.adva_metadata
ADVA-specific configuration and status information for probes. Insertion handled by `opensampl load probe`.

```yaml
probe_uuid: "123e4567-e89b-12d3-a456-426614174000"  # Reference to probe
type: "Phase"  # Measurement type
start: "2024-01-01T00:00:00"  # Start timestamp
frequency: 1  # Sampling frequency
timemultiplier: 1  # Time multiplier
multiplier: 1  # Value multiplier
title: "ClockProbe1"  # Probe title
adva_probe: "ClockProbe"  # Probe type
adva_reference: "GPS"  # Reference source
adva_reference_expected_ql: "QL-NONE"  # Expected quality level
adva_source: "TimeClock"  # Source type
adva_direction: "NA"  # Direction
adva_version: 1.0  # Version number
adva_status: "RUNNING"  # Operating status
adva_mtie_mask: "G823-PDH"  # MTIE mask type
adva_mask_margin: 0  # Mask margin
```

## Notes

- All tables use UUIDs as primary keys which are automatically generated.
- Table relationships are maintained through UUID references
- Geographic coordinates use WGS84 projection (SRID 4326) by default
- Boolean fields (public) are optional and can be null
