Metadata-Version: 2.4
Name: driftlessflip
Version: 0.15.0
Summary: Time-domain fluorescence lifetime photometry acquisition for PicoQuant TimeHARP 260
Author: Mohebi and Associates
License-Expression: MIT
Project-URL: Homepage, https://github.com/mohebi-n-associates/DriftlessFLIP
Project-URL: Repository, https://github.com/mohebi-n-associates/DriftlessFLIP
Project-URL: Issues, https://github.com/mohebi-n-associates/DriftlessFLIP/issues
Project-URL: Changelog, https://github.com/mohebi-n-associates/DriftlessFLIP/blob/main/WHATS_NEW.md
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: Microsoft :: Windows
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: h5py<4,>=3.11
Requires-Dist: jsonschema<5,>=4.23
Requires-Dist: numpy<3,>=2.0
Requires-Dist: pynwb<4,>=3.0
Requires-Dist: scipy<2,>=1.11
Requires-Dist: PySide6<7,>=6.7
Provides-Extra: dev
Requires-Dist: pytest<9,>=8.3; extra == "dev"
Provides-Extra: packaging
Requires-Dist: pyinstaller==6.22.0; extra == "packaging"
Dynamic: license-file

# DriftlessFLIP

A desktop application for **time-domain fluorescence lifetime photometry** with a PicoQuant
TimeHARP 260 PICO DUAL. It acquires TTTR photon streams, turns them into per-sample decay
curves, and derives intensity, mean photon emission time, and phasor coordinates — while
keeping the raw photon records so any of those numbers can be recomputed later.

It runs without hardware. A built-in simulator reproduces the board's record semantics, so
the whole workflow can be learned, tested, and demonstrated on any machine.

> **Ready for real experiments.** The acquisition path has been validated with a physical
> TimeHARP 260 across multiple bench sessions and real recordings, including a sustained
> high-volume run. The simulator and automated suite cover the same processing and storage
> path. Before collecting irreplaceable data on a new computer, board, detector, or timing
> setup, verify that exact hardware profile and follow the deployment checks in
> [`DESIGN_PLAN.md`](docs/DESIGN_PLAN.md) section 7.

---

## What it does

**Acquisition**

- Preview live signal without saving anything, to set discriminator levels and timing.
- Record for a fixed duration, or loop N repeats on a start-to-start interval.
- Start from software or from an external TTL trigger.
- Two detector channels, each with its own experiment-specific name such as `DA in PFC`.
- Four named TTL marker inputs, with automatic rising-to-falling pulse pairing.
- Free-text operator notes, timestamped from the moment you start typing.

**Analysis**

- Per-sample decay histograms, intensity, mean photon emission time, and first-harmonic
  phasor coordinates.
- Background subtraction from a separately recorded background, scaled for detector dead
  time so a busier detector is not over-corrected.
- Per-channel afterpulse correction across the complete decay curve.
- Single- and double-exponential fitting, convolved with a Gaussian instrument response
  and re-excited every laser period, with per-parameter fixed/floating control.
- Phasor plot with the universal semicircle and lifetime reference points.

**Data you can trust later**

- Bit-exact raw TTTR retained, so any derived quantity can be recomputed when an
  algorithm changes.
- Every recording writes raw HDF5, NWB, CSV, canonical metadata JSON, a run log, and a
  checksummed manifest, all cross-verified against each other.
- An interrupted recording can be recovered, reproducing the corrections the run was
  applying.
- Read-only import of supported legacy histogram recordings for comparison and analysis.

---

## Installation for end users

Download `DriftlessFLIP-<version>-Setup.exe` from the corresponding
[GitHub Release](https://github.com/mohebi-n-associates/DriftlessFLIP/releases), run it,
and launch **DriftlessFLIP** from the Start Menu. The installer contains the application,
Python runtime, and open-source dependencies; users do not need Conda, Python, or terminal
commands. It installs per user and does not remove or overwrite recordings, profiles, or
settings in `Documents\DriftlessFLIP` during upgrades or uninstall.

PicoQuant's driver and licensed `TH260Lib64.dll` remain separate installations and are
never bundled. For hardware acquisition, obtain TH260Lib from PicoQuant or install
PicoQuant's [snAPI](https://github.com/PicoQuant/snAPI), which supplies the DLL, and keep
the PicoQuant device driver installed. If automatic discovery does not find the DLL,
select its installed location in DriftlessFLIP's **TH260 DLL** field. Simulator mode works
without either component. Installer builds are currently unsigned, so Windows may show a
publisher warning until code signing is added.

## Installation for developers

Use 64-bit Python 3.11 or newer in a dedicated
[Conda](https://docs.conda.io/) environment. Clone this repository, open PowerShell in
the clone, and run:

```powershell
conda create -n driftlessflip python=3.12
conda activate driftlessflip
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
```

This editable installation uses the files in the clone, so code changes take effect
without reinstalling. Run the tests with `python -m pytest -q`.

For a non-editable installation of the latest published Python package, use:

```powershell
python -m pip install --upgrade driftlessflip
```

Developers must obtain the PicoQuant driver and licensed `TH260Lib64.dll` separately too;
pip and Conda cannot install them. Install TH260Lib from PicoQuant or use the DLL supplied
with [snAPI](https://github.com/PicoQuant/snAPI). Keep it at the default path or select its
actual location in the GUI's **TH260 DLL** field. Never commit the vendor DLL to this
repository. Simulator mode works without the driver or DLL.

To create the application executable and Setup installer locally or through GitHub
Actions, follow [Building the Windows executable and installer](docs/BUILD_WINDOWS_INSTALLER.md).
The short local build is:

```powershell
python -m pip install -e ".[dev,packaging]"
.\scripts\build_windows_installer.ps1
```

The result is `installer-dist\DriftlessFLIP-<version>-Setup.exe`. The automation attaches
that executable and its SHA-256 checksum to every published GitHub Release. Release tags
must exactly match the application version as `v<version>`.

---

## Getting started

Launch the interface:

```powershell
driftlessflip gui
```

### The normal workflow

1. **Preview.** Press **Preview** and watch the count rates, live traces, and decay curve.
   Nothing is saved. Adjust discriminator thresholds, channel offsets, and `t0`, then
   preview again — each preview re-applies the current settings.
2. **Calibrate `t0` against the live decay.** In **Hardware & calibration**, press **Fit
   decay curve** and then **Apply fitted t0**. `t0` is live-adjustable, so the running
   preview picks it up at once and MPET updates with it — no throwaway recording needed.
   Settings that reprogram the board, such as channel timing offsets, still need the
   preview stopped and restarted, because TH260Lib forbids changing them mid-measurement.
3. **Check the rate.** The count-rate readout is coloured against the pile-up and detector
   dead-time limits for your sync rate. If it reads over, reduce excitation power — never
   raise the discriminator threshold to bring the rate down. A higher threshold discards
   valid photon events rather than fixing pile-up, and costs signal-to-noise for nothing
   in return; lower power also reduces photobleaching and photodamage to the sample.
   A detected rate of roughly 2×10⁶–1.5×10⁷ counts/s is a reasonable operating range for
   this hardware; at the low end, aim for the signal to sit at least 10× above whatever
   the detector reads with no sample connected.
4. **Record a background.** Same settings, no signal, ten seconds is usually enough. Tick
   **This run is a background measurement**.
5. **Record.** Load the background as the reference, then press **Start recording**.
6. **Review.** Open the recording, integrate the decay over a time range, fit it, and read
   the phasor position.

### From the terminal

```powershell
driftlessflip preview --duration 10
driftlessflip simulate --output recordings --prefix bg --duration 10 --background
driftlessflip simulate --output recordings --prefix experiment --background-reference recordings\bg_<timestamp>
driftlessflip device-info
driftlessflip verify recordings\experiment_<timestamp>
driftlessflip import-legacy path\to\legacy-recording
driftlessflip recover recordings\experiment_<timestamp>
```

---

## Key concepts

### Preview is not a recording

Preview and recording share the device, so only one runs at a time. Preview exists for
setup: it shows count rates, traces, and the summed decay curve you set CFD levels and
`t0` from, and it is bounded by a rolling window rather than the recording duration, so it
can run indefinitely at fixed memory.

Preview **cannot** produce a recording. The module imports nothing from the storage layer,
so it cannot create a run folder, a raw file, or a manifest. While previewing, the control
reads `Previewing — not saving`.

### What a recording contains

Each run creates one immutable folder named for its prefix and start time:

```text
experiment_20260802T193052.123456Z/
  experiment_20260802T193052.123456Z.tttr.h5        raw TTTR records and applied corrections
  experiment_20260802T193052.123456Z.nwb            histograms and derived traces
  experiment_20260802T193052.123456Z_metrics.csv    per-sample, per-channel values
  experiment_20260802T193052.123456Z_metadata.json  canonical RunMetadata
  experiment_20260802T193052.123456Z_manifest.json  sizes and SHA-256 of every artifact
  experiment_20260802T193052.123456Z_run.log        structured event log
```

The raw TTTR stream is authoritative. One versioned `RunMetadata` object is embedded in
every artifact and cross-checked between them, so a package that verifies is internally
consistent. `driftlessflip verify` re-checks that at any time.

### Settings files and profiles

Two ways to keep a configuration, for two different jobs.

A **settings profile** is a named instrument setup kept between sessions in your home
directory. It deliberately leaves out the fields that belong to one particular run — the
recording prefix, the background reference, and whether the run is a background — so
loading a routine setup cannot drag last week's background along with it.

A **settings file** is everything, written wherever you choose: hardware, analysis,
experiment metadata, the background reference, output location and prefix included. Use
**Export all…** and **Import all…** on the Acquire tab. It is indented, key-sorted JSON
that you can read, hand-edit, diff between sessions, keep alongside a protocol, or send
to a collaborator so they can reproduce your setup exactly.

Importing never repairs a file quietly. A file that is not a settings file, or that has a
key the application does not know, is refused with the offending detail named. A file
that loads but asks for something unavailable here — a background recording that has
moved, a value outside its allowed range — is applied and the problem reported, so you
can see and fix it rather than meeting it at record time.

The settings file is a setup document, not a provenance record: what a recording actually
used is written inside that recording.

### Where settings come from at startup

Everything DriftlessFLIP keeps for you lives in one visible folder:

```text
Documents/DriftlessFLIP/
  default_settings.json     loaded at every start
  profiles/                 named instrument setups
  recordings/               where recordings go unless you point them elsewhere
```

It is deliberately not a hidden dotfolder: these files exist to be read, hand-edited,
copied to a colleague, and handed over with a rig. Settings kept under `~/.driftlessflip`
by an earlier version are moved here automatically on first start, without overwriting
anything already present.

On Windows the folder is found through the location Explorer itself uses, so a Documents
folder redirected to OneDrive — the default on a new machine — is followed rather than
guessed at. Because that can make for a deep path, and because a recording's prefix and
timestamp appear both in its folder name and in every file inside it, DriftlessFLIP warns
before arming if a run's longest path would exceed the 260-character Windows limit. That
failure would otherwise land at finalisation, after the data had been collected.

DriftlessFLIP loads `default_settings.json` when it exists, so the application opens on
your rig rather than on factory defaults. Press **Set as default** to write the current
settings there. Having no such file is the ordinary first-run state.
If the file exists but cannot be read, the application still starts, on built-in defaults,
and says so in the status line and the Diagnostics tab — it will not fail silently and
leave you believing your own settings are in force.

Three ways to load a setup, in the order you are likely to want them:

1. **The default file**, applied automatically at startup.
2. **Import all…**, for a settings file someone sent you or that you keep with a protocol.
3. **From recording…**, which rebuilds a setup from a past recording's own metadata.

The third has two forms. Next to *Settings profile* it loads a past recording's instrument
settings into the current session while keeping your output location and recording name.
Next to *Settings file* it writes that recording's configuration out as a settings file,
prefix and background reference included, so a session can be turned into a reusable
setup. The same thing without the interface:

```powershell
driftlessflip export-settings recordings\experiment_<timestamp> --output my_rig.settings.json
```

Settings are read from the recording's canonical metadata JSON. The NWB file carries an
identical copy of the same block, and both are checksummed in the manifest.

### Background subtraction

Record a background with matching settings and no signal, then load it as the reference
for an experiment. It is refused unless it matches on sample rate, lifetime binning, laser
rate, sync configuration, and every per-channel discriminator and offset; the error names
the field that differs. Duration may differ, since the correction uses the mean background
per sample.

The measured background is scaled by the ratio of detector live fractions before
subtraction, because a detector busy with signal photons collects less background than it
did when measured alone. The background's identity and the correction actually applied are
stored with the recording, so it stays reproducible even if the background file is later
moved.

### Which settings can change while acquiring

`t0`, the MPET window, and the afterpulse ratio are applied when metrics are recalculated,
so they can be retuned **during a preview** with immediate effect. Everything else is
locked while a measurement runs: TH260Lib forbids reprogramming the board mid-measurement,
and the sample rate, lifetime binning, and output location fix buffer shapes and identity
at start.

During a **recording** everything is locked, including those three, because metadata is
captured when the recording starts and a later edit would let the stored configuration
disagree with the derived outputs. Raw TTTR retention means any of them can be reapplied
offline. For the reasoning behind exactly which settings fall on which side of that line —
and how the raw T3 stream makes it possible — see [`SOFTWARE_TUTORIAL.md` section
3](docs/SOFTWARE_TUTORIAL.md#3-raw-retention-vs-hardware-settings-what-you-can-change-after-recording).

### External TTL markers

The PICO DUAL marker inputs are Marker 1/pin 9, Marker 2/pin 4, Marker 3/pin 5, and
Marker 4/pin 10. The acquisition trigger line is pin 13 and measurement status is output on
pin 3, with ground on pin 12 or 14.

`TH260_SetMarkerEdges` selects one active edge per input, so a single input cannot capture
both polarities. To measure pulse width, split the source to two inputs, set one rising and
one falling, and give both the same **Pulse signal** name. Each rising edge is then matched
to the next falling edge automatically.

The exact event table retains macro-sync count, recording-relative time, derived UTC
timestamp, edge, input name and pin, source bitmask, and sample index. The per-sample
`external_marker_bits` trace is only a summary — use the event table when timing or event
multiplicity matters.

---

## Interpreting the numbers

**MPET is not a fitted lifetime.** It is the photon-weighted mean arrival time within the
configured analysis window, minus `t0`. Truncation by the laser period and the window
shifts it below the exponential time constant, so the interface labels it *mean photon
emission time* rather than lifetime. Use the fit when you want `tau`.

**Microtime is nominal `dtime * resolution`**, the bin's left edge, matching the TimeHARP
convention. Bin centres exist only as a display axis. This means a calibrated `t0` has to
absorb the offset between a bin's left edge and the mean arrival time within it, which is
why `t0` is determined by fitting rather than assumed.

**Phasor spans one full laser period**, independent of the MPET window. A cropped window
rotates and shortens the vector and moves even a pure single-exponential decay off the
universal semicircle.

**Corrections happen before windowing.** Background and afterpulsing are removed across the
complete decay curve, so changing the MPET window cannot change how much was subtracted.

Every one of these choices is recorded in the `calculation_contract` and
`algorithm_versions` fields of each recording, so a file states the conventions it was
computed under.

---

## Current limitations

- Physical TimeHARP operation is validated and ready for experiments. A new acquisition
  computer, board, detector, or timing configuration still needs its own profile check;
  the supplied bench procedures make that verification reproducible.
- **The board can enter an unresponsive state that neither DriftlessFLIP nor the vendor's
  own software can currently distinguish from "not connected."** No software fix is known.
  If `device-info` or a board scan fails immediately after working moments earlier: close
  every process that might hold the device, and if it still won't open, power-cycle the
  computer (unplugging power for ~15 s before restart has been reported to help; this is
  a board/driver-level condition, not something DriftlessFLIP causes or can detect from
  software alone).
- Histogram memory is bounded before arming but allocated for the whole requested run, so
  a memory budget caps run length rather than removing the limit.
- The software FIFO handoff queue defaults to 256 chunks (at most 128 MiB) and is
  configurable from Acquire. A sustained full queue or a hardware FIFO-full flag stops
  acquisition and preserves a verified truncated prefix. Increasing the queue absorbs
  temporary storage/CPU stalls; it cannot reconstruct records the board already lost or
  compensate indefinitely when processing is slower than acquisition.
- NWB uses valid core types plus lossless canonical metadata; formal fibre-photometry
  extension mappings are not done.
- Unsupported legacy formats are refused rather than guessed.
- The simulator models a Gaussian instrument response, periodic re-excitation,
  afterpulsing, dead time, and structured background, but not measured per-instrument IRF
  shape or timing jitter.

[`ROADMAP.md`](docs/ROADMAP.md) lists what remains and why.

---

## Development

```powershell
pytest
```

The suite covers T3 decoding against a hand-calculated golden stream, correction arithmetic,
phasor geometry, fit parameter recovery, storage verification, recovery, overrun salvage,
looping, and cross-validation against independent golden fixtures. GitHub Actions
runs it on Python 3.11 and 3.12, on Ubuntu and Windows.

---

## Documents

| File | What it covers |
|---|---|
| [`docs/FAQ.md`](docs/FAQ.md) | Common questions on theory, setup, and calibration, answered in one place |
| [`educational/`](educational/README.md) | Background notes: what the measurement is, how to run an experiment, how to calibrate the instrument, how to compare lifetimes across sessions, and how it compares with frequency-domain methods |
| [`docs/SOFTWARE_TUTORIAL.md`](docs/SOFTWARE_TUTORIAL.md) | Practical workflow for running an experiment, plus a button-by-button tour of every tab |
| [`docs/OUTPUT_DATA_STRUCTURE.md`](docs/OUTPUT_DATA_STRUCTURE.md) | What each file in a recording folder contains, with real example data |
| [`WHATS_NEW.md`](WHATS_NEW.md) | Dated release history and the reasoning behind each change |
| [`docs/ROADMAP.md`](docs/ROADMAP.md) | What remains, and the constraints on future work |
| [`docs/DESIGN_PLAN.md`](docs/DESIGN_PLAN.md) | Measurement model, hardware constraints, and deployment checks |
| [`docs/SOFTWARE_ARCHITECTURE.md`](docs/SOFTWARE_ARCHITECTURE.md) | How the software is built |
| [`docs/BUILD_WINDOWS_INSTALLER.md`](docs/BUILD_WINDOWS_INSTALLER.md) | How maintainers build and automatically attach the Windows installer to releases |
| [`docs/SYNC_TIMEBASE_BENCH_PROCEDURE.md`](docs/SYNC_TIMEBASE_BENCH_PROCEDURE.md) | Bench procedure for the sync time base |
| [`docs/CFD_THRESHOLD_BENCH_PROCEDURE.md`](docs/CFD_THRESHOLD_BENCH_PROCEDURE.md) | Bench procedure for verifying CFD threshold/zero-cross against the real pulse |
| [`docs/INDEPENDENT_AUDIT.md`](docs/INDEPENDENT_AUDIT.md) | Current independent audit: latest assessment and open findings |
| [`docs/ARCHIVED_AUDIT.md`](docs/ARCHIVED_AUDIT.md) | Resolved audit findings, dated by fix commit and later verification |
| [`docs/HARDWARE_API_COMPARISON.md`](docs/HARDWARE_API_COMPARISON.md) | Why the hardware adapter uses low-level TH260Lib calls instead of PicoQuant's snAPI |

---

## Development, inspiration, and license

DriftlessFLIP is developed by **Mohebi and Associates**.

The software was inspired by **iFLIP2 from the Hong Lab**.

Released under the [MIT License](LICENSE). PicoQuant drivers and `TH260Lib64.dll` are
separately licensed third-party components and are not distributed under it. Third-party
manuals and journal articles are cited by link in [`DESIGN_PLAN.md`](docs/DESIGN_PLAN.md), are
not stored in this repository, and are outside the scope of the MIT license.
