Metadata-Version: 2.3
Name: persist_ext
Version: 1.6.1
Dynamic: Keywords
Summary: PersIst is a JupyterLab extension to enable persistent interactive visualizations in JupyterLab notebooks.
Project-URL: Homepage, https://github.com/visdesignlab/persist
Project-URL: Bug Tracker, https://github.com/visdesignlab/persist/issues
Project-URL: Repository, https://github.com/visdesignlab/persist.git
Author-email: Kiran Gadhave <kirangadhave2@gmail.com>
License: BSD 3-Clause License
        
        Copyright (c) 2023, Kiran Gadhave
        All rights reserved.
        
        Redistribution and use in source and binary forms, with or without
        modification, are permitted provided that the following conditions are met:
        
        1. Redistributions of source code must retain the above copyright notice, this
           list of conditions and the following disclaimer.
        
        2. Redistributions in binary form must reproduce the above copyright notice,
           this list of conditions and the following disclaimer in the documentation
           and/or other materials provided with the distribution.
        
        3. Neither the name of the copyright holder nor the names of its
           contributors may be used to endorse or promote products derived from
           this software without specific prior written permission.
        
        THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
        AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
        IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
        DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
        FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
        DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
        SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
        CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
        OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
        OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
License-File: LICENSE
Classifier: Framework :: Jupyter
Classifier: Framework :: Jupyter :: JupyterLab
Classifier: Framework :: Jupyter :: JupyterLab :: 4
Classifier: Framework :: Jupyter :: JupyterLab :: Extensions
Classifier: Framework :: Jupyter :: JupyterLab :: Extensions :: Prebuilt
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.8
Requires-Dist: altair>=5
Requires-Dist: anywidget
Requires-Dist: fastparquet
Requires-Dist: ipywidgets
Requires-Dist: jupyterlab>=4.0.4
Requires-Dist: lzstring
Requires-Dist: pandas>=0.25
Requires-Dist: paretoset
Requires-Dist: pyarrow
Requires-Dist: scikit-learn
Requires-Dist: traittypes==0.2.1
Requires-Dist: vega-datasets
Provides-Extra: dev
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: vega-datasets; extra == 'dev'
Description-Content-Type: text/markdown

# Persist

## Persistent and Reusable Interactions in Computational Notebooks

[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/visdesignlab/persist/main?urlpath=lab)

This repository contains source code for Persist ([PyPi](https://pypi.org/project/persist_ext/)) extension.

Persist is a JupyterLab extension to enable persistent interactive outputs in JupyterLab notebooks. Check out the introductory video below.

https://github.com/visdesignlab/persist/assets/14944083/c6a9347b-7c93-4d0d-9e60-e10707578327

[Watch on Youtube with CC](https://www.youtube.com/watch?v=DXHXPvRHN9I)

## Getting Started

### Requirements

```markdown
- JupyterLab >= 4.0.0 or Jupyter Notebook >= 7.0.0
- pandas >= 0.25
- altair >= 5
- ipywidgets
- anywidget
```

### Install

To install the extension, execute:

```bash
pip install persist_ext
```

If the Jupyter server was already running, you might have to reload the browser page and restart the kernel.

### Uninstall

To remove the extension, execute:

```bash
pip uninstall persist_ext
```

### Example

After installing the extension, you can use the following code snippet to create an Persist-enabled interactive data table.

```bash

```

TODO:

- describe a simple example to use persist.
- link to a notebook that introduces persist and altair
- link to the documentation

### Persist and Vega-Altair charts

Persist works with Vega-Altair charts directly for the most part. Vega-Altair and Vega-Lite offer multiple ways to write a specification. However Persist has certain requirements that need to be fulfilled.

- The selection parameters in the chart should be named. Vega-Altair's default behavior is to generate a name of selection parameter with auto-incremented numeric suffix. The value of the generated selection parameter keeps incrementing on subsequent re-executions of the cell. Persist relies on consistent names to replay the interactions, and passing the name parameter fixes allows Persist to work reliably.

- The point selections should have at least the fields attribute specified. Vega-Altair supports selections without fields by using the auto-generated indices to define selections. The indices are generated with the default order of rows in the source dataset. Using the indices directly for selection can cause Persist to operate on incorrect rows if the source dataset order changes.

- Dealing with datetime in Pandas is challenging. To standardize the way datetime conversion takes place within VegaLite and within Pandas when using Vega-Altair, the TimeUnit transforms and encodings must be specified in UTC. e.g `month(Date)` should be `utcmonth(Date)`.

### Publication

Persist is developed as part of a [publication](https://osf.io/preprints/osf/9x8eq) and will appear in EuroVis 2024.

![Teaser image from the pre-print. The figure describes the workflow showing high level working of Persist technique.](public/imgs/teaser.png)

### Supplementary Material

Supplementary material including example notebooks, walkthrough notebooks, notebooks used in the study (including participant notebooks) and the analysis notebooks can be accessed [here](https://github.com/visdesignlab/persist_examples).

#### Abstract

> Computational notebooks, such as Jupyter, support rich data visualization. However, even when visualizations in notebooks are interactive, they still are a dead end: Interactive data manipulations, such as selections, applying labels, filters, categorizations, or fixes to column or cell values, could be efficiently apply in interactive visual components, but interactive components typically cannot manipulate Python data structures. Furthermore, actions performed in interactive plots are volatile, i.e., they are lost as soon as the cell is re-run, prohibiting reusability and reproducibility. To remedy this, we introduce Persist, a family of techniques to capture and apply interaction provenance to enable persistence of interactions. When interactions manipulate data, we make the transformed data available in dataframes that can be accessed in downstream code cells. We implement our approach as a JupyterLab extension that supports tracking interactions in Vega-Altair plots and in a data table view. Persist can re-execute the interaction provenance when a notebook or a cell is re-executed enabling reproducibility and re-use.
>
> We evaluated Persist in a user study targeting data manipulations with 11 participants skilled in Python and Pandas, comparing it to traditional code-based approaches. Participants were consistently faster with Persist, were able to correctly complete more tasks, and expressed a strong preference for Persist.

## Contributing

Persist uses [hatch](https://hatch.pypa.io/latest/) to manage the development, build and publish workflows. You can install `hatch` using `pipx`, `pip` or Homebrew (on MacOS or Unix).

##### **pipx**

Install `hatch` globally in isolated environment. We recommend this way.

```bash
pipx install hatch
```

##### **pip**

Install hatch in the current Python environment.

_**WARNING**_: This may change the system Python installation.

```bash
pip install hatch
```

##### **Homebrew**

```bash
pip install hatch
```

Jupyter extensions use a custom version of `yarn` package manager called `jlpm`. When any relevant command is run, `hatch` should automatically install and setup up `jlpm`.
After installing `hatch` with your preferred method follow instructions below for workflow you want. We prefix all commands with `hatch run` to ensure they are run in proper environments.

### Development

Run the `setup` script from `package.json`:

```bash
hatch run jlpm setup
```

When setup is completed, open three terminal windows and run the follow per terminal.

#### Widgets

Setup vite dev server to build the widgets

```bash
hatch run watch_widgets
```

#### Extension

Start dev server to watch and build the extension

```bash
hatch run watch_extension
```

#### Lab

Run JupyterLab server with `minimize` flag set to `false`, which gives better stack traces aqnd debugging experience.

```bash
hatch run run_lab
```

### Build

To build the extension as a standalone Python package, run:

```bash
hatch run build_extension
```

### Publish

To publish the extension, first we create a proper version. We can run any of the following

```bash
hatch version patch # x.x.1
hatch version minor # x.1.x
hatch version major # 1.x.x
```

You can also append release candidate label:

```bash
hatch version rc
```

Finally you can directly specify the exact version:

```bash
hatch version "1.3.0"
```

Once the proper version is set, build the extension using the `build` workflow.

When the build is successful, you can publish the extension if you have proper authorization:

```bash
hatch publish
```

### Acknowledgements

The widget architecture of Persist is created using [anywidget](https://github.com/manzt/anywidget) projects.

The interactive visualizations used by Persist are based on the excellent, [Vega-Lite](https://github.com/vega/vega-lite) and [Vega-Altair](https://github.com/altair-viz/altair) projects. Specifially the implementation of [JupyterChart](https://github.com/altair-viz/altair/blob/main/altair/jupyter/jupyter_chart.py) class in Vega-Altair was of great help in understanding how Vega-Altair chart can be turned into a widget. We gratefully acknowledge funding from the National Science Foundation (IIS 1751238 and CNS 213756).

<!-- ### Citing
```bibtex
@article{gadhave_2023,
 title={Persist: Persistent and Reusable Interactions in Computational Notebooks},
 url={osf.io/9x8eq},
 DOI={10.31219/osf.io/9x8eq},
 publisher={OSF Preprints},
 author={Gadhave, Kiran, Cutler, Zach and Lex, Alexander},
 year={2023},
 month={Dec}
}
``` -->
