Metadata-Version: 2.4
Name: crny
Version: 1.1.1
Summary: A file format made for testing
Project-URL: Homepage, https://gitlab.com/acfma-group/Crny
Project-URL: Issues, https://gitlab.com/acfma-group/Crny/-/incidents
Project-URL: docs, https://acfma-group.gitlab.io/Crny/
Author-email: Andrea Carli <carliandrea.pub@gmail.com>
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# CRNY

<div align="center">
<img src="https://gitlab.com/acfma-group/Crny/-/raw/main/images/crnyLogo.png?ref_type=heads" height="200" width="250" alt="Project Logo">
</div>

CRNY is a file format made for testing intended to be a compact yet practical file that could be shared as simply as a txt file without the need for specialized software or infrastructure.
Without any kind of tracking, the knowledge transfer between colleagues is only trust-based. This is fine if you trust the other party but it could be inconvenient when new tests or more details are required.
With CRNY, you could actually look at the file's history and schedule a pipeline to re-enact those tests. Think of it as a CI test for data. 
Since CRNY should be able to run on any setup, no external libraries have been used: Python basic package should be enough.

## The basic idea
Since CRNY has been made for testing, it should accommodate every kind of data and data frames there is. This requires a specialize structure called *crnyFrame*.
While using crnyFrames, fields must be thought as independent items, giving the necessity of a columnar-based file format similar to [Apache Parquet](https://parquet.apache.org/) from which it takes the idea of storing data as "footer-first file" (i.e. reading the footer first and then search for the column).
This approach largely improves the IO stage when not every column is required.
Beside that, CRNY also stores the changes made to each individual column as deltas and applies them in reverse when rollbacks are required.
Here's a practical visualization on how the data is stored on a .crny file.

<div align="center">
<img src="https://gitlab.com/acfma-group/Crny/-/raw/main/images/crnySchema.png?ref_type=heads" height="200" width="250" alt="Crny Schema">
</div>

The history can be changed when the crnyFrame is loaded onto memory but, once written as a compressed file format, it cannot be changed.
To ensure this behavior, once a new change is made, an hash is associated with the transformation based on the previous column layout. Since you as an author have all the right to change the aspect and content of your file, you shouldn't be able to overwrite existing file or history without consistency, potentially undermining other's work.
You should think of your crnyFrame as a Git branch and the act of saving it as a file as a "push" into remote.
 
