Metadata-Version: 2.1
Name: pyspark-util
Version: 0.1.2
Summary: PySpark utility functions
Home-page: https://github.com/harupy/pyspark-util
Maintainer: harupy
Maintainer-email: hkawamura0130@gmail.com
License: UNKNOWN
Project-URL: Bug Tracker, https://github.com/harupy/pyspark-util/issues
Project-URL: Source Code, https://github.com/harupy/pyspark-util
Platform: UNKNOWN
Requires-Python: >=3.5
Description-Content-Type: text/markdown
Requires-Dist: pyspark (>=2.4.4)

# pyspark-util

A set of pyspark utility functions.

```python
import pyspark_util as psu

data = [(1, 2, 3)]
columns = ['a', 'b', 'c']
df = spark.createDataFrame(data, columns)
prefixed = psu.prefix_columns(df, 'x')
prefixed.show()

# output:
+---+---+---+
|x_a|x_b|x_c|
+---+---+---+
|  1|  2|  3|
+---+---+---+
```

## Development

### Setup

```
docker-compose build
docker-compose up -d
```

### Lint

```
docker exec psu-cnt ./tools/lint.sh
```

### Test

```
docker exec psu-cnt ./tools/test.sh
```


