Metadata-Version: 2.1
Name: ddbimp
Version: 0.2
Summary: Easily load data from CSV to test out your DynamoDB table design
Home-page: https://github.com/AlexJReid/dynamodb-dev-importer
Author: AlexJReid
Author-email: alexjreid@me.com
License: Apache
Platform: UNKNOWN
Description-Content-Type: text/markdown
Requires-Dist: boto3

# dynamodb-dev-importer

_Easily load data from CSV to test out your DynamoDB table design._

When working with DynamoDB, it is common practice to minimise the number of tables used, ideally down to just one.

Techniques such as sparse indexes and GSI overloading allow a lot of flexibility and efficiency.

Designing a good schema that supports your query patterns can be challenging. Often it is nice to try things out with a small amount of data.
I personally find it convenient to enter data into a spreadsheet and play around with it there.

When ready to try out an approach with DynamoDB, it's a hassle to then create a items in a table through the AWS Console or CLI, so this script:
- reads a CSV file (exported from your spreadsheet) and imports it into a DynamoDB table 
- columns 0 and 1 are used for the key: partition key `pk: S` and sort key `sk: S` - your target table needs these keys defined
- column 2, if not an empty string, is set to `data: S`
- all other columns are added as non-key attributes

Your CSV should contain columns for:
- pk
- sk
- data (optional)
- anything after those three can contain arbitrary attributes of form `attribute_name: value` i.e. `city: Edinburgh`

Example row:
```
PERSON-1,sales-Q1-2019,Alex,jan: 12012,feb: 1927
```

Will yield an item like this:
```
{
    pk: 'PERSON-1',
    sk: 'sales-Q1-2019',
    data: 'Alex',
    jan: 12012,
    feb 1927
}
```

## Usage
Assuming DynamoDB table `example(pk, sk)` is setup and you're in a virtual environment. If you already have boto3 installed, you don't need to install any packages.
```
$ pip install ddbimp
$ ddbimp --table example --skip 1 example.csv
```

## Key ideas
- Table consists of partition key `pk: S` and sort key `sk: S` - their meaning varies depending on the item
- A secondary index swaps the sort and partition keys, so the partition key is `sk: S` and sort key `pk: S`
- A final secondary index uses `sk: S` and `data: S` where data is an arbitrary value you might want to search for, the meaning of `data` depends on the item it is part of
- Group items through a shared partition key, store _sub_ items with a sort key e.g. 
    - e.g. `pk:PERSON-1, sk:sales-Q1-2019, jan:12012, feb:1927`

See [example.csv](example.csv) for an example input file.

AWS recently [released a preview build of a tool called NoSQL Workbench](https://aws.amazon.com/blogs/aws/nosql-workbench-for-amazon-dynamodb-available-in-preview/). It builds on the above ideas. I've tried it out and it seems pretty good, but I am a luddite and am faster working in a spreadsheet right now. I'd certainly recommend giving it a try.

## Useful resources
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-indexes.html
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.html
- https://www.youtube.com/watch?v=6yqfmXiZTlM

## Caveats, TODO
- Uses your default AWS profile
- Region needs to be set
- Make work directly with a Google Sheets via sheets API

