Metadata-Version: 2.4
Name: aws-resource-inventory
Version: 0.1.0
Summary: AWS multi-service resource inventory with tag-based filtering and advanced optimizations
License-Expression: MIT
License-File: LICENSE
Keywords: aws,scanner,cloud,infrastructure,devops
Author: Ishan Sharma
Requires-Python: >=3.10
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Systems Administration
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: boto3 (>=1.40.34)
Requires-Dist: botocore (>=1.40.34)
Requires-Dist: click (>=8.3.0)
Requires-Dist: pyfiglet (>=1.0.4)
Requires-Dist: rich (>=14.1.0)
Requires-Dist: setuptools (>=80.9.0)
Requires-Dist: typer (>=0.17.4)
Requires-Dist: urllib3 (>=2.5.0)
Project-URL: Homepage, https://github.com/ishuar/aws-resource-inventory
Project-URL: Repository, https://github.com/ishuar/aws-resource-inventory
Description-Content-Type: text/markdown

# AWS Resource Inventory

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![Code Style: Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/ishuar/aws-resource-inventory/main.svg)](https://results.pre-commit.ci/latest/github/ishuar/aws-resource-inventory/main)

A comprehensive AWS multi-service scanner with tag-based filtering, parallel processing, advanced logging capabilities, and optimization features. This tool enables efficient discovery and analysis of AWS resources across multiple regions and services with intelligent caching, rich output formats, and detailed AWS API tracing.

## Features

- Scans eight AWS services — EC2, S3, ECS, EFS, VPC, RDS, ELB, and Auto Scaling — across multiple regions concurrently, with or without tag filters.
- Discovers resources from 100+ AWS services via the Resource Groups Tagging API (`--all-services`, requires a tag filter).
- Table, JSON, and Markdown output; results are rendered in the terminal and written to a file for further processing.
- Result caching with a 10-minute TTL (`--cache` / `--no-cache`).
- Configurable parallelism per region (`--max-workers`) and per service (`--service-workers`), dry-run preview, continuous refresh mode, and graceful Ctrl+C handling.
- Debug traces (`--debug`), full AWS API tracing (`--verbose`), and custom log files (`--log-file`) — see [Logging Architecture](docs/LOGGING_ARCHITECTURE.md).

## 🏗️ Supported AWS Services

| Service          | Description                                                                                                             | Resources Scanned                                                 |
|------------------|-------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| **All Services** | With [Resource Groups Tagging API](https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/overview.html) | 100+ AWS services when using tags                                 |
| **EC2**          | Elastic Compute Cloud                                                                                                   | Instances, Volumes, Security Groups, AMIs, Snapshots              |
| **S3**           | Simple Storage Service                                                                                                  | Buckets and their configurations                                  |
| **ECS**          | Elastic Container Service                                                                                               | Clusters, Services, Task Definitions, Capacity Providers          |
| **VPC**          | Virtual Private Cloud                                                                                                   | VPCs, Subnets, Route Tables, IGWs, NAT Gateways, DHCP Options, Peering Connections, Endpoints |
| **Auto Scaling** | Auto Scaling Groups                                                                                                     | ASGs, Launch Configurations, Launch Templates                     |
| **ELB**          | Elastic Load Balancing                                                                                                  | Load Balancers (ALB/NLB), Target Groups, Listeners, Listener Rules |
| **RDS**          | Relational Database Service                                                                                             | DB Instances, DB Clusters, DB Snapshots, Aurora Cluster Snapshots |
| **EFS**          | Elastic File System                                                                                                     | File Systems (with size and mount-target details)                 |

> **📚 Architecture Details**: For detailed information about the scanning architecture and service implementation patterns, see [Architecture Documentation](docs/Architecture.md).

## 📋 Prerequisites

Before installing the AWS Resource Inventory, ensure you have the following dependencies:

### Required Software
- **Python 3.10+** - Runtime environment
- **Poetry** - Python dependency management and packaging
- **pip** - Python package installer
- **AWS CLI** - AWS command line interface for authentication
- **pre-commit** - Git hooks framework (for development)

### AWS Configuration
- Valid AWS credentials configured via:
  - AWS SSO (`aws sso login`)
  - AWS CLI (`aws configure`)
  - Environment variables
  - IAM roles (for EC2/containers)

### Required IAM Permissions

The scanner is read-only. This policy covers every API call it makes —
the eight service scanners, the credential check (`sts`), and the tag
scan (`tag:GetResources`, only exercised with `--tag-key`/`--tag-value`/
`--all-services`):

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AwsResourceInventoryReadOnly",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity",
                "ec2:Describe*",
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation",
                "s3:GetBucketTagging",
                "ecs:List*",
                "ecs:Describe*",
                "elasticloadbalancing:Describe*",
                "elasticfilesystem:Describe*",
                "rds:Describe*",
                "autoscaling:Describe*",
                "tag:GetResources"
            ],
            "Resource": "*"
        }
    ]
}
```

> [!NOTE]
> `ec2:Describe*` also covers the VPC scanner and Auto Scaling launch
> templates — both use the EC2 API. The AWS-managed `ReadOnlyAccess`
> policy works too if you prefer not to maintain a custom one.

## 🚀 **Installation**

### Automated Setup (Recommended)

The easiest way to get started is using our automated setup script:

```bash
# Clone the repository
git clone https://github.com/ishuar/aws-resource-inventory.git
cd aws-resource-inventory

# Run the automated setup script
./setup.sh
```

The setup script will:
1. ✅ Check and install Python 3.10+
2. ✅ Install Poetry (Python dependency manager)
3. ✅ Install pre-commit (Git hooks framework)
4. ✅ Set up pre-commit hooks
5. ✅ Install all project dependencies via Poetry
6. ✅ Run verification tests
7. ✅ Install AWS CLI (if not present)
8. ✅ Provide AWS configuration guidance

> [!Tip]
> For detailed setup instructions and troubleshooting, see [setup.sh](setup.sh).

### Manual Installation

If you prefer manual installation:

```bash
# Install dependencies (macOS with Homebrew)
brew install python3 poetry pre-commit awscli

# Install project dependencies
poetry install

# Set up pre-commit hooks
pre-commit install --install-hooks

# Verify installation
./run_quick_tests.sh
```

## 💻 Usage

### Command Structure

All commands follow this pattern:
```bash
poetry run aws-inventory [GLOBAL OPTIONS] COMMAND [COMMAND OPTIONS]
```

Global options apply to all commands: `--verbose` / `-v` (AWS API tracing,
use with `--debug`) and `--log-file` / `-l` (custom log file path). The
main command is `scan`.

> [!IMPORTANT]
> Global options must come **before** the command. `aws-inventory scan --verbose`
> fails with `No such option: --verbose` — the correct form is
> `aws-inventory --verbose scan`.

### Basic Commands

```bash
# Display help and available commands
poetry run aws-inventory --help

# Display scan command help and options
poetry run aws-inventory scan --help

# Basic scan with default settings (all supported services)
poetry run aws-inventory scan --regions us-east-1,eu-west-1,eu-central-1,us-west-2

# Scan specific services
poetry run aws-inventory scan --service ec2

# Scan specific regions
poetry run aws-inventory scan --regions us-east-1,eu-west-1
```

### Debug and Logging Options

```bash
# Enable debug mode for detailed execution traces
poetry run aws-inventory scan --debug --regions us-east-1

# Enable verbose AWS API tracing (requires --debug)
poetry run aws-inventory --verbose scan --debug --service ec2

# Custom log file for debug output
poetry run aws-inventory --log-file /tmp/my-scan.log scan --debug --regions us-east-1

# Combine verbose logging with custom log file
poetry run aws-inventory --verbose --log-file /tmp/aws-api-trace.log scan --debug --service ec2 --service s3

# Debug with dry run (no actual scanning)
poetry run aws-inventory --verbose scan --debug --dry-run --service vpc
```

### Service-Specific Scanning

```bash
# Scan only EC2 resources
poetry run aws-inventory scan --service ec2

# Scan multiple services
poetry run aws-inventory scan --service ec2 --service s3 --service vpc

# Scan all built-in services in specific regions
poetry run aws-inventory scan --regions us-east-1,us-west-2

# Combine service and region filtering
poetry run aws-inventory scan --service ec2 --regions eu-central-1,eu-west-1

# Scan ALL AWS services using Resource Groups API (requires tags)
poetry run aws-inventory scan --all-services --tag-key Environment --tag-value Production
```

### Tag-Based Filtering

```bash
# Filter by environment tag
poetry run aws-inventory scan --tag-key Environment --tag-value Production

# Filter by application tag
poetry run aws-inventory scan --tag-key app --tag-value web-server

# Filter by cost center in specific regions
poetry run aws-inventory scan --regions us-east-1 --tag-key CostCenter --tag-value Engineering
```

### Output Formats

```bash
# Default table format (human-readable)
poetry run aws-inventory scan --format table

# JSON format for programmatic processing
poetry run aws-inventory scan --format json --output results.json

# Markdown format for documentation
poetry run aws-inventory scan --format md --output report.md

# Export filtered results to JSON
poetry run aws-inventory scan --tag-key Environment --tag-value Production --format json --output prod-resources.json
```

### Advanced Options

```bash
# Dry run (preview without execution)
poetry run aws-inventory scan --dry-run --service ec2

# Disable caching for fresh data
poetry run aws-inventory scan --no-cache

# Configure worker threads for performance
poetry run aws-inventory scan --max-workers 10 --service-workers 6

# Continuous refresh mode with custom interval
poetry run aws-inventory scan --refresh --refresh-interval 30 --service ec2

# Debug mode with performance timing
poetry run aws-inventory --verbose scan --debug --max-workers 1 --service ec2
```

### Real-World Examples

```bash
# Production infrastructure audit with comprehensive logging
poetry run aws-inventory --verbose --log-file prod-audit.log scan \
    --debug --tag-key Environment --tag-value Production \
    --format json --output production-audit.json

# Regional compliance check with detailed tracing
poetry run aws-inventory --verbose --log-file compliance-trace.log scan \
    --debug --regions eu-west-1,eu-central-1 \
    --service ec2 --format md --output eu-compliance-report.md

# Application-specific resource discovery across all AWS services
poetry run aws-inventory scan \
    --all-services --tag-key Application --tag-value MyApp \
    --format table --regions us-east-1

# Development environment troubleshooting with verbose logging
poetry run aws-inventory --verbose --log-file dev-debug.log scan \
    --debug --regions us-west-2 \
    --tag-key Environment --tag-value Development \
    --no-cache --dry-run

# Performance analysis with sequential processing
poetry run aws-inventory --verbose --log-file perf-analysis.log scan \
    --debug --max-workers 1 --service-workers 1 \
    --service ec2 --service s3 --regions us-east-1
```

## 🔧 Configuration

### AWS Profile Configuration

```bash
# Set AWS profile
export AWS_PROFILE=your-profile-name

# Login via SSO
aws sso login --profile $AWS_PROFILE

# Verify credentials
aws sts get-caller-identity
```

### Logging Configuration

The scanner features a comprehensive logging system with multiple configuration options:

```bash
# Global logging options (apply to all commands)
poetry run aws-inventory --verbose --log-file /path/to/logfile.log scan --debug

# Debug modes explained:
# --debug: Enable debug mode with rich console output and file logging
# --verbose: Enable AWS API tracing (requires --debug)
# --log-file: Custom log file path (default: .debug_logs/aws_scanner_debug_TIMESTAMP.log)
```

Logging levels: normal runs show progress and results; `--debug` adds
execution traces, timing, and caller context; `--verbose` together with
`--debug` adds full AWS API tracing including HTTP requests and responses.

> [!Tip]
> **📖 Detailed Logging Guide**: For comprehensive logging documentation, configuration examples, and troubleshooting, see [Logging Architecture](docs/LOGGING_ARCHITECTURE.md).

### Performance Tuning

- `--max-workers` (1-20) controls region-level parallelism; `--service-workers` (1-10) controls service-level parallelism.
- `--cache` speeds up repeated scans (10-minute TTL); `--no-cache` forces fresh data.

> [!NOTE]
> Verbose logging adds roughly 10-20% overhead — use it for troubleshooting, not routine scans.

## 📁 **Project Structure**

```
aws-resource-inventory/
├── aws_scanner.py               # Scan orchestration across regions
├── cli.py                       # Command-line interface (typer)
├── setup.sh                     # Automated setup script
├── run_quick_tests.sh           # Smoke-test script
├── pyproject.toml               # Project configuration and dependencies
│
├── services/                    # One scanner module per AWS service
│   ├── registry.py              # Single source of truth: service → scanner + output processor
│   ├── ec2_service.py           # Instances, volumes, security groups, AMIs, snapshots
│   ├── s3_service.py            # Buckets (region-filtered, tag-enriched)
│   ├── ecs_service.py           # Clusters, services, task definitions, capacity providers
│   ├── efs_service.py           # File systems
│   ├── elb_service.py           # Load balancers, target groups, listeners, rules
│   ├── rds_service.py           # DB instances, clusters, snapshots (incl. Aurora)
│   ├── vpc_service.py           # VPC networking components
│   └── autoscaling_service.py   # ASGs, launch configurations, launch templates
│
├── aws_scanner_lib/             # Core library modules
│   ├── engine.py                # Shared scanning engine: pagination, concurrency, error policy
│   ├── records.py               # Resource — the typed record every output consumes
│   ├── clients.py               # The only boto3 client factory (pooling, adaptive retries)
│   ├── scan.py                  # Region/service scan orchestration + caching hooks
│   ├── outputs.py               # Table / JSON / Markdown output processing
│   ├── resource_groups_utils.py # Resource Groups Tagging API path (--all-services, tags)
│   ├── cache.py                 # Result caching with TTL
│   └── logging.py               # Unified logging with AWS API tracing
│
├── scripts/
│   └── e2e-diff.sh              # Before/after functional comparison against real AWS
│
├── docs/                        # Documentation
│   ├── adr/                     # Architecture decision records
│   ├── Architecture.md          # System architecture and design patterns
│   └── LOGGING_ARCHITECTURE.md  # Logging system documentation
│
└── tests/                       # Test suite — runs with ZERO AWS credentials (moto)
```

## 🧪 Testing

> [!NOTE]
> The suite needs no AWS credentials: moto fakes AWS, and the test fixtures
> force fake credentials so no test can ever touch a real account.

```bash
# Run the full suite (coverage is reported automatically)
poetry run pytest

# Run one test file
poetry run pytest tests/test_engine.py

# Quick smoke checks (help, dry-run, formats)
./run_quick_tests.sh
```

After merging a change that touches scan behaviour, verify against real
AWS with the before/after comparison script:

```bash
scripts/e2e-diff.sh                        # compares origin/main~1 vs origin/main
scripts/e2e-diff.sh --tag-key team         # exercise the tag-scan path too
```

## 🐛 Troubleshooting

### Common Issues

1. **AWS Credentials**: Ensure AWS credentials are properly configured
   ```bash
   aws sts get-caller-identity
   ```

2. **Python Version**: Verify Python 3.10+ is installed
   ```bash
   python3 --version
   ```

3. **Dependencies**: Reinstall dependencies if needed
   ```bash
   poetry install --no-cache
   ```

4. **Permissions**: Ensure your AWS user/role has necessary permissions for the services you're scanning

### Debug and Logging Troubleshooting

The advanced logging system provides powerful debugging capabilities:

```bash
# Basic debug information
poetry run aws-inventory scan --debug --dry-run

# Verbose AWS API tracing for troubleshooting
poetry run aws-inventory --verbose --log-file debug-trace.log scan --debug --service ec2

# Check debug log files (automatically created)
ls .debug_logs/

# Monitor real-time logging
tail -f .debug_logs/aws_scanner_debug_*.log

# Filter AWS API calls
grep -E "(boto|botocore|HTTP)" .debug_logs/aws_scanner_debug_*.log
```

Common logging issues: no AWS API logs usually means one of `--debug` or
`--verbose` is missing (both are required, and `--verbose` goes before
`scan`); for sequential, readable debugging use `--max-workers 1`; if log
files are missing, check permissions on the `.debug_logs/` directory.

> [!Tip]
> **🔧 Advanced Troubleshooting**: For detailed logging troubleshooting and configuration options, see [Logging Architecture - Troubleshooting Section](docs/LOGGING_ARCHITECTURE.md#troubleshooting).

## 📌 Quick Reference

### Most Common Commands

```bash
# Quick scan with basic output
poetry run aws-inventory scan --regions us-east-1

# Debug mode with detailed logging
poetry run aws-inventory scan --debug --regions us-east-1

# Full AWS API tracing (development/troubleshooting)
poetry run aws-inventory --verbose --log-file trace.log scan --debug --service ec2

# Tag-based filtering across all AWS services
poetry run aws-inventory scan --all-services --tag-key Environment --tag-value Production

# Production audit with comprehensive logging
poetry run aws-inventory --verbose --log-file audit.log scan --debug \
    --tag-key Environment --tag-value Production --format json --output audit.json
```

### Flag Combinations Guide

| Scenario         | Command Pattern                                          | Purpose                        |
|------------------|----------------------------------------------------------|--------------------------------|
| **Basic Scan**   | `poetry run aws-inventory scan`                            | Standard resource discovery    |
| **Debug Mode**   | `poetry run aws-inventory scan --debug`                    | Detailed execution information |
| **API Tracing**  | `poetry run aws-inventory --verbose scan --debug`          | Full AWS API call logging      |
| **Custom Logs**  | `poetry run aws-inventory --log-file path scan --debug`    | Custom log file location       |
| **All Services** | `poetry run aws-inventory scan --all-services --tag-key X` | Discover 100+ AWS services     |

## 📖 Documentation

The project includes comprehensive documentation covering all aspects of the system:

- [Architecture Documentation](docs/Architecture.md) — system design, component interactions, and patterns
- [Architecture Decision Records](docs/adr/) — why things are built the way they are
- [Shell Completion](docs/SHELL_COMPLETION.md) — tab-completion setup for the CLI
- [Logging Architecture](docs/LOGGING_ARCHITECTURE.md) — the logging system: API tracing, configuration, integration patterns, troubleshooting, performance

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch: `git checkout -b feature/amazing-feature`
3. Install development dependencies: Use [`./setup.sh`](./setup.sh)
4. Set up pre-commit hooks: `pre-commit install --install-hooks`
5. Make your changes and run tests: `poetry run pytest`
6. Test logging changes: `poetry run aws-inventory --verbose scan --debug --dry-run`
7. Commit with a conventional-commit message (titles feed the release changelog): `git commit -m 'feat: add amazing feature'`
8. Push to the branch: `git push origin feature/amazing-feature`
9. Open a Pull Request

> [!TIP]
> **Development Notes:**
> - Use the debug and verbose flags extensively during development
> - Check the logging architecture documentation when modifying logging behavior
> - Ensure all new features include appropriate logging and error handling

## 📝 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- Built with [Typer](https://typer.tiangolo.com/) for CLI interface
- Styled with [Rich](https://rich.readthedocs.io/) for beautiful console output
- Powered by [Boto3](https://boto3.amazonaws.com/) for AWS integration
- Managed with [Poetry](https://python-poetry.org/) for dependency management

---

> _Made with ❤️ for the AWS community_

