Metadata-Version: 2.4
Name: ibm_watsonx_data_integration_mcp
Version: 1.0.0rc1
Summary: MCP server for IBM watsonx.data Integration
License-File: LICENSE
Requires-Python: <3.13,>=3.11
Requires-Dist: beautifulsoup4
Requires-Dist: docling<2.95.0,>=2.94.0
Requires-Dist: fastmcp<3.4.0
Requires-Dist: httpx
Requires-Dist: ibm-watsonx-data-integration>=1.1.0
Requires-Dist: lancedb
Requires-Dist: mcp[cli]
Requires-Dist: pyjwt[crypto]>=2.13.0
Requires-Dist: requests
Requires-Dist: sentence-transformers
Requires-Dist: starlette>=1.2.1
Description-Content-Type: text/markdown

# IBM watsonx.data Integration MCP Server

An MCP server built on top of the `ibm-watsonx-data-integration` SDK with intelligent documentation retrieval using vector search.

---

## Requirements

- **Python**: 3.11-3.12 (3.12 recommended)

---

## Setup Instructions

There are three ways to run this MCP server:

### Option 1: Using `uvx` (Recommended)

The fastest way to run the server without installation.

**Verify Installation:**
```bash
uvx --python 3.12 --from git+ssh://git@github.ibm.com/watsonx-data-integration/di-mcp.git data-intg-mcp --help
```

**MCP Server Configuration:**
```json
{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "uvx",
      "args": [
        "--python",
        "3.12",
        "--from",
        "git+ssh://git@github.ibm.com/watsonx-data-integration/di-mcp.git",
        "data-intg-mcp"
      ],
      "env": {
        "WATSONX_API_KEY": "API_KEY"
      }
    }
  }
}
```

---

### Option 2: Using `uv` as a Tool

Install and run as a `uv` tool.

**Setup:**
```bash
uv tool install --python 3.12 git+ssh://git@github.ibm.com/watsonx-data-integration/di-mcp.git
```

**Verify Installation:**
```bash
uv tool run --python 3.12 data-intg-mcp --help
```

**MCP Server Configuration:**
```json
{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "uv",
      "args": [
        "tool",
        "run",
        "data-intg-mcp"
      ],
      "env": {
        "WATSONX_API_KEY": "API_KEY"
      }
    }
  }
}
```

---

### Option 3: Using `pip install` with Virtual Environment

Install the package in a virtual environment.

**Setup:**
```bash
python3.12 -m venv .venv
source .venv/bin/activate
pip install git+ssh://git@github.ibm.com/watsonx-data-integration/di-mcp.git
```

**Verify Installation:**
```bash
data-intg-mcp --help
```

**MCP Server Configuration:**
```json
{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "<Path to .venv>/bin/data-intg-mcp",
      "args": [],
      "env": {
        "WATSONX_API_KEY": "API_KEY"
      }
    }
  }
}
```

---

## MCP Client Integration

### 🤖 Claude Desktop Integration

To connect this MCP server to Claude Desktop:

Navigate to: **Claude → Settings → Developer → Edit Config**

Add one of the MCP server configurations above to your `claude_desktop_config.json` file.

---

### 🤖 Bob (VS Code) Integration

To connect this MCP server to Bob via VS Code:

Navigate to: **Open VS Code → Open Bob → Click the ⚙️ Settings icon → Navigate to the MCP section → Click Open next to Global MCPs**

Add one of the MCP server configurations above to your Bob MCP config file.

---

## Configuration Notes

### Windows UTF-8 Configuration

**Important for Windows Users:** Due to a limitation in fastmcp 3.2.4, the server requires UTF-8 encoding to properly load skill files that contain emoji and special characters. On Windows, you must set the `PYTHONUTF8` environment variable:

```json
{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "PYTHONUTF8": "1",
        "WATSONX_API_KEY": "your-api-key"
      }
    }
  }
}
```

**Why is this needed?**
- Windows defaults to cp1252 encoding, which cannot decode UTF-8 emoji (✅, ❌, →) in SKILL.md files
- Setting `PYTHONUTF8=1` forces Python to use UTF-8 encoding for all file operations
- Without this setting, the server will fail to start on Windows with a `UnicodeDecodeError`

**This applies to all installation methods** (uvx, uv tool, pip install) when running on Windows.

---

### Authentication

The server supports both **SaaS** (IBM Cloud) and **On-Premises** (Cloud Pak for Data) authentication. It automatically detects which authentication method to use based on the environment variables you set.

#### SaaS Authentication (IBM Cloud)

For SaaS deployments, set the `WATSONX_API_KEY` environment variable:

```json
{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "WATSONX_API_KEY": "your-ibm-cloud-api-key"
      }
    }
  }
}
```

**Region Selection:**

When using SaaS authentication, the AI assistant should ask you to specify your IBM Cloud region when needed. Generated examples should not hardcode a specific region; templates should use a neutral placeholder such as `YOUR_REGION`. Available regions (from the `IBMCloudRegion` enum):

- `TORONTO` - Toronto, Canada
- `DALLAS` - Dallas, USA
- `FRANKFURT` - Frankfurt, Germany
- `LONDON` - London, United Kingdom
- `TOKYO` - Tokyo, Japan
- `SYDNEY` - Sydney, Australia

The region is passed as a parameter to the tools that require it, ensuring you're connecting to the correct regional endpoint.

#### On-Premises Authentication (Cloud Pak for Data)

For On-Premises deployments, you have two options:

**Option 1: Username + Password**
```json
{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "CP4D_USERNAME": "your-username",
        "CP4D_PASSWORD": "your-password",
        "CP4D_URL": "https://your-cp4d-cluster.com",
        "CP4D_DISABLE_SSL_VERIFICATION": "false"
      }
    }
  }
}
```

**Option 2: Username + Zen API Key**
```json
{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "CP4D_USERNAME": "your-username",
        "ZEN_API_KEY": "your-zen-api-key",
        "CP4D_URL": "https://your-cp4d-cluster.com",
        "CP4D_DISABLE_SSL_VERIFICATION": "false"
      }
    }
  }
}
```

**Note on SSL Verification:**
- `CP4D_DISABLE_SSL_VERIFICATION`: Set to `"true"` to disable SSL certificate verification for On-Premises deployments (useful for self-signed certificates). Default is `"false"`.
- **Security Warning**: Only disable SSL verification in development/testing environments. Always use proper SSL certificates in production.

#### Configuring TLS for On-Premises Deployments

If your On-Premises cluster uses self-signed certificates or custom Certificate Authorities (CA), you may need to configure TLS properly instead of disabling SSL verification. Here's how to set up TLS:

**Step 1: Log in to your OpenShift cluster**
```bash
oc login <CLUSTER_URL>
```

**Step 2: Extract the cluster's CA certificate**
```bash
openssl s_client -showcerts \
  -connect <CLUSTER>:443 </dev/null > ca.crt
```

Replace `<CLUSTER>` with your cluster hostname (e.g., `your-cp4d-cluster.com`).

**Step 3: Clean up the certificate file**

Open the `ca.crt` file and remove all text except the certificate blocks. The file should only contain lines like:
```
-----BEGIN CERTIFICATE-----
[certificate content]
-----END CERTIFICATE-----
```

Remove any connection information, verification messages, or other output from the openssl command.

**Step 4: Configure the MCP server to use the CA certificate**

Add the `REQUESTS_CA_BUNDLE` environment variable to your MCP server configuration, pointing to the cleaned `ca.crt` file:

```json
{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "CP4D_USERNAME": "your-username",
        "CP4D_PASSWORD": "your-password",
        "CP4D_URL": "https://your-cp4d-cluster.com",
        "CP4D_DISABLE_SSL_VERIFICATION": "false",
        "REQUESTS_CA_BUNDLE": "/path/to/ca.crt"
      }
    }
  }
}
```

Replace `/path/to/ca.crt` with the absolute path to your cleaned certificate file.

**Authentication Detection Priority:**
1. SaaS (IAMAuthenticator) - if `WATSONX_API_KEY` is set
2. On-Premises (ICP4DAuthenticator) - if `CP4D_USERNAME` and `CP4D_PASSWORD` are set
3. On-Premises (ZenApiKeyAuthenticator) - if `CP4D_USERNAME` and `ZEN_API_KEY` are set

The server validates authentication credentials on startup and will display which authentication method was detected.

---

## Usage

### Offline Documentation Mode

If your environment cannot access GitHub Pages, start the server with [`--use-local-docs`](src/di_mcp/main.py:291). In this mode, the server reads packaged HTML files from [`src/di_mcp/bundled_docs`](src/di_mcp/bundled_docs), converts them into cached markdown files, and then builds the lanceDB index in the normal runtime cache.

Example:
```bash
data-intg-mcp --use-local-docs
```

Important:
- Offline docs may not be the most up to date because they do not use the live documentation site.
- Refresh the packaged docs periodically before publishing or deploying if you want the bundled copy to stay current.

To refresh the packaged HTML docs, run:
```bash
python tools/download_docs.py
```

This downloads the latest SDK HTML pages and stores them under [`src/di_mcp/bundled_docs/html`](src/di_mcp/bundled_docs) so they can be packaged with the server.

### Available Tools

The MCP server provides the following tools:

#### Documentation Tools

1. **`search_sdk_documentation`**: Search SDK documentation using semantic similarity
   - Use for conceptual questions and "how to" guidance
   - Searches across all indexed SDK documentation
   - Returns relevant sections with source information
   - Example: "How do I configure a Kafka Consumer?"

2. **`get_model_reference`**: Look up exact model definitions
   - Use for exact field lists, enum/Literal values, and type signatures
   - Returns authoritative reference for a specific SDK model class
   - Includes complete field definitions with types and defaults
   - Strips Pydantic boilerplate methods for clarity
   - Provides fuzzy suggestions if class name is not found
   - Example: `get_model_reference(class_name='Schedule')` returns all valid values for `repeat_mode` field

**When to use which tool:**
- Use `search_sdk_documentation` for: "How do I...", conceptual explanations, examples, tutorials
- Use `get_model_reference` for: "What are the valid values for...", "What fields does X have?", exact type signatures

#### Version Tools

3. **`get_mcp_version`**: Get the current version of the MCP server
4. **`get_sdk_version`**: Get the version of the IBM watsonx.data Integration SDK

#### Resource Tools

5. **`list_resources`**: List all available MCP resources (via ResourcesAsTools transform)
6. **`read_resource`**: Read a specific resource by URI (via ResourcesAsTools transform)

#### Execution Tools

7. **`execute_script`**: Execute generated Python scripts for validation (stdio only, requires auth)

#### Stage Discovery Tools (Streaming)

8. **`list_available_streaming_stages`**: List available streaming stages (stdio only, requires auth)
   - Returns all available stage information for a streaming flow
   - Critical for determining available stages when creating streaming flows

9. **`list_all_available_stage_configurations_streaming`**: Get streaming stage configurations (stdio only, requires auth)
   - Returns comprehensive configuration information for specified streaming stage types
   - Includes all available properties, their types, and default values
   - Critical for determining available properties for streaming flow stages

#### Stage Discovery Tools (Batch)

10. **`list_available_batch_stages`**: List available batch stages (stdio only, requires auth)
    - Returns all available stage information for a batch flow
    - Critical for determining available stages when creating batch flows

11. **`list_all_available_stage_configurations_batch`**: Get batch stage configurations (stdio only, requires auth)
    - Returns comprehensive configuration information for specified batch stage types
    - Includes all available properties, their types, and default values
    - Critical for determining available properties for batch flow stages

---

### Available Resources

The MCP server provides several resources that contain best practices, templates, and guidance:

#### Best Practices Resource

- **`watsonx://best_practices`**: Canonical usage rules, sequencing patterns, and best practices for the IBM watsonx.data Integration SDK
  - Core concepts and platform-centric architecture
  - Authentication patterns
  - Configuration management
  - Streaming and batch flow best practices
  - Common invalid patterns to avoid
  - Code generation best practices

#### Intent Document Resources

Intent documents are **MANDATORY** when creating batch or streaming flows. They ensure complete requirements capture before code generation.

- **`intent://overview`**: Overview of the intent document generation approach
- **`intent://batch-flow`**: Template and instructions for batch flow intent documents
- **`intent://streaming-flow`**: Template and instructions for streaming flow intent documents

**Workflow for flow creation:**
1. Read the appropriate intent resource
2. Generate a complete intent document (must include Mermaid diagram)
3. Get user approval
4. Generate and execute Python code

#### Skills Resources

Skills provide authoritative patterns and complete working examples for specific operations:

- **`skill://platform/SKILL.md`**: Platform operations and authentication
- **`skill://project/SKILL.md`**: Project creation, listing, and management
- **`skill://batch-flows/SKILL.md`**: Batch flow operations and configuration
- **`skill://streaming-flows/SKILL.md`**: Streaming flow operations and configuration
- **`skill://jobs-and-job-runs/SKILL.md`**: Job creation, execution, and monitoring

---

## Testing

### Running Tests

The project includes comprehensive unit tests for all modules. To run the tests:

```bash
# Install test dependencies
uv pip install -e ".[dev]"

# Run all unit tests with coverage (using poe tasks)
uv run poe unit

# Run unit tests serially (useful for debugging)
uv run poe unit-serial

# Run specific test file or pattern
uv run poe test tests/unit/test_crawler.py
uv run poe test tests/unit/test_main.py -k test_name

# Run tests for specific Python versions
uv run poe unit311  # Python 3.11
uv run poe unit312  # Python 3.12

# Or use pytest directly
pytest tests/unit/
pytest --cov=src/di_mcp --cov-report=html --cov-report=term-missing tests/unit/
```

### Running Integration Tests

**Option 1: Ollama**
```bash
ollama run qwen3-coder:30b

uv run --group integration-local poe integration
```

**Option 2: Claude**
```bash
export AWS_ACCESS_KEY_ID=<key_id>
export AWS_SECRET_ACCESS_KEY=<access_key>

uv run --group integration-remote poe integration-claude
```



### Coverage

After running tests with coverage, open `htmlcov/index.html` in your browser to view detailed coverage reports.

## Advanced Usage

### Setting PYTHONPATH for Local SDK Development

If you're developing or testing with a local version of the IBM watsonx.data Integration SDK, you can set the `PYTHONPATH` environment variable to point to your local SDK directory:

```json
{
  "mcpServers": {
    "data-intg-mcp": {
      "command": "...",
      "args": [...],
      "env": {
        "PYTHONPATH": "<Path to SDK>",
        "WATSONX_API_KEY": "API_KEY"
      }
    }
  }
}
```

Replace `<Path to SDK>` with the absolute path to your local SDK directory.

### Cache Management

The server caches documentation and creates a vector database for efficient retrieval. To clear all cached data:

```bash
data-intg-mcp --clear-cache
```

This removes:
- `resources_cache/` - Cached documentation files
- `raw_resource_html/` - Raw HTML files
- `lance_db/` - Vector database (LanceDB)

Use this command when you need to refresh the documentation cache or troubleshoot issues with the vector database.

### HTTP/SSE Transport

By default, the MCP server runs over stdio transport (for local MCP clients like Claude Desktop or Bob). You can also run it as an HTTP server using Server-Sent Events (SSE) transport for remote access:

```bash
# Run as HTTP/SSE server (API key is optional for SSE)
data-intg-mcp --transport sse --host 0.0.0.0 --port 8000

# If you need SDK operations that require authentication, set the API key:
export WATSONX_API_KEY="your-api-key" # pragma: allowlist secret
data-intg-mcp --transport sse --host 0.0.0.0 --port 8000
```

**Tool Availability by Transport:**

| Tool | stdio | SSE | Notes |
|------|-------|-----|-------|
| `search_sdk_documentation` | ✓ | ✓ | Semantic search across SDK docs |
| `get_model_reference` | ✓ | ✓ | Look up exact model definitions |
| `get_mcp_version` | ✓ | ✓ | Get MCP server version |
| `get_sdk_version` | ✓ | ✓ | Get SDK version |
| `list_resources` | ✓ | ✓ | List available resources |
| `read_resource` | ✓ | ✓ | Read specific resource by URI |
| `execute_script` | ✓ | ✗ | Disabled in SSE for security |
| `list_available_streaming_stages` | ✓ | ✗ | Requires authentication |
| `list_all_available_stage_configurations_streaming` | ✓ | ✗ | Requires authentication |
| `list_available_batch_stages` | ✓ | ✗ | Requires authentication |
| `list_all_available_stage_configurations_batch` | ✓ | ✗ | Requires authentication |

**Important Security Notes:**
- When using SSE transport, tools requiring authentication or local execution are automatically disabled
- The SSE server allows remote connections, so ensure proper network security measures are in place
- `WATSONX_API_KEY` is optional for SSE transport (only needed if using authenticated tools)
- For stdio transport, `WATSONX_API_KEY` is required
- Environment variables must be set on the server side, not in the client configuration
- Consider using authentication/authorization mechanisms when exposing the server over HTTP

**MCP Client Configuration for SSE:**

For remote MCP clients, configure the server URL:

```json
{
  "mcpServers": {
    "data-intg-mcp-remote": {
      "url": "http://127.0.0.1:8000/sse"
    }
  }
}
```

Note: Unlike stdio transport, environment variables cannot be passed from the client config when using SSE. They must be set in the server's environment before starting the server.

**Command Line Options:**
- `--transport`: Choose `stdio` (default) or `sse`
- `--host`: Host to bind to for SSE (default: 0.0.0.0, ignored for stdio)
- `--port`: Port to bind to for SSE (default: 8000, ignored for stdio)
- `--disable-execution-tools`: Disable execute_script and stage discovery tools (recommended for remote deployments)
- `--clear-cache`: Clear all cached documentation before starting
- `--use-local-docs`: Use bundled local HTML docs instead of retrieving documentation live
