Metadata-Version: 2.2
Name: jina
Version: 3.33.1
Summary: Multimodal AI services & pipelines with cloud-native stack: gRPC, Kubernetes, Docker, OpenTelemetry, Prometheus, Jaeger, etc.
Home-page: https://github.com/jina-ai/jina/
Download-URL: https://github.com/jina-ai/jina/tags
Author: Jina AI
Author-email: hello@jina.ai
License: Apache 2.0
Project-URL: Documentation, https://jina.ai/serve
Project-URL: Source, https://github.com/jina-ai/jina/
Project-URL: Tracker, https://github.com/jina-ai/jina/issues
Keywords: jina cloud-native cross-modal multimodal neural-search query search index elastic neural-network encoding embedding serving docker container image video audio deep-learning mlops
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Unix Shell
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Database :: Database Engines/Servers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: grpcio-reflection<=1.68.0,>=1.46.0
Requires-Dist: uvloop; platform_system != "Windows"
Requires-Dist: pyyaml>=5.3.1
Requires-Dist: packaging>=20.0
Requires-Dist: opentelemetry-exporter-otlp>=1.12.0
Requires-Dist: prometheus_client>=0.12.0
Requires-Dist: jina-hubble-sdk>=0.30.4
Requires-Dist: opentelemetry-instrumentation-grpc>=0.35b0
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.13.0
Requires-Dist: opentelemetry-exporter-prometheus>=0.33b0
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.33b0
Requires-Dist: uvicorn<=0.23.1
Requires-Dist: numpy
Requires-Dist: docarray>=0.16.4
Requires-Dist: requests
Requires-Dist: pydantic<2.0.0
Requires-Dist: jcloud>=0.0.35
Requires-Dist: filelock
Requires-Dist: pathspec
Requires-Dist: aiohttp
Requires-Dist: opentelemetry-sdk>=1.14.0
Requires-Dist: urllib3<2.0.0,>=1.25.9
Requires-Dist: python-multipart
Requires-Dist: opentelemetry-api>=1.12.0
Requires-Dist: grpcio-health-checking<=1.68.0,>=1.46.0
Requires-Dist: aiofiles
Requires-Dist: fastapi>=0.76.0
Requires-Dist: docker
Requires-Dist: grpcio<=1.68.0,>=1.46.0
Requires-Dist: opentelemetry-instrumentation-aiohttp-client>=0.33b0
Requires-Dist: websockets
Requires-Dist: protobuf>=3.19.0
Provides-Extra: core
Requires-Dist: grpcio-health-checking<=1.68.0,>=1.46.0; extra == "core"
Requires-Dist: opentelemetry-instrumentation-grpc>=0.35b0; extra == "core"
Requires-Dist: grpcio-reflection<=1.68.0,>=1.46.0; extra == "core"
Requires-Dist: pyyaml>=5.3.1; extra == "core"
Requires-Dist: numpy; extra == "core"
Requires-Dist: grpcio<=1.68.0,>=1.46.0; extra == "core"
Requires-Dist: packaging>=20.0; extra == "core"
Requires-Dist: docarray>=0.16.4; extra == "core"
Requires-Dist: pydantic<2.0.0; extra == "core"
Requires-Dist: urllib3<2.0.0,>=1.25.9; extra == "core"
Requires-Dist: jina-hubble-sdk>=0.30.4; extra == "core"
Requires-Dist: opentelemetry-api>=1.12.0; extra == "core"
Requires-Dist: jcloud>=0.0.35; extra == "core"
Requires-Dist: protobuf>=3.19.0; extra == "core"
Provides-Extra: numpy
Requires-Dist: numpy; extra == "numpy"
Provides-Extra: protobuf
Requires-Dist: protobuf>=3.19.0; extra == "protobuf"
Provides-Extra: grpcio
Requires-Dist: grpcio<=1.68.0,>=1.46.0; extra == "grpcio"
Provides-Extra: grpcio-reflection
Requires-Dist: grpcio-reflection<=1.68.0,>=1.46.0; extra == "grpcio-reflection"
Provides-Extra: grpcio-health-checking
Requires-Dist: grpcio-health-checking<=1.68.0,>=1.46.0; extra == "grpcio-health-checking"
Provides-Extra: pyyaml
Requires-Dist: pyyaml>=5.3.1; extra == "pyyaml"
Provides-Extra: packaging
Requires-Dist: packaging>=20.0; extra == "packaging"
Provides-Extra: docarray
Requires-Dist: docarray>=0.16.4; extra == "docarray"
Provides-Extra: jina-hubble-sdk
Requires-Dist: jina-hubble-sdk>=0.30.4; extra == "jina-hubble-sdk"
Provides-Extra: jcloud
Requires-Dist: jcloud>=0.0.35; extra == "jcloud"
Provides-Extra: opentelemetry-api
Requires-Dist: opentelemetry-api>=1.12.0; extra == "opentelemetry-api"
Provides-Extra: opentelemetry-instrumentation-grpc
Requires-Dist: opentelemetry-instrumentation-grpc>=0.35b0; extra == "opentelemetry-instrumentation-grpc"
Provides-Extra: devel
Requires-Dist: uvloop; platform_system != "Windows" and extra == "devel"
Requires-Dist: sgqlc; extra == "devel"
Requires-Dist: opentelemetry-exporter-otlp>=1.12.0; extra == "devel"
Requires-Dist: prometheus_client>=0.12.0; extra == "devel"
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.13.0; extra == "devel"
Requires-Dist: opentelemetry-exporter-prometheus>=0.33b0; extra == "devel"
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.33b0; extra == "devel"
Requires-Dist: uvicorn<=0.23.1; extra == "devel"
Requires-Dist: requests; extra == "devel"
Requires-Dist: strawberry-graphql>=0.96.0; extra == "devel"
Requires-Dist: filelock; extra == "devel"
Requires-Dist: pathspec; extra == "devel"
Requires-Dist: aiohttp; extra == "devel"
Requires-Dist: opentelemetry-sdk>=1.14.0; extra == "devel"
Requires-Dist: watchfiles>=0.18.0; extra == "devel"
Requires-Dist: python-multipart; extra == "devel"
Requires-Dist: aiofiles; extra == "devel"
Requires-Dist: fastapi>=0.76.0; extra == "devel"
Requires-Dist: docker; extra == "devel"
Requires-Dist: opentelemetry-instrumentation-aiohttp-client>=0.33b0; extra == "devel"
Requires-Dist: websockets; extra == "devel"
Provides-Extra: perf
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.13.0; extra == "perf"
Requires-Dist: opentelemetry-exporter-prometheus>=0.33b0; extra == "perf"
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.33b0; extra == "perf"
Requires-Dist: opentelemetry-sdk>=1.14.0; extra == "perf"
Requires-Dist: opentelemetry-exporter-otlp>=1.12.0; extra == "perf"
Requires-Dist: opentelemetry-instrumentation-aiohttp-client>=0.33b0; extra == "perf"
Requires-Dist: prometheus_client>=0.12.0; extra == "perf"
Requires-Dist: uvloop; extra == "perf"
Provides-Extra: standard
Requires-Dist: filelock; extra == "standard"
Requires-Dist: opentelemetry-exporter-prometheus>=0.33b0; extra == "standard"
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.33b0; extra == "standard"
Requires-Dist: pathspec; extra == "standard"
Requires-Dist: aiofiles; extra == "standard"
Requires-Dist: uvicorn<=0.23.1; extra == "standard"
Requires-Dist: fastapi>=0.76.0; extra == "standard"
Requires-Dist: docker; extra == "standard"
Requires-Dist: aiohttp; extra == "standard"
Requires-Dist: opentelemetry-sdk>=1.14.0; extra == "standard"
Requires-Dist: opentelemetry-exporter-otlp>=1.12.0; extra == "standard"
Requires-Dist: opentelemetry-instrumentation-aiohttp-client>=0.33b0; extra == "standard"
Requires-Dist: requests; extra == "standard"
Requires-Dist: websockets; extra == "standard"
Requires-Dist: python-multipart; extra == "standard"
Requires-Dist: prometheus_client>=0.12.0; extra == "standard"
Requires-Dist: uvloop; extra == "standard"
Provides-Extra: uvloop
Requires-Dist: uvloop; extra == "uvloop"
Provides-Extra: prometheus-client
Requires-Dist: prometheus_client>=0.12.0; extra == "prometheus-client"
Provides-Extra: opentelemetry-sdk
Requires-Dist: opentelemetry-sdk>=1.14.0; extra == "opentelemetry-sdk"
Provides-Extra: opentelemetry-exporter-otlp
Requires-Dist: opentelemetry-exporter-otlp>=1.12.0; extra == "opentelemetry-exporter-otlp"
Provides-Extra: opentelemetry-exporter-prometheus
Requires-Dist: opentelemetry-exporter-prometheus>=0.33b0; extra == "opentelemetry-exporter-prometheus"
Provides-Extra: opentelemetry-instrumentation-aiohttp-client
Requires-Dist: opentelemetry-instrumentation-aiohttp-client>=0.33b0; extra == "opentelemetry-instrumentation-aiohttp-client"
Provides-Extra: opentelemetry-instrumentation-fastapi
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.33b0; extra == "opentelemetry-instrumentation-fastapi"
Provides-Extra: standrad
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.13.0; extra == "standrad"
Provides-Extra: opentelemetry-exporter-otlp-proto-grpc
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.13.0; extra == "opentelemetry-exporter-otlp-proto-grpc"
Provides-Extra: fastapi
Requires-Dist: fastapi>=0.76.0; extra == "fastapi"
Provides-Extra: uvicorn
Requires-Dist: uvicorn<=0.23.1; extra == "uvicorn"
Provides-Extra: docker
Requires-Dist: docker; extra == "docker"
Provides-Extra: pathspec
Requires-Dist: pathspec; extra == "pathspec"
Provides-Extra: filelock
Requires-Dist: filelock; extra == "filelock"
Provides-Extra: requests
Requires-Dist: requests; extra == "requests"
Provides-Extra: websockets
Requires-Dist: websockets; extra == "websockets"
Provides-Extra: pydantic
Requires-Dist: pydantic<2.0.0; extra == "pydantic"
Provides-Extra: python-multipart
Requires-Dist: python-multipart; extra == "python-multipart"
Provides-Extra: aiofiles
Requires-Dist: aiofiles; extra == "aiofiles"
Provides-Extra: aiohttp
Requires-Dist: aiohttp; extra == "aiohttp"
Provides-Extra: test
Requires-Dist: pytest<8.0.0; extra == "test"
Requires-Dist: kubernetes<31.0.0,>=18.20.0; extra == "test"
Requires-Dist: opentelemetry-test-utils>=0.33b0; extra == "test"
Requires-Dist: pytest-lazy-fixture; extra == "test"
Requires-Dist: scipy>=1.6.1; extra == "test"
Requires-Dist: mock; extra == "test"
Requires-Dist: Pillow; extra == "test"
Requires-Dist: prometheus-api-client>=0.5.1; extra == "test"
Requires-Dist: pytest-reraise; extra == "test"
Requires-Dist: pytest-mock; extra == "test"
Requires-Dist: pytest-repeat; extra == "test"
Requires-Dist: pytest-custom_exit_code; extra == "test"
Requires-Dist: black==24.3.0; extra == "test"
Requires-Dist: psutil; extra == "test"
Requires-Dist: requests-mock; extra == "test"
Requires-Dist: pytest-kind==22.11.1; extra == "test"
Requires-Dist: coverage==6.2; extra == "test"
Requires-Dist: pytest-cov==3.0.0; extra == "test"
Requires-Dist: pytest-timeout; extra == "test"
Requires-Dist: flaky; extra == "test"
Requires-Dist: pytest-asyncio<0.23.0; extra == "test"
Provides-Extra: scipy
Requires-Dist: scipy>=1.6.1; extra == "scipy"
Provides-Extra: pillow
Requires-Dist: Pillow; extra == "pillow"
Provides-Extra: pytest
Requires-Dist: pytest<8.0.0; extra == "pytest"
Provides-Extra: pytest-timeout
Requires-Dist: pytest-timeout; extra == "pytest-timeout"
Provides-Extra: pytest-mock
Requires-Dist: pytest-mock; extra == "pytest-mock"
Provides-Extra: pytest-cov
Requires-Dist: pytest-cov==3.0.0; extra == "pytest-cov"
Provides-Extra: coverage
Requires-Dist: coverage==6.2; extra == "coverage"
Provides-Extra: pytest-repeat
Requires-Dist: pytest-repeat; extra == "pytest-repeat"
Provides-Extra: pytest-asyncio
Requires-Dist: pytest-asyncio<0.23.0; extra == "pytest-asyncio"
Provides-Extra: pytest-reraise
Requires-Dist: pytest-reraise; extra == "pytest-reraise"
Provides-Extra: flaky
Requires-Dist: flaky; extra == "flaky"
Provides-Extra: mock
Requires-Dist: mock; extra == "mock"
Provides-Extra: requests-mock
Requires-Dist: requests-mock; extra == "requests-mock"
Provides-Extra: pytest-custom-exit-code
Requires-Dist: pytest-custom_exit_code; extra == "pytest-custom-exit-code"
Provides-Extra: black
Requires-Dist: black==24.3.0; extra == "black"
Provides-Extra: kubernetes
Requires-Dist: kubernetes<31.0.0,>=18.20.0; extra == "kubernetes"
Provides-Extra: pytest-kind
Requires-Dist: pytest-kind==22.11.1; extra == "pytest-kind"
Provides-Extra: pytest-lazy-fixture
Requires-Dist: pytest-lazy-fixture; extra == "pytest-lazy-fixture"
Provides-Extra: torch
Requires-Dist: torch; extra == "torch"
Provides-Extra: cicd
Requires-Dist: sgqlc; extra == "cicd"
Requires-Dist: jsonschema; extra == "cicd"
Requires-Dist: tensorflow>=2.0; extra == "cicd"
Requires-Dist: torch; extra == "cicd"
Requires-Dist: bs4; extra == "cicd"
Requires-Dist: strawberry-graphql>=0.96.0; extra == "cicd"
Requires-Dist: portforward<0.4.3,>=0.2.4; extra == "cicd"
Provides-Extra: psutil
Requires-Dist: psutil; extra == "psutil"
Provides-Extra: strawberry-graphql
Requires-Dist: strawberry-graphql>=0.96.0; extra == "strawberry-graphql"
Provides-Extra: sgqlc
Requires-Dist: sgqlc; extra == "sgqlc"
Provides-Extra: bs4
Requires-Dist: bs4; extra == "bs4"
Provides-Extra: jsonschema
Requires-Dist: jsonschema; extra == "jsonschema"
Provides-Extra: portforward
Requires-Dist: portforward<0.4.3,>=0.2.4; extra == "portforward"
Provides-Extra: tensorflow
Requires-Dist: tensorflow>=2.0; extra == "tensorflow"
Provides-Extra: opentelemetry-test-utils
Requires-Dist: opentelemetry-test-utils>=0.33b0; extra == "opentelemetry-test-utils"
Provides-Extra: prometheus-api-client
Requires-Dist: prometheus-api-client>=0.5.1; extra == "prometheus-api-client"
Provides-Extra: watchfiles
Requires-Dist: watchfiles>=0.18.0; extra == "watchfiles"
Provides-Extra: urllib3
Requires-Dist: urllib3<2.0.0,>=1.25.9; extra == "urllib3"
Provides-Extra: all
Requires-Dist: grpcio-reflection<=1.68.0,>=1.46.0; extra == "all"
Requires-Dist: uvloop; platform_system != "Windows" and extra == "all"
Requires-Dist: pyyaml>=5.3.1; extra == "all"
Requires-Dist: sgqlc; extra == "all"
Requires-Dist: pytest<8.0.0; extra == "all"
Requires-Dist: packaging>=20.0; extra == "all"
Requires-Dist: kubernetes<31.0.0,>=18.20.0; extra == "all"
Requires-Dist: opentelemetry-exporter-otlp>=1.12.0; extra == "all"
Requires-Dist: opentelemetry-test-utils>=0.33b0; extra == "all"
Requires-Dist: pytest-lazy-fixture; extra == "all"
Requires-Dist: scipy>=1.6.1; extra == "all"
Requires-Dist: portforward<0.4.3,>=0.2.4; extra == "all"
Requires-Dist: mock; extra == "all"
Requires-Dist: prometheus_client>=0.12.0; extra == "all"
Requires-Dist: jina-hubble-sdk>=0.30.4; extra == "all"
Requires-Dist: opentelemetry-instrumentation-grpc>=0.35b0; extra == "all"
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.13.0; extra == "all"
Requires-Dist: opentelemetry-exporter-prometheus>=0.33b0; extra == "all"
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.33b0; extra == "all"
Requires-Dist: uvicorn<=0.23.1; extra == "all"
Requires-Dist: Pillow; extra == "all"
Requires-Dist: tensorflow>=2.0; extra == "all"
Requires-Dist: prometheus-api-client>=0.5.1; extra == "all"
Requires-Dist: pytest-reraise; extra == "all"
Requires-Dist: numpy; extra == "all"
Requires-Dist: torch; extra == "all"
Requires-Dist: docarray>=0.16.4; extra == "all"
Requires-Dist: requests; extra == "all"
Requires-Dist: pydantic<2.0.0; extra == "all"
Requires-Dist: strawberry-graphql>=0.96.0; extra == "all"
Requires-Dist: pytest-mock; extra == "all"
Requires-Dist: pytest-repeat; extra == "all"
Requires-Dist: pytest-custom_exit_code; extra == "all"
Requires-Dist: jcloud>=0.0.35; extra == "all"
Requires-Dist: filelock; extra == "all"
Requires-Dist: black==24.3.0; extra == "all"
Requires-Dist: psutil; extra == "all"
Requires-Dist: pathspec; extra == "all"
Requires-Dist: jsonschema; extra == "all"
Requires-Dist: requests-mock; extra == "all"
Requires-Dist: aiohttp; extra == "all"
Requires-Dist: pytest-kind==22.11.1; extra == "all"
Requires-Dist: opentelemetry-sdk>=1.14.0; extra == "all"
Requires-Dist: urllib3<2.0.0,>=1.25.9; extra == "all"
Requires-Dist: watchfiles>=0.18.0; extra == "all"
Requires-Dist: coverage==6.2; extra == "all"
Requires-Dist: bs4; extra == "all"
Requires-Dist: python-multipart; extra == "all"
Requires-Dist: opentelemetry-api>=1.12.0; extra == "all"
Requires-Dist: grpcio-health-checking<=1.68.0,>=1.46.0; extra == "all"
Requires-Dist: aiofiles; extra == "all"
Requires-Dist: fastapi>=0.76.0; extra == "all"
Requires-Dist: docker; extra == "all"
Requires-Dist: grpcio<=1.68.0,>=1.46.0; extra == "all"
Requires-Dist: opentelemetry-instrumentation-aiohttp-client>=0.33b0; extra == "all"
Requires-Dist: websockets; extra == "all"
Requires-Dist: pytest-cov==3.0.0; extra == "all"
Requires-Dist: pytest-timeout; extra == "all"
Requires-Dist: flaky; extra == "all"
Requires-Dist: pytest-asyncio<0.23.0; extra == "all"
Requires-Dist: protobuf>=3.19.0; extra == "all"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: download-url
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: summary

# Jina-Serve
<a href="https://pypi.org/project/jina/"><img alt="PyPI" src="https://img.shields.io/pypi/v/jina?label=Release&style=flat-square"></a>
<a href="https://discord.jina.ai"><img src="https://img.shields.io/discord/1106542220112302130?logo=discord&logoColor=white&style=flat-square"></a>
<a href="https://pypistats.org/packages/jina"><img alt="PyPI - Downloads from official pypistats" src="https://img.shields.io/pypi/dm/jina?style=flat-square"></a>
<a href="https://github.com/jina-ai/jina/actions/workflows/cd.yml"><img alt="Github CD status" src="https://github.com/jina-ai/jina/actions/workflows/cd.yml/badge.svg"></a>

Jina-serve is a framework for building and deploying AI services that communicate via gRPC, HTTP and WebSockets. Scale your services from local development to production while focusing on your core logic.

## Key Features

- Native support for all major ML frameworks and data types
- High-performance service design with scaling, streaming, and dynamic batching
- LLM serving with streaming output
- Built-in Docker integration and Executor Hub
- One-click deployment to Jina AI Cloud
- Enterprise-ready with Kubernetes and Docker Compose support

<details>
<summary><strong>Comparison with FastAPI</strong></summary>

Key advantages over FastAPI:

- DocArray-based data handling with native gRPC support
- Built-in containerization and service orchestration
- Seamless scaling of microservices
- One-command cloud deployment
</details>

## Install 

```bash
pip install jina
```

See guides for [Apple Silicon](https://jina.ai/serve/get-started/install/apple-silicon-m1-m2/) and [Windows](https://jina.ai/serve/get-started/install/windows/).

## Core Concepts

Three main layers:
- **Data**: BaseDoc and DocList for input/output
- **Serving**: Executors process Documents, Gateway connects services
- **Orchestration**: Deployments serve Executors, Flows create pipelines

## Build AI Services

Let's create a gRPC-based AI service using StableLM:

```python
from jina import Executor, requests
from docarray import DocList, BaseDoc
from transformers import pipeline


class Prompt(BaseDoc):
    text: str


class Generation(BaseDoc):
    prompt: str
    text: str


class StableLM(Executor):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.generator = pipeline(
            'text-generation', model='stabilityai/stablelm-base-alpha-3b'
        )

    @requests
    def generate(self, docs: DocList[Prompt], **kwargs) -> DocList[Generation]:
        generations = DocList[Generation]()
        prompts = docs.text
        llm_outputs = self.generator(prompts)
        for prompt, output in zip(prompts, llm_outputs):
            generations.append(Generation(prompt=prompt, text=output))
        return generations
```

Deploy with Python or YAML:

```python
from jina import Deployment
from executor import StableLM

dep = Deployment(uses=StableLM, timeout_ready=-1, port=12345)

with dep:
    dep.block()
```

```yaml
jtype: Deployment
with:
 uses: StableLM
 py_modules:
   - executor.py
 timeout_ready: -1
 port: 12345
```

Use the client:

```python
from jina import Client
from docarray import DocList
from executor import Prompt, Generation

prompt = Prompt(text='suggest an interesting image generation prompt')
client = Client(port=12345)
response = client.post('/', inputs=[prompt], return_type=DocList[Generation])
```

## Build Pipelines

Chain services into a Flow:

```python
from jina import Flow

flow = Flow(port=12345).add(uses=StableLM).add(uses=TextToImage)

with flow:
    flow.block()
```

## Scaling and Deployment

### Local Scaling

Boost throughput with built-in features:
- Replicas for parallel processing
- Shards for data partitioning
- Dynamic batching for efficient model inference

Example scaling a Stable Diffusion deployment:

```yaml
jtype: Deployment
with:
 uses: TextToImage
 timeout_ready: -1
 py_modules:
   - text_to_image.py
 env:
  CUDA_VISIBLE_DEVICES: RR
 replicas: 2
 uses_dynamic_batching:
   /default:
     preferred_batch_size: 10
     timeout: 200
```

### Cloud Deployment

#### Containerize Services

1. Structure your Executor:
```
TextToImage/
├── executor.py
├── config.yml
├── requirements.txt
```

2. Configure:
```yaml
# config.yml
jtype: TextToImage
py_modules:
 - executor.py
metas:
 name: TextToImage
 description: Text to Image generation Executor
```

3. Push to Hub:
```bash
jina hub push TextToImage
```

#### Deploy to Kubernetes
```bash
jina export kubernetes flow.yml ./my-k8s
kubectl apply -R -f my-k8s
```

#### Use Docker Compose
```bash
jina export docker-compose flow.yml docker-compose.yml
docker-compose up
```

#### JCloud Deployment

Deploy with a single command:
```bash
jina cloud deploy jcloud-flow.yml
```

## LLM Streaming

Enable token-by-token streaming for responsive LLM applications:

1. Define schemas:
```python
from docarray import BaseDoc


class PromptDocument(BaseDoc):
    prompt: str
    max_tokens: int


class ModelOutputDocument(BaseDoc):
    token_id: int
    generated_text: str
```

2. Initialize service:
```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel


class TokenStreamingExecutor(Executor):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.model = GPT2LMHeadModel.from_pretrained('gpt2')
```

3. Implement streaming:
```python
@requests(on='/stream')
async def task(self, doc: PromptDocument, **kwargs) -> ModelOutputDocument:
    input = tokenizer(doc.prompt, return_tensors='pt')
    input_len = input['input_ids'].shape[1]
    for _ in range(doc.max_tokens):
        output = self.model.generate(**input, max_new_tokens=1)
        if output[0][-1] == tokenizer.eos_token_id:
            break
        yield ModelOutputDocument(
            token_id=output[0][-1],
            generated_text=tokenizer.decode(
                output[0][input_len:], skip_special_tokens=True
            ),
        )
        input = {
            'input_ids': output,
            'attention_mask': torch.ones(1, len(output[0])),
        }
```

4. Serve and use:
```python
# Server
with Deployment(uses=TokenStreamingExecutor, port=12345, protocol='grpc') as dep:
    dep.block()


# Client
async def main():
    client = Client(port=12345, protocol='grpc', asyncio=True)
    async for doc in client.stream_doc(
        on='/stream',
        inputs=PromptDocument(prompt='what is the capital of France ?', max_tokens=10),
        return_type=ModelOutputDocument,
    ):
        print(doc.generated_text)
```

## Support

Jina-serve is backed by [Jina AI](https://jina.ai) and licensed under [Apache-2.0](./LICENSE).
