Metadata-Version: 2.4
Name: privatetestsetgenerationforLLMeval
Version: 2.3.1
Summary: A package for generating evaluation set for LLM-based chatbots in a diverse and private manner
Home-page: https://github.com/AmadeusITGroup/privatetestsetgenerationforLLMeval
Author: Ilias
Author-email: ilias.driouich@amadeus.com
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=1.1.0
Requires-Dist: numpy>=1.19.0
Dynamic: author
Dynamic: author-email
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# LLM Test Set Generation Framework

This framework automates the creation of synthetic question-answer datasets for LLM evaluation from PDF sources.  
It extracts real-world domain text, ensures topic diversity, removes sensitive data (PII), and generates synthetic Q&A pairs.

---

##  Features

- **PDF Ingestion**: Extracts clean sentences from any PDF.
- **Embeddings**: Uses Azure OpenAI embeddings to represent sentences numerically.
- **Topic Diversification**: Clusters sentences into diverse topics using KMeans.
- **Privacy Filtering**: Detects and removes sensitive information (PII) with Presidio.
- **Synthetic QA Generation**: Generates rich, varied Q&A pairs via GPT-4 from sanitized text.
- **Modular Pipeline**: Built as a flexible multi-agent graph using LangGraph.

---

##  How It Works

Each step is an autonomous "agent" in a pipeline:

- **Diversity Agent**: Clusters data into meaningful topics.
- **Privacy Agent**: Analyzes clusters and flags sensitive content.
- **Synthetic Data Generator**: Creates QA samples while explaining coverage strategies.

## Usage 


To use the tool, follow these steps:

 ```bash
   pip install privatetestsetgenerationforLLMeval

   privatetestsetgeneration



```

  
   
## Contributing

Contributions to improve the tool are welcome! Feel free to open issues for bugs or feature requests, or submit pull requests for enhancements.



## Acknowledgements

This project utilizes various libraries, including LangChain for document processing and Presidio for PII detection and anonymization.



Clone the repository:
Install dependencies:
pip install -r requirements.txt
Create a .env file with the following variables:
AZURE_OPENAI_KEY=<your_azure_openai_api_key>
AZURE_OPENAI_API_VERSION=2023-12-01-preview
AZURE_ENDPOINT=https://<your_azure_resource>.openai.azure.com/
EMBEDDING_DEPLOYMENT_NAME=text-embedding-ada-002
GPT_4_MODEL_DEPLOYMENT_NAME=gpt-4



Run the main pipeline:

# to complete

To provide a path to a PDF.
To select the number of synthetic Q&A pairs to generate.
The pipeline will:

Process your document.
Create a topic-augmented, privacy-safe synthetic dataset.
Save a .csv file (e.g., qa_pairs10.csv) containing the generated Q&A pairs.


📄 Example Output


Question	Answer
What are the primary goals of digital transformation?	To leverage technology to optimize business processes, enhance customer experiences, and create new revenue streams.
How can data privacy be ensured when migrating to the cloud?	Through encryption, compliance audits, access control, and anonymization of sensitive data.
