Q&A Generator

The Q&A Generator automatically produces Question & Answer pairs from document sources. It supports any LLM provider — Workbench (default), OpenAI, Anthropic, or Gemini — via the backend parameter.

• Generate Q&A pairs from single documents or entire folders
• Support for DOCX and TXT formats
• Work with PyArrow tables containing document data
• Customizable number of Q&A pairs per document

Provider setup

Set your provider's API key as an environment variable, or pass it directly via subscription_key=:

Workbench (default)AnthropicOpenAIGemini

import os
os.environ['WORKBENCH_SUBSCRIPTION_KEY'] = '<your-key>'

import os
os.environ['ANTHROPIC_API_KEY'] = '<your-key>'

import os
os.environ['OPENAI_API_KEY'] = '<your-key>'

import os
os.environ['GOOGLE_API_KEY'] = '<your-key>'

Example Code

"""
Document Q&A Generation Example
"""
import os

import pyarrow as pa
from ydata.synthesizers.text.model.qa import DocumentQAGeneration

if __name__ == "__main__":
    #Authenticate to ydata-sdk
    os.environ['YDATA_LICENSE_KEY'] = '{add-your-key}'  # Replace with your license key
    # Step 1: Initialize the Q&A generator
    # You can use either OpenAI or Anthropic as the provider
    print("Initializing Q&A Generator...")
    qa_generator = DocumentQAGeneration()

    # Step 2: Generate Q&A pairs from a single document
    print("\n=== Processing Single Document ===")
    single_doc_result = qa_generator.generate(
        input_source="path/to/your/documents/folder/doc.docx",  # Replace with your document path
        docs_extension="docx",  # Supported formats: "docx" or "txt"
        num_qa_pairs=10,  # Number of Q&A pairs to generate
    )
    print("Single document Q&A pairs:")
    print(single_doc_result)

    # Step 3: Generate Q&A pairs from multiple documents in a folder
    print("\n=== Processing Multiple Documents ===")
    folder_result = qa_generator.generate(
        input_source="path/to/your/documents/folder/",  # Replace with your folder path
        docs_extension="docx",  # Process all documents with this extension
        num_qa_pairs=20,  # Number of Q&A pairs per document
    )
    print("Multiple documents Q&A pairs:")
    print(folder_result)

    # Step 4: Generate Q&A pairs from a PyArrow table
    print("\n=== Processing PyArrow Table ===")
    # Create a sample table with document content
    documents_table = pa.table({
        "text": [
            "This is a sample document about machine learning. It discusses various algorithms and their applications.",
            "Another document about data science and its importance in modern business."
        ],
        "metadata": [
            {"source": "doc1", "author": "John Doe"},
            {"source": "doc2", "author": "Jane Smith"}
        ]
    })

    table_result = qa_generator.generate(
        input_source=documents_table,
        num_qa_pairs=3,  # Number of Q&A pairs per document
    )
    print("PyArrow table Q&A pairs:")
    print(table_result)