version: "1.0.0" name: benchmark-qed-setup description: > Initialize and configure benchmark-qed workspaces for RAG benchmarking. Use when: setting up a new benchmarking project, initializing config files for question generation or evaluation, downloading sample datasets, or modifying benchmark-qed settings.yaml configuration. Also use when the user mentions "benchmark-qed config", workspace setup, or needs to prepare a benchmarking environment — even if they don't say "setup" explicitly.

Benchmark-QED Workspace Setup

Initialize workspaces, generate configuration files, download datasets, and manage settings for the benchmark-qed RAG benchmarking tool.

Prerequisites

benchmark-qed requires Python 3.11+ and uv. Run commands with uvx to avoid installing globally:

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed <command>

Pin a specific version for reproducibility:

uvx --from "git+https://github.com/microsoft/benchmark-qed@v1.2.3" benchmark-qed <command>

If uvx is unavailable, install uv first:

pip install uv && uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed <command>

Procedure

Step 1 — Initialize a Workspace

Option A (Recommended): Interactive wizard

The interactive wizard guides you through configuration with sensible defaults:

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed init <root_directory>

This walks through:

• Config type selection (autoq, autoe_pairwise, autoe_reference, autoe_assertion)
• LLM provider selection with Azure-specific prompts (endpoint, API version)
• Section-by-section customization (press Enter to accept defaults)
• Automatic YAML validation before writing

Option B: Non-interactive (template-based)

Generate a static template and edit manually:

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed config init <config_type> <root_directory>

Config types (pick one):

Example:

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed config init autoq ./my_workspace

Storage options for config init:

When --storage-type blob is combined with --account-url or --connection-string, the generated config and prompt files are also uploaded directly to the blob container.

Example (blob):

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed config init autoq ./my_workspace \
  --storage-type blob \
  --container-name my-datasets \
  --account-url https://myaccount.blob.core.windows.net \
  --base-dir experiments/run1

This creates:

root/
├── .env              # API key placeholder
├── input/            # Place your data here
├── settings.yaml     # Main configuration file
└── prompts/          # LLM prompt templates

Step 2 — Download Sample Data (Optional)

Download sample datasets for testing. This command has an interactive confirmation prompt with no --yes flag — use one of these approaches to avoid hanging:

Bash/Linux/macOS:

echo y | uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed data download <dataset> <output_dir>

PowerShell:

"y" | uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed data download <dataset> <output_dir>

Available datasets: AP_news, podcast, example_answers

Storage options for data download:

Example (download to blob):

echo y | uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed data download AP_news datasets \
  --storage-type blob \
  --container-name my-datasets \
  --account-url https://myaccount.blob.core.windows.net

Step 3 — Gather Configuration Choices from the User

Before writing any values into settings.yaml, prompt the user with `ask_user` to collect the LLM / auth / endpoint settings. Do not guess — these decisions are environment-specific and getting them wrong wastes downstream LLM calls. Use enum/boolean fields whenever possible so the user picks from a known set rather than typing free-form text.

Ask in a single `ask_user` form (split into two if the workflow is autoq, since autoq also needs an embedding model). Tailor the follow-up fields based on the provider/auth choice — if the first answer reveals an Azure provider, ask the Azure-only fields in a second form.

LLM (chat) fields to collect

Embedding fields to collect (autoq only)

Ask the same shape of questions for the embedding model:

Input data fields (autoq only)

Eval-config-specific fields (autoe_*)

Only ask the questions relevant to the chosen config_type:

• autoe_pairwise: base.name + base.answer_base_path, plus a list of others (each with name and answer_base_path), and question_sets.
• autoe_reference: reference.name + reference.answer_base_path, list of generated, and question_sets.
• autoe_assertion: in single-RAG mode, generated.name + generated.answer_base_path and assertions.assertions_path. In multi-RAG mode (rag_methods provided), ask for input_dir, output_dir, rag_methods list, and question_sets.

Storage fields (all config types, optional)

Ask the user if they want to use Azure Blob Storage for input/output. If yes, collect:

If storage is enabled, write the appropriate input.storage, input_storage, and/or output_storage blocks into settings.yaml.

If the user declines a field, fall back to the documented default and call out the assumption in your response.

Step 4 — Apply the Answers

Use the answers from Step 3 to edit settings.yaml and .env directly:

# LLM configuration (template — substitute values from ask_user answers)
chat_model:
  model: <model from ask_user>
  llm_provider: <llm_provider from ask_user>
  auth_type: <auth_type from ask_user>
  api_key: ${<api_key_env_var>}        # only when auth_type=api_key
  concurrent_requests: <concurrent_requests>
  init_args:                            # only for azure.* providers
    azure_endpoint: <azure_endpoint>
    api_version: "<api_version>"        # azure.openai.* only
  retry:                                # scaffolded by `config init`; keep unless customized
    type: exponential_backoff
    max_retries: 6
    base_delay: 2.0
    max_delay: 30.0
    jitter: true
 
# Input data (autoq only)
input:
  dataset_path: <dataset_path>
  input_type: <input_type>
  text_column: <text_column>

Rules when writing the YAML:

• Omit api_key entirely when auth_type=azure_managed_identity — do not leave ${OPENAI_API_KEY} in place.
• Omit init_args for non-Azure providers.
• Quote api_version (it would otherwise be parsed as a date).
• For azure_managed_identity, do not add anything to .env for that key.
• For api_key auth, append <api_key_env_var>=<placeholder> to .env if the variable is missing, and tell the user to replace the placeholder with their real key before running any command.

For the full set of optional fields, read references/config-reference.md.

Step 5 — Review Settings with the User

After writing settings.yaml, show the user the generated configuration and ask if they want to customize anything. This is critical — the generated config uses sensible defaults, but users often need to tune dataset-specific or environment-specific values.

1. Read the generated settings.yaml and display its contents to the user (use show_file).
2. Use ask_user with a boolean field: "Would you like to customize any settings before proceeding?"
3. If the user wants changes, use ask_user with a free-text string field: "Describe what you'd like to change" — let them say it in their own words (e.g., "increase num_questions to 50 for all types", "change the model to gpt-4o", "set trials to 6 and add a custom criterion"). Then apply the requested changes to settings.yaml.
4. After applying changes, show the updated file and ask again: "Any other changes?" (boolean). Repeat until the user says no.

Do not limit the user to predefined sections — they should be able to modify any field in settings.yaml by describing what they want.

Sections the user is most likely to customize (call these out):

• autoq: num_questions per type, num_clusters, chunk_size, assertion settings, concurrent_requests
• autoe_pairwise: trials, criteria, question_sets
• autoe_reference: score_min/score_max, trials
• autoe_assertion: pass_threshold, trials

For the full set of optional fields and best practices, read references/config-reference.md.

Step 6 — Validate Configuration

The benchmark-qed CLI validates settings.yaml via pydantic at startup, so any missing or malformed fields are reported when you run a command. After applying the answers, run the actual target command (e.g. benchmark-qed autoq …) — config errors surface immediately, before any LLM calls.

Best Practices

See references/config-reference.md for detailed best practices covering LLM configuration, prompts, question generation, assertion generation, evaluation, and retrieval.

Key highlights:

• Use ${OPENAI_API_KEY} env var substitution — never hardcode secrets
• Use benchmark-qed init (interactive wizard) to avoid manual YAML errors
• Pin a specific version of benchmark-qed for reproducibility in CI/CD

Gotchas

• The data download command blocks on typer.confirm(). Always use echo y | uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed data download ... to prevent hanging.
• Environment variables in YAML use ${VAR_NAME} syntax (resolved at load time via python-dotenv).
• The .env file must be in the workspace root directory, not the project root.
• Config types autoe_pairwise, autoe_reference, and autoe_assertion generate different settings.yaml templates — use the correct type for your evaluation method.
• Prompts are copied as .txt files using Python string.Template syntax ($variable or ${variable}).
• `prompt_config` key: The runtime expects prompt_config (singular) for all autoe config types. Both benchmark-qed init and config init now generate the correct key. If you hand-edit YAML, ensure you use prompt_config, not prompts_config.
• `config init --storage-type blob`: When combined with --account-url or --connection-string, the command uploads the generated settings.yaml and prompt files directly to blob storage. Without those auth options, it only scaffolds the storage YAML sections locally.
• Blob URI format: CLI commands accept blob://<container>/<key> for config paths. The CLI downloads the config and all sibling files (prompt templates) to a temp directory so relative paths resolve correctly. Credentials can be passed via --account-url/--connection-string or the environment variables AZURE_STORAGE_ACCOUNT_URL/AZURE_STORAGE_CONNECTION_STRING.
• Storage config in YAML: AutoQ uses input.storage (nested under input) and output_storage (top-level). AutoE uses input_storage and output_storage (both top-level). When storage is omitted, local filesystem is used.
• graphrag-llm backend: As of the graphrag-llm migration, built-in providers (OpenAI, Azure OpenAI, Azure AI Inference) are served by graphrag-llm's LiteLLM-backed factory. Custom providers must implement the graphrag_llm.completion.LLMCompletion or graphrag_llm.embedding.LLMEmbedding interfaces (the older benchmark_qed.llm.type.base.ChatModel / EmbeddingModel Protocols have been removed). See references/config-reference.md.