mirror of https://github.com/langgenius/dify-docs.git synced 2026-03-26 13:18:34 +07:00

Files

Alter-xyz 4339f79a55 feat: add translation A/B testing framework (#564 )

Add tools for comparing translation quality between different models
(e.g., Sonnet vs Opus) or prompt variations. Useful for evaluating
translation improvements before deploying changes.

- run_test.py: Test runner with Dify API streaming
- compare.py: Generate similarity reports between variants
- Example spec and documentation included

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>

2025-11-28 16:02:13 +08:00

1.7 KiB

Raw Blame History

Translation Testing Framework

A/B testing for Dify translation workflows. Primary user: Claude Code.

Important

DO NOT commit test results - results/ is gitignored
DO NOT commit real API keys - always redact with app-*** before committing
DO NOT commit mock_docs/ - temporary files copied for testing

Quick Start

# Setup (first time)
./setup.sh
source venv/bin/activate

# Run test
python run_test.py <spec.md>

# Compare results
python compare.py results/<folder>/

Test Spec Format

# Test Title

## keys
app-xxx
Description A

app-yyy
Description B

## test_content
(Inline content - Claude Code generates this for each test)

# OR reference existing file:
## test_file
en/guides/workflow/some-doc.md

Workflow

User describes test scenario
Claude Code creates spec with ## test_content tailored to the issue
Run: source venv/bin/activate && python run_test.py spec.md
Analyze: python compare.py results/<folder>/
Redact API keys with app-*** before committing

Example: Punctuation Test

# Punctuation Test

## keys
app-***
Sonnet

app-***
Opus

## test_content
---
title: Test Doc
---

# Test

Sentence with commas, colons: semicolons; and more.

- Item one, comma
- Item two; semicolon

See example-model-comparison.md for a complete example.

Files

File	Purpose
run_test.py	Test runner
compare.py	Generate comparison reports
example-model-comparison.md	Example test spec
results/	Output (gitignored)
mock_docs/	Temp test files (gitignored)

Language Policy

All code and documentation in English (international project).

1.7 KiB Raw Blame History