LLM工程评估

模型评估测模型能力，应用评估测系统质量。你的 RAG 管道检索准确吗？函数调用可靠吗？回答一致性如何？没有评估，你无法知道系统是否真正可用。

类型： 构建 语言： Python 前置条件： Phase 11 Lesson 06（RAG） 预计时间： ~60 分钟

学习目标

• 理解 LLM 应用评估的分类体系
• 实现 LLM-as-Judge 评估器
• 构建自动化评估流水线
• 掌握回归测试和 CI/CD 集成

评估分类

LLM-as-Judge

JUDGE_PROMPT = """你是一个评估助手。请根据以下标准评估回复的质量。

评估标准：
1. 准确性（1-5）：回复中的事实是否正确
2. 完整性（1-5）：是否完整回答了问题
3. 相关性（1-5）：回复是否与问题相关
4. 清晰度（1-5）：回复是否清晰易懂

问题：{question}
回复：{response}
参考答案：{reference}

请按以下 JSON 格式输出评估结果：
{{
  "accuracy": <1-5>,
  "completeness": <1-5>,
  "relevance": <1-5>,
  "clarity": <1-5>,
  "overall": <1-5>,
  "reasoning": "<评估理由>"
}}"""


async def llm_judge(question, response, reference, judge_model):
    """LLM 评估器"""
    prompt = JUDGE_PROMPT.format(
        question=question,
        response=response,
        reference=reference,
    )

    result = await judge_model.generate(prompt, temperature=0)
    import json
    return json.loads(result)

评估流水线

class EvalPipeline:
    """评估流水线"""

    def __init__(self, rag_system, judge_model):
        self.rag = rag_system
        self.judge = judge_model

    async def run(self, test_cases):
        """运行评估"""
        results = []

        for case in test_cases:
            # 生成回答
            response, retrieved = self.rag.generate(case['question'])

            # 评估检索质量
            retrieval_score = self._eval_retrieval(
                case['question'], retrieved, case.get('relevant_docs', []))

            # 评估生成质量
            generation_score = await llm_judge(
                case['question'], response,
                case.get('reference_answer', ''), self.judge)

            results.append({
                'question': case['question'],
                'response': response,
                'retrieval_score': retrieval_score,
                'generation_score': generation_score,
            })

        # 汇总
        summary = self._summarize(results)
        return results, summary

    def _eval_retrieval(self, query, retrieved, relevant):
        """评估检索质量"""
        if not relevant:
            return None

        retrieved_set = set(doc.metadata.get('source') for doc, _ in retrieved)
        relevant_set = set(doc['source'] for doc in relevant)

        precision = len(retrieved_set & relevant_set) / max(len(retrieved_set), 1)
        recall = len(retrieved_set & relevant_set) / max(len(relevant_set), 1)
        f1 = 2 * precision * recall / max(precision + recall, 1e-10)

        return {"precision": precision, "recall": recall, "f1": f1}

    def _summarize(self, results):
        """汇总评估结果"""
        n = len(results)
        avg_retrieval_f1 = sum(
            r['retrieval_score']['f1'] for r in results if r['retrieval_score']
        ) / max(sum(1 for r in results if r['retrieval_score']), 1)

        avg_overall = sum(
            r['generation_score']['overall'] for r in results
        ) / n

        return {
            "num_cases": n,
            "avg_retrieval_f1": avg_retrieval_f1,
            "avg_generation_score": avg_overall,
        }

回归测试

def regression_test(system, baseline_results, threshold=0.1):
    """回归测试：确保系统改进不会降低已有功能"""
    failures = []

    for case_id, baseline in baseline_results.items():
        current = system.evaluate(case_id)

        for metric, value in baseline.items():
            if current[metric] < value - threshold:
                failures.append({
                    'case_id': case_id,
                    'metric': metric,
                    'baseline': value,
                    'current': current[metric],
                    'delta': current[metric] - value,
                })

    return failures

工具推荐

关键术语

延伸阅读

• Zheng et al., 2023 — “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” — LLM-as-Judge 方法论
• Es et al., 2023 — “Ragas: Automated Evaluation of Retrieval Augmented Generation” — Ragas 论文
• promptfoo — 提示评估工具