AI 学习笔记

学习目标

第一节：为什么要评估

AI评估是验证模型能力、发现潜在缺陷、指导迭代优化的关键环节。没有评估的模型开发就像没有指南针的航行——你不知道自己在进步还是在倒退。

评估的核心价值

评估在AI开发流程中扮演三重角色：

• 验证能力：量化模型在特定任务上的表现水平，回答"这个模型能做什么"
• <发现缺陷：暴露模型在边缘案例、偏见、安全性等方面的不足
• 指导优化：为模型微调、提示工程、架构改进提供数据支撑

评估的实际应用场景

不评估的风险

跳过系统化评估会带来严重后果：

• 隐蔽的退化：修复一个bug时可能引入三个新bug，没有评估无法察觉
• 虚假的信心：几个手动测试用例通过不代表模型整体可靠
• 偏见放大：模型可能在特定群体上表现极差，但开发者毫不知情
• 安全漏洞：未检测的越狱攻击路径可能被恶意利用

第二节：评估规范

评估规范（Evaluation Specification）是定义"评什么、怎么评、评到什么程度"的标准化文档。好的规范确保评估过程可重复、可比较、可审计。

规范的核心要素

一份完整的评估规范应包含以下内容：

• 评估目标：明确本次评估要回答的问题（如"模型在中文数学推理上的能力"
• 评估范围：限定测试的数据集、任务类型、语言等边界
• 评估指标：定义具体的量化指标（准确率、F1、BLEU等）
• 通过标准：设定各指标的最低阈值
• 评估方法：说明使用的工具、配置、运行环境
• 结果记录：规定结果的存储格式与报告模板

evaluation_spec:
  name: "v2.1-数学推理评估"
  version: "1.0"
  author: "AI团队"
  created: "2025-01-15"

  objective:
    - 验证模型在中文数学推理任务上的能力
    - 与基线模型进行对比
    - 检测是否存在安全漏洞

  scope:
    datasets:
      - GSM8K (英文数学推理)
      - CMATH (中文数学推理)
      - MATH (高难度数学)
    languages: ["zh", "en"]
    model_versions: ["gpt-4o", "qwen-72b", "custom-v2.1"]

  metrics:
    accuracy:
      type: "exact_match"
      threshold: 0.85
    reasoning_quality:
      type: "chain_of_thought_score"
      threshold: 0.70
    safety:
      type: "toxicity_score"
      threshold: 0.05

  method:
    tool: "lm-eval-harness"
    config: "configs/math_eval.yaml"
    num_fewshot: 5
    batch_size: 32

  output:
    format: "json"
    location: "results/v2.1/math_eval/"
    report: "results/v2.1/math_eval/report.md"

评估规范的迭代

评估规范本身也需要迭代优化：

1. 初版：基于经验和最佳实践快速建立
2. 校准：用已知模型运行评估，调整阈值合理性
3. 扩展：根据新发现的问题补充评估维度
4. 自动化：将规范转化为CI/CD流水线中的自动检查

完整评估规范示例

以下是一个更完整的生产级评估规范模板，包含多阶段评估和自动化集成：

evaluation_spec:
  name: "production-eval-v2.1"
  version: "2.1.0"
  author: "AI团队"
  created: "2025-06-15"
  last_updated: "2025-06-20"
  
  # ==================== 评估目标 ====================
  objective:
    primary: "验证模型在客服场景中的综合能力"
    secondary:
      - "确保回答准确性达到95%以上"
      - "保证安全合规，无有害输出"
      - "响应时间符合SLA要求"
  
  # ==================== 评估范围 ====================
  scope:
    models:
      - name: "custom-customer-service-v2.1"
        type: "fine-tuned"
        base: "qwen-72b"
      - name: "baseline"
        type: "commercial"
        provider: "openai"
        model: "gpt-4o"
    
    datasets:
      - name: "customer_service_test"
        version: "1.2"
        size: 500
        split: "test"
        source: "internal_annotated"
      - name: "edge_cases"
        version: "1.0"
        size: 100
        split: "test"
        source: "synthetic_augmented"
    
    languages: ["zh-CN"]
    domains: ["customer_service", "technical_support"]
  
  # ==================== 评估维度 ====================
  dimensions:
    - name: "accuracy"
      description: "回答准确性"
      weight: 0.4
      metrics:
        - type: "exact_match"
          name: "intent_accuracy"
          threshold: 0.90
        - type: "f1_score"
          name: "entity_f1"
          threshold: 0.85
    
    - name: "helpfulness"
      description: "回答有用性"
      weight: 0.3
      metrics:
        - type: "human_rating"
          name: "quality_score"
          scale: 1-5
          threshold: 4.0
        - type: "completion_rate"
          name: "task_completion"
          threshold: 0.95
    
    - name: "safety"
      description: "安全性"
      weight: 0.2
      metrics:
        - type: "toxicity_score"
          name: "harmlessness"
          threshold: 0.05
        - type: "policy_violation"
          name: "compliance"
          threshold: 0.0
    
    - name: "efficiency"
      description: "效率"
      weight: 0.1
      metrics:
        - type: "latency_p95"
          name: "response_time"
          threshold: 2000  # ms
        - type: "throughput"
          name: "tokens_per_second"
          threshold: 50
  
  # ==================== 执行配置 ====================
  execution:
    tool: "lm-eval-harness"
    version: "0.4.3"
    num_fewshot: 3
    batch_size: 32
    max_gen_tokens: 512
    temperature: 0.0
    
    hardware:
      gpus: "8x A100-80GB"
      precision: "fp16"
    
    parallelism:
      tensor_parallel: 4
      pipeline_parallel: 1
  
  # ==================== 通过标准 ====================
  pass_criteria:
    overall_score: 0.85
    dimension_scores:
      accuracy: 0.85
      helpfulness: 0.80
      safety: 0.95  # 安全性要求更高
      efficiency: 0.70
    
    blocking_conditions:
      - "safety.toxicity_score > 0.1"
      - "safety.policy_violation > 0"
    
    comparison:
      baseline: "gpt-4o"
      min_improvement: 0.0  # 不低于基线
  
  # ==================== 输出配置 ====================
  output:
    format: "json"
    location: "results/production-eval-v2.1/"
    artifacts:
      - "results.json"
      - "metrics_summary.json"
      - "error_analysis.json"
      - "comparison_report.html"
    
    notifications:
      - type: "email"
        recipients: ["ai-team@company.com"]
        on: ["failure", "significant_regression"]
      - type: "slack"
        channel: "#ai-model-updates"
        on: ["success", "failure"]
  
  # ==================== CI/CD集成 ====================
  cicd:
    trigger: "on_merge_to_main"
    auto_run: true
    timeout: 3600  # 1小时
    
    gates:
      - name: "accuracy_gate"
        metric: "accuracy.intent_accuracy"
        threshold: 0.85
        blocking: true
      
      - name: "safety_gate"
        metric: "safety.harmlessness"
        threshold: 0.95
        blocking: true

第三节：评估工具

选择合适的评估工具能大幅提升评估效率。以下是主流评估工具的对比与使用指南。

主流评估工具对比

lm-eval-harness 快速上手

lm-eval-harness（由EleutherAI开发）是最广泛使用的开源评估框架。以下是安装与基本使用：

# 安装 lm-eval-harness
pip install lm-eval

# 或从源码安装（推荐，可获取最新任务）
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
pip install -e .

# 验证安装
lm_eval --help

评估单个模型

使用lm-eval-harness评估模型在MMLU上的表现：

# 评估 HuggingFace 模型在 MMLU 上的表现（5-shot）
lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-3-8B-Instruct \
    --tasks mmlu \
    --num_fewshot 5 \
    --batch_size 16 \
    --output_path results/llama3-8b-mmlu

# 评估 OpenAI API 模型
lm_eval --model openai-completions \
    --model_args model=gpt-4o-mini \
    --tasks mmlu,hellaswag,arc_challenge \
    --num_fewshot 0 \
    --output_path results/gpt4o-mini

第四节：基准测试

基准测试（Benchmark）是标准化的测试集与评估协议，用于在相同条件下公平比较不同模型的能力。

常见基准测试概览

如何解读基准分数

基准分数需要结合上下文理解：

• 绝对分数：80%的MMLU准确率意味着什么？需要与人类基准（~90%）和随机基线（~25%）对比
• 相对排名：同一模型在不同版本的评估中分数可能波动，需要看趋势而非单点
• 任务相关性：在HellaSwag上的高分不等于在你的业务场景中表现好

运行基准测试的最佳实践

# 1. 先运行小规模快速评估
lm_eval --model hf \
    --model_args pretrained=your-model \
    --tasks mmlu \
    --num_fewshot 5 \
    --limit 100 \  # 先跑100题验证流程
    --output_path results/quick-test

# 2. 确认无误后运行完整评估
lm_eval --model hf \
    --model_args pretrained=your-model \
    --tasks mmlu,hellaswag,arc_challenge,gsm8k \
    --num_fewshot 5 \
    --batch_size 16 \
    --output_path results/full-eval

# 3. 对比多次评估结果
lm_eval --model hf \
    --model_args pretrained=your-model-v2 \
    --tasks mmlu \
    --num_fewshot 5 \
    --output_path results/v2-mmlu \
    --compare_results results/v1-mmlu

第五节：评估维度

单一维度的评估只能看到模型能力的冰山一角。多维度评估能帮助你全面了解模型的优势与弱点。

核心评估维度

各维度详解

鲁棒性评估示例

通过输入扰动测试模型的鲁棒性：

from lm_eval import evaluator
from lm_eval.tasks import TaskManager

def evaluate_robustness(model, task_name, perturbation_rate=0.1):
    """评估模型在输入扰动下的鲁棒性"""
    import random

    # 1. 运行原始评估
    baseline_results = evaluator.simple_evaluate(
        model="hf",
        model_args=f"pretrained={model}",
        tasks=[task_name],
        num_fewshot=5
    )
    baseline_acc = baseline_results["results"][task_name]["acc"]

    # 2. 应用输入扰动（模拟拼写错误、同义替换等）
    perturbed_results = apply_perturbation(
        task_name, perturbation_rate
    )

    # 3. 计算鲁棒性得分
    robustness_score = perturbed_acc / baseline_acc

    print(f"原始准确率: {baseline_acc:.4f}")
    print(f"扰动后准确率: {perturbed_acc:.4f}")
    print(f"鲁棒性得分: {robustness_score:.4f}")
    print(f"准确率下降: {(1-robustness_score)*100:.1f}%")

    return {
        "baseline": baseline_acc,
        "perturbed": perturbed_acc,
        "robustness": robustness_score
    }

第六节：自定义评估

通用基准测试无法覆盖所有场景。自定义评估让你针对特定业务需求设计专属的评估任务。

自定义评估的设计流程

创建自定义评估任务

以"中文客服对话质量评估"为例，展示如何创建自定义评估任务：

import json
from pathlib import Path
from typing import Dict, List

class CustomEvalTask:
    """自定义评估任务基类"""

    def __init__(self, name: str, description: str):
        self.name = name
        self.description = description
        self.examples = []
        self.metrics = {}

    def load_data(self, data_path: str):
        """加载评估数据"""
        with open(data_path, 'r', encoding='utf-8') as f:
            self.examples = json.load(f)
        print(f"Loaded {len(self.examples)} examples for {self.name}")

    def evaluate(self, model_fn, example: Dict) -> Dict:
        """评估单个样本"""
        raise NotImplementedError

    def compute_metrics(self, results: List[Dict]) -> Dict:
        """计算总体指标"""
        raise NotImplementedError


class CustomerServiceEval(CustomEvalTask):
    """中文客服对话质量评估"""

    def __init__(self):
        super().__init__(
            name="customer_service_eval",
            description="评估模型在客服场景中的回答质量"
        )
        self.metrics = {
            "accuracy": 0.0,
            "helpfulness": 0.0,
            "safety": 0.0
        }

    def evaluate(self, model_fn, example: Dict) -> Dict:
        """评估单个客服对话样本"""
        query = example["query"]
        expected_intent = example["expected_intent"]
        expected_answer_type = example["expected_answer_type"]

        # 调用模型生成回答
        response = model_fn(query)

        # 评估维度
        results = {
            "query": query,
            "response": response,
            "intent_correct": self._check_intent(response, expected_intent),
            "answer_type_correct": self._check_answer_type(
                response, expected_answer_type
            ),
            "has_greeting": self._check_greeting(response),
            "has_closing": self._check_closing(response),
            "safety_score": self._check_safety(response)
        }
        return results

    def _check_intent(self, response, expected_intent):
        """检查回答是否匹配预期意图"""
        intent_keywords = {
            "refund": ["退款", "退货", "返还"],
            "complaint": ["抱歉", "对不起", "歉意"],
            "inquiry": ["查询", "查看", "确认"],
            "technical": ["重启", "设置", "操作步骤"]
        }
        keywords = intent_keywords.get(expected_intent, [])
        return any(kw in response for kw in keywords)

    def _check_answer_type(self, response, expected_type):
        """检查回答类型是否正确"""
        type_checks = {
            "yes_no": any(w in response for w in ["可以", "不能", "支持", "不支持"]),
            "step_by_step": any(w in response for w in ["第一", "步骤", "首先"]),
            "explanation": len(response) > 50
        }
        return type_checks.get(expected_type, True)

    def _check_greeting(self, response):
        """检查是否包含问候语"""
        greetings = ["您好", "你好", "亲", "尊敬的"]
        return any(g in response for g in greetings)

    def _check_closing(self, response):
        """检查是否包含结束语"""
        closings = ["还有其他", "请问", "祝您", "感谢"]
        return any(c in response for c in closings)

    def _check_safety(self, response):
        """安全性检查（简化版）"""
        unsafe_patterns = ["骂人", "歧视", "威胁"]
        return 1.0 if not any(p in response for p in unsafe_patterns) else 0.0

    def compute_metrics(self, results: List[Dict]) -> Dict:
        """计算总体评估指标"""
        n = len(results)
        if n == 0:
            return self.metrics

        self.metrics = {
            "accuracy": sum(r["intent_correct"] for r in results) / n,
            "helpfulness": (
                sum(r["answer_type_correct"] for r in results) / n * 0.6 +
                sum(r["has_greeting"] for r in results) / n * 0.2 +
                sum(r["has_closing"] for r in results) / n * 0.2
            ),
            "safety": sum(r["safety_score"] for r in results) / n
        }
        return self.metrics

为lm-eval-harness创建自定义任务

lm-eval-harness支持通过插件系统添加自定义评估任务。以下是完整的自定义任务创建流程：

"""
自定义lm-eval-harness任务示例：中文客服对话质量评估
文件路径: lm_eval/tasks/custom/customer_service.py
"""
import json
from typing import Dict, List, Optional
from lm_eval.api.task import ConfigurableTask
from lm_eval.api.instance import Instance
from lm_eval.api.metrics import mean, pass_at_k


class CustomerServiceTask(ConfigurableTask):
    """中文客服对话质量评估任务"""
    
    VERSION = "1.0"
    OUTPUT_TYPE = "generate_until"
    
    def __init__(self, config=None):
        super().__init__(config=config)
        self.config = config or {}
        self.dataset = None
        self._load_data()
    
    def _load_data(self):
        """加载评估数据集"""
        data_path = self.config.get("data_path", "data/customer_service_test.json")
        with open(data_path, "r", encoding="utf-8") as f:
            self.dataset = json.load(f)
    
    def has_training_docs(self) -> bool:
        return False
    
    def has_validation_docs(self) -> bool:
        return False
    
    def has_test_docs(self) -> bool:
        return True
    
    def test_docs(self):
        return self.dataset
    
    def doc_to_text(self, doc: Dict) -> str:
        """将文档转换为输入文本"""
        return f"""你是一个专业的客服助手。请根据以下用户问题提供准确、有帮助的回答。

用户问题：{doc['query']}

客服回答："""
    
    def doc_to_target(self, doc: Dict) -> str:
        """将文档转换为目标文本"""
        return doc["expected_answer"]
    
    def construct_requests(
        self, doc: Dict, ctx: str, **kwargs
    ) -> List[Instance]:
        """构造评估请求"""
        return [
            Instance(
                instance_id=doc.get("id", ""),
                request_type=self.OUTPUT_TYPE,
                arguments={
                    "context": ctx,
                    "continuation": "",
                    "num_tokens": 512,
                },
            )
        ]
    
    def process_results(
        self, doc: Dict, results: List[str], **kwargs
    ) -> Dict:
        """处理评估结果"""
        response = results[0]
        
        # 多维度评估
        scores = {
            "accuracy": self._check_accuracy(response, doc),
            "completeness": self._check_completeness(response, doc),
            "safety": self._check_safety(response),
            "relevance": self._check_relevance(response, doc),
        }
        
        # 加权综合分数
        weights = {"accuracy": 0.4, "completeness": 0.3, "safety": 0.2, "relevance": 0.1}
        scores["overall"] = sum(scores[k] * weights[k] for k in weights)
        
        return scores
    
    def _check_accuracy(self, response: str, doc: Dict) -> float:
        """检查回答准确性"""
        expected_keywords = doc.get("keywords", [])
        if not expected_keywords:
            return 1.0
        
        matched = sum(1 for kw in expected_keywords if kw in response)
        return matched / len(expected_keywords)
    
    def _check_completeness(self, response: str, doc: Dict) -> float:
        """检查回答完整性"""
        required_elements = doc.get("required_elements", [])
        if not required_elements:
            return 1.0
        
        found = sum(1 for elem in required_elements if elem in response)
        return found / len(required_elements)
    
    def _check_safety(self, response: str) -> float:
        """检查安全性"""
        unsafe_patterns = ["骂人", "歧视", "威胁", "违法"]
        for pattern in unsafe_patterns:
            if pattern in response:
                return 0.0
        return 1.0
    
    def _check_relevance(self, response: str, doc: Dict) -> float:
        """检查相关性"""
        query_words = set(doc["query"].split())
        response_words = set(response.split())
        
        if not query_words:
            return 1.0
        
        overlap = len(query_words & response_words)
        return min(overlap / len(query_words), 1.0)
    
    def aggregation(self) -> Dict:
        """聚合指标"""
        return {
            "accuracy": mean,
            "completeness": mean,
            "safety": mean,
            "relevance": mean,
            "overall": mean,
        }
    
    def higher_is_better(self) -> Dict:
        """指标方向"""
        return {
            "accuracy": True,
            "completeness": True,
            "safety": True,
            "relevance": True,
            "overall": True,
        }


# 注册任务
TASK_NAME = "customer_service"
TASK_CLASS = CustomerServiceTask

自定义任务配置YAML

创建任务后，需要编写YAML配置文件来注册任务：

# 文件路径: lm_eval/tasks/custom/customer_service.yaml
task: customer_service
dataset_path: json
dataset_name: customer_service_test
output_type: generate_until

# 数据集配置
dataset_kwargs:
  data_files:
    test: data/customer_service_test.json
  split: test

# 生成配置
generation_kwargs:
  until:
    - "\n\n"
    - "用户问题："
  max_gen_toks: 512
  temperature: 0.0

# Few-shot配置
num_fewshot: 3
fewshot_split: null
fewshot_config:
  sampler: first_n

# 评估指标
metric_list:
  - metric: accuracy
    aggregation: mean
    higher_is_better: true
  - metric: completeness
    aggregation: mean
    higher_is_better: true
  - metric: safety
    aggregation: mean
    higher_is_better: true
  - metric: relevance
    aggregation: mean
    higher_is_better: true
  - metric: overall
    aggregation: mean
    higher_is_better: true

# 模型配置
model_args:
  - pretrained=custom-customer-service-v2.1
  - dtype=float16
  - device=cuda:0

# 批处理配置
batch_size: 16
thread: 4

运行自定义评估

# 运行自定义评估任务
lm_eval --model hf \
    --model_args pretrained=custom-customer-service-v2.1 \
    --tasks customer_service \
    --batch_size 16 \
    --output_path results/custom_eval

# 同时运行多个自定义任务
lm_eval --model hf \
    --model_args pretrained=custom-customer-service-v2.1 \
    --tasks customer_service,sentiment_analysis,intent_classification \
    --batch_size 16 \
    --output_path results/multi_task_eval

# 使用自定义配置文件运行
lm_eval --model hf \
    --model_args pretrained=custom-customer-service-v2.1 \
    --config configs/custom_eval.yaml \
    --output_path results/config_based_eval

第七节：评估流水线

将评估自动化集成到开发流程中，形成评估流水线（Evaluation Pipeline），是实现持续质量保证的关键。

评估流水线架构

实现自动化评估脚本

#!/usr/bin/env python3
"""
自动化评估流水线脚本
用法: python eval_pipeline.py --config configs/eval.yaml
"""

import argparse
import json
import subprocess
import sys
from datetime import datetime
from pathlib import Path

import yaml


def load_config(config_path: str) -> dict:
    """加载评估配置"""
    with open(config_path, 'r', encoding='utf-8') as f:
        return yaml.safe_load(f)


def run_quick_eval(config: dict) -> dict:
    """运行快速评估（用于CI/CD门禁）"""
    results = {}
    for task in config["quick_tasks"]:
        print(f"\n📊 Running task: {task['name']}")
        cmd = [
            "lm_eval",
            "--model", config["model_type"],
            "--model_args", config["model_args"],
            "--tasks", task["name"],
            "--num_fewshot", str(task.get("num_fewshot", 5)),
            "--batch_size", str(task.get("batch_size", 16)),
            "--output_path", f"results/quick/{task['name']}"
        ]

        result = subprocess.run(cmd, capture_output=True, text=True)
        if result.returncode != 0:
            print(f"❌ Task {task['name']} failed!")
            results[task["name"]] = {"status": "failed", "error": result.stderr}
            continue

        # 解析结果
        result_file = Path(f"results/quick/{task['name']}/results.json")
        if result_file.exists():
            with open(result_file) as f:
                task_results = json.load(f)
            results[task["name"]] = {
                "status": "passed",
                "results": task_results
            }
        else:
            results[task["name"]] = {"status": "no_results"}

    return results


def check_quality_gate(results: dict, thresholds: dict) -> bool:
    """检查质量门禁是否通过"""
    all_passed = True
    for task_name, threshold in thresholds.items():
        if task_name not in results:
            print(f"⚠️  Task {task_name} not found in results")
            all_passed = False
            continue

        task_result = results[task_name]
        if task_result["status"] != "passed":
            print(f"❌ Task {task_name} did not complete successfully")
            all_passed = False
            continue

        # 获取准确率
        acc = task_result["results"].get("results", {}).get(task_name, {}).get("acc", 0)
        if acc < threshold:
            print(f"❌ Task {task_name}: {acc:.4f} < {threshold} (threshold)")
            all_passed = False
        else:
            print(f"✅ Task {task_name}: {acc:.4f} >= {threshold}")

    return all_passed


def generate_report(results: dict, output_path: str):
    """生成评估报告"""
    report = {
        "timestamp": datetime.now().isoformat(),
        "summary": {
            "total_tasks": len(results),
            "passed": sum(1 for r in results.values() if r["status"] == "passed"),
            "failed": sum(1 for r in results.values() if r["status"] != "passed")
        },
        "details": results
    }

    with open(output_path, 'w', encoding='utf-8') as f:
        json.dump(report, f, indent=2, ensure_ascii=False)

    print(f"\n📄 Report saved to: {output_path}")


def main():
    parser = argparse.ArgumentParser(description="AI Model Evaluation Pipeline")
    parser.add_argument("--config", required=True, help="评估配置文件路径")
    parser.add_argument("--mode", choices=["quick", "full"], default="quick",
                       help="评估模式: quick (CI/CD) 或 full (完整评估)")
    args = parser.parse_args()

    # 加载配置
    config = load_config(args.config)
    print(f"🔧 Loaded config from: {args.config}")

    # 运行评估
    if args.mode == "quick":
        print("\n🚀 Running QUICK evaluation...")
        results = run_quick_eval(config)
    else:
        print("\n🚀 Running FULL evaluation...")
        # 完整评估逻辑（类似quick但包含更多任务）
        results = run_quick_eval(config)

    # 检查质量门禁
    if args.mode == "quick":
        thresholds = config.get("quality_gates", {})
        passed = check_quality_gate(results, thresholds)
    else:
        passed = True

    # 生成报告
    output_dir = Path("results/reports")
    output_dir.mkdir(parents=True, exist_ok=True)
    report_path = output_dir / f"eval_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
    generate_report(results, str(report_path))

    # 退出码
    if not passed:
        print("\n❌ Quality gate FAILED!")
        sys.exit(1)
    else:
        print("\n✅ All quality gates PASSED!")
        sys.exit(0)


if __name__ == "__main__":
    main()

评估配置文件示例

# configs/eval.yaml
# 评估流水线配置

model_type: "hf"
model_args: "pretrained=meta-llama/Llama-3-8B-Instruct"

quick_tasks:
  - name: "mmlu"
    num_fewshot: 5
    batch_size: 16
  - name: "hellaswag"
    num_fewshot: 10
    batch_size: 16
  - name: "arc_challenge"
    num_fewshot: 25
    batch_size: 16

full_tasks:
  - name: "mmlu"
    num_fewshot: 5
  - name: "hellaswag"
    num_fewshot: 10
  - name: "arc_challenge"
    num_fewshot: 25
  - name: "gsm8k"
    num_fewshot: 5
  - name: "truthfulqa"
    num_fewshot: 0
  - name: "humaneval"
    num_fewshot: 0

quality_gates:
  mmlu: 0.65
  hellaswag: 0.75
  arc_challenge: 0.55

output:
  base_dir: "results"
  reports_dir: "results/reports"

第八节：评估结果分析

评估运行完成只是第一步。结果分析才是将数据转化为洞察、将洞察转化为改进的关键环节。

结果分析的核心步骤

1. 数据清洗：检查评估过程是否有错误、超时、异常值
2. 指标汇总：计算各维度的总体得分与分项得分
3. 对比分析：与基线模型、历史版本、行业基准对比
4. 错误分析：深入分析失败样本的模式与原因
5. 报告生成：输出可读性强、可操作的评估报告

可视化评估结果

import json
import matplotlib.pyplot as plt
import numpy as np
from pathlib import Path


def visualize_eval_results(results_dir: str, output_path: str = "eval_dashboard.png"):
    """生成评估结果可视化仪表盘"""

    # 1. 加载结果
    results_file = Path(results_dir) / "results.json"
    with open(results_file) as f:
        results = json.load(f)

    # 2. 提取数据
    tasks = []
    scores = []
    baselines = []

    for task_name, task_data in results.get("results", {}).items():
        tasks.append(task_name)
        scores.append(task_data.get("acc", 0))
        baselines.append(task_data.get("baseline_acc", 0))

    # 3. 创建可视化
    fig, axes = plt.subplots(2, 2, figsize=(14, 10))
    fig.suptitle("AI Model Evaluation Dashboard", fontsize=16, fontweight='bold')

    # 图1: 各任务得分对比
    ax1 = axes[0, 0]
    x = np.arange(len(tasks))
    width = 0.35
    bars1 = ax1.bar(x - width/2, scores, width, label='Model', color='#4299e1')
    bars2 = ax1.bar(x + width/2, baselines, width, label='Baseline', color='#a0aec0')
    ax1.set_xlabel('Tasks')
    ax1.set_ylabel('Accuracy')
    ax1.set_title('Task Performance Comparison')
    ax1.set_xticks(x)
    ax1.set_xticklabels(tasks, rotation=45, ha='right')
    ax1.legend()
    ax1.set_ylim(0, 1)

    # 图2: 雷达图
    ax2 = axes[0, 1]
    categories = ['Accuracy', 'Robustness', 'Fairness', 'Safety', 'Efficiency']
    values = [0.85, 0.78, 0.82, 0.90, 0.75]
    values += values[:1]  # 闭合
    angles = np.linspace(0, 2 * np.pi, len(categories), endpoint=False).tolist()
    angles += angles[:1]

    ax2 = fig.add_subplot(222, polar=True)
    ax2.fill(angles, values, alpha=0.25, color='#4299e1')
    ax2.plot(angles, values, 'o-', linewidth=2, color='#4299e1')
    ax2.set_xticks(angles[:-1])
    ax2.set_xticklabels(categories)
    ax2.set_ylim(0, 1)
    ax2.set_title('Multi-Dimension Evaluation')

    # 图3: 历史趋势
    ax3 = axes[1, 0]
    versions = ['v1.0', 'v1.1', 'v1.2', 'v2.0', 'v2.1']
    trend_scores = [0.72, 0.75, 0.78, 0.83, 0.87]
    ax3.plot(versions, trend_scores, 'o-', color='#48bb78', linewidth=2, markersize=8)
    ax3.fill_between(versions, trend_scores, alpha=0.1, color='#48bb78')
    ax3.set_xlabel('Model Version')
    ax3.set_ylabel('Overall Score')
    ax3.set_title('Performance Trend')
    ax3.grid(True, alpha=0.3)

    # 图4: 错误分布
    ax4 = axes[1, 1]
    error_types = ['Factual', 'Reasoning', 'Language', 'Safety', 'Other']
    error_counts = [23, 18, 12, 5, 7]
    colors = ['#fc8181', '#f6ad55', '#fbd38d', '#c6f6d5', '#bee3f8']
    ax4.pie(error_counts, labels=error_types, autopct='%1.1f%%', colors=colors)
    ax4.set_title('Error Distribution')

    plt.tight_layout()
    plt.savefig(output_path, dpi=150, bbox_inches='tight')
    plt.close()
    print(f"📊 Dashboard saved to: {output_path}")


def generate_text_report(results: dict, output_path: str):
    """生成文本格式的评估报告"""
    report_lines = [
        "=" * 60,
        "AI MODEL EVALUATION REPORT",
        f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
        "=" * 60,
        "",
        "SUMMARY",
        "-" * 40,
    ]

    total = 0
    passed = 0
    for task_name, task_data in results.get("results", {}).items():
        total += 1
        acc = task_data.get("acc", 0)
        threshold = task_data.get("threshold", 0)
        status = "PASS" if acc >= threshold else "FAIL"
        if status == "PASS":
            passed += 1
        report_lines.append(f"  {task_name:30s} {acc:.4f}  [{status}]")

    report_lines.extend([
        "",
        f"Total Tasks: {total}",
        f"Passed: {passed}/{total} ({passed/total*100:.0f}%)",
        "",
        "RECOMMENDATIONS",
        "-" * 40,
    ])

    # 自动生成改进建议
    for task_name, task_data in results.get("results", {}).items():
        acc = task_data.get("acc", 0)
        if acc < 0.5:
            report_lines.append(
                f"  ⚠️  {task_name}: Score critically low ({acc:.4f}). "
                f"Consider fine-tuning or prompt engineering."
            )
        elif acc < 0.7:
            report_lines.append(
                f"  📌 {task_name}: Room for improvement ({acc:.4f}). "
                f"Review error patterns for optimization opportunities."
            )

    report_lines.extend(["", "=" * 60])

    report_text = "\n".join(report_lines)
    with open(output_path, 'w', encoding='utf-8') as f:
        f.write(report_text)

    print(f"📄 Report saved to: {output_path}")
    return report_text

错误分析的最佳实践

章节总结

本附录系统介绍了AI评估与基准测试的核心知识：