4. METEOR指标实现与语义匹配

任务说明

本案例实现METEOR（Metric for Evaluation of Translation with Explicit ORdering）指标，用于评估机器翻译或文本生成结果与参考文本之间的语义匹配质量。METEOR在精确匹配基础上引入词干还原和同义词匹配，并计算碎片化惩罚，从而比单纯BLEU更能反映语义等价性。你需要完成一个可独立运行的Python实现，输入为候选文本与参考文本，输出METEOR分数。

核心实现

import re
from collections import Counter
from nltk.stem import PorterStemmer
from nltk.corpus import wordnet

stemmer = PorterStemmer()

def tokenize(text):
    return re.findall(r'\b\w+\b', text.lower())

def get_synonyms(word):
    syns = set()
    for syn in wordnet.synsets(word):
        for lemma in syn.lemmas():
            syns.add(lemma.name().lower().replace('_', ' '))
    return syns

def align_words(cand_tokens, ref_tokens):
    """贪心对齐：优先精确匹配，其次词干匹配，最后同义词匹配"""
    cand_matched = [False] * len(cand_tokens)
    ref_matched = [False] * len(ref_tokens)
    stem_cand = [stemmer.stem(w) for w in cand_tokens]
    stem_ref = [stemmer.stem(w) for w in ref_tokens]
    syn_cache = {}

    # 阶段1：精确匹配
    for i, cw in enumerate(cand_tokens):
        for j, rw in enumerate(ref_tokens):
            if not cand_matched[i] and not ref_matched[j] and cw == rw:
                cand_matched[i] = True
                ref_matched[j] = True
                break

    # 阶段2：词干匹配
    for i, cs in enumerate(stem_cand):
        if cand_matched[i]:
            continue
        for j, rs in enumerate(stem_ref):
            if not ref_matched[j] and cs == rs:
                cand_matched[i] = True
                ref_matched[j] = True
                break

    # 阶段3：同义词匹配
    for i, cw in enumerate(cand_tokens):
        if cand_matched[i]:
            continue
        if cw not in syn_cache:
            syn_cache[cw] = get_synonyms(cw)
        for j, rw in enumerate(ref_tokens):
            if not ref_matched[j] and rw in syn_cache[cw]:
                cand_matched[i] = True
                ref_matched[j] = True
                break

    return cand_matched, ref_matched

def meteor_score(candidate, reference, alpha=0.9, beta=3.0, gamma=0.5):
    cand_tokens = tokenize(candidate)
    ref_tokens = tokenize(reference)

    if not cand_tokens or not ref_tokens:
        return 0.0

    cand_matched, ref_matched = align_words(cand_tokens, ref_tokens)
    m = sum(cand_matched)
    if m == 0:
        return 0.0

    # 准确率与召回率
    precision = m / len(cand_tokens)
    recall = m / len(ref_tokens)
    if precision + recall == 0:
        return 0.0
    fmean = (10 * precision * recall) / (recall + 9 * precision)

    # 碎片化惩罚：统计匹配词块数
    chunks = 0
    in_chunk = False
    for i in range(len(cand_tokens)):
        if cand_matched[i]:
            if not in_chunk:
                chunks += 1
                in_chunk = True
        else:
            in_chunk = False

    if chunks > 0:
        frag_penalty = gamma * (chunks / m) ** beta
    else:
        frag_penalty = 0.0

    return (1 - frag_penalty) * fmean

# 示例调用
if __name__ == "__main__":
    cand = "The cat sat on the mat"
    ref = "A cat is sitting on a mat"
    score = meteor_score(cand, ref)
    print(f"METEOR score: {score:.4f}")

依赖与运行

pip install nltk
python -c "import nltk; nltk.download('wordnet'); nltk.download('omw-1.4')"
python meteor.py

验证方法

运行上述示例，预期输出约为 METEOR score: 0.5000（精确值取决于同义词匹配结果）。判定标准：

• 候选与参考完全相同时，输出1.0；
• 无语义重叠时输出0.0；
• 同义词替换（如“car”与“automobile”）应获得非零分数；
• 词序打乱但词集相同，分数应低于完全匹配但高于随机词序。

关键设计说明

本实现采用贪心对齐策略，按精确→词干→同义词三级顺序匹配，确保每个参考词最多匹配一个候选词。碎片化惩罚通过统计连续匹配块数计算，惩罚词序混乱。同义词匹配依赖WordNet，需提前下载语料。该实现覆盖METEOR核心计算逻辑，适用于短文本质量评估场景。