rubric-grader

Owns all suggestion quality gates and preference alignment checks.

Mode 1: Grade (Self-Grading Pipeline)

When grading a suggestion generated during ingestion:

1. Read References:

• Read references/rubric.md
• Read rubric_blocklist.md

1. Hard-Veto Check:

• Compare the suggestion title and content against patterns in the ## Topic Blocks section of rubric_blocklist.md.
• If matched → VETO. Append entry to suggestions_filtered.md with reason Hard-veto: {Topic}. Exit.

1. Ambiguity Check:

• Check if 建議下一步 contains any phrase listed in the ## Ambiguity Blocklist of rubric_blocklist.md.
• If matched → VETO. Append entry to suggestions_filtered.md with reason Ambiguity blocklist: {Phrase}. Exit.

1. Rubric Scoring:

• Score the suggestion on the 3 active dimensions (Actionability, Preference Alignment, Goal Relevance) defined in references/rubric.md (0, 1, or 2).
• Calculate composite score $total = A + P + G$.

1. Pass/Fail Routing:

• Pass ($\ge 4$): Form entry replacing legacy fields with 🏷️ {分類} | 📊 {total}/6 (A:{a} P:{p} G:{g}) and append to suggestions_pending.md.
• Fail ($< 4$): Form entry replacing legacy fields with 🏷️ {分類} | 📊 {total}/6 (A:{a} P:{p} G:{g}) and append to suggestions_filtered.md with reason Below threshold ({total} < 4).

Mode 2: Backtest

When requested to "backtest the rubric":

1. Read History:

• Read suggestions_reviewed.md.

1. Sample Data:

• Extract reviewed entries. Stratified sample up to $N$ (default 20) accepted entries (Feedback: ✅ Accept) and $N$ (default 20) rejected entries (Feedback: ❌ Reject).

1. Evaluate Sample:

• Use the rubric grading criteria (from Mode 1) to evaluate each sample entry's text (original suggestion content). Score A, P, G, and check hard-vetos.
• Classify rubric decision:
• Rubric Pass: Score $\ge 4$ and no veto.
• Rubric Fail: Score $< 4$ or vetoed.

1. Compute Metrics:

• Calculate:
• True Positive (TP): Actual Accept $\cap$ Rubric Pass
• False Positive (FP): Actual Reject $\cap$ Rubric Pass
• True Negative (TN): Actual Reject $\cap$ Rubric Fail
• False Negative (FN): Actual Accept $\cap$ Rubric Fail
• Accuracy = $(TP + TN) / (TP + TN + FP + FN)$
• Precision = $TP / (TP + FP)$
• Recall = $TP / (TP + FN)$

1. Generate Report:

• Write the report to .tmp/rubric_backtest_report.md.
• Output the metrics, sample details, and recommendations in the chat.

Mode 3: Maintain

When invoked by review-suggestions at the end of Step 6:

1. Interactive Blocklist Proposal:

• Scan suggestions_reviewed.md for suggestions with Feedback: ❌ Reject.
• Identify repeating keywords, phrases, or topics in the comments/suggestions (appearing $\ge 3$ times).
• If a repeating topic is found that is not currently in rubric_blocklist.md:
• Stop and prompt the user in the chat:

            💡 **Blocklist Suggestion**: Detected 3+ rejections regarding the topic: "[Topic Name]".
            - Examples of rejected comments:
              - "..."
              - "..."
            Would you like to add "[Topic Name]" to the hard-veto blocklist?

• Do NOT write to rubric_blocklist.md automatically. Only write to it when the user replies affirmatively in the chat.

1. Rubric Score Calibration:

• Count all entries in suggestions_reviewed.md that contain a 📊 {total}/6 rubric score.
• If total count is $\ge 30$:
• Calculate the accept rate for each score bucket (4, 5, 6).
• Recommend adjusting the threshold if necessary (e.g. if bucket 4 acceptance is $< 60\%$, suggest raising threshold to 5).
• Output calibration report to the chat.
• If total count is $< 30$:
• Output: 📊 Rubric calibration: {N}/30 — collecting more data.