Skip to content

annotate

Build and verify a PII gold set with HUMAN annotators (first-class). Launch the browser annotator, label spans per the codebook, export per-annotator label files, then compute inter-annotator agreement (Cohen's/Fleiss' kappa) and draft an adjudicated gold. Use when the user says "annotate PII", "label this transcript", "build a gold set", "inter-annotator agreement", "review annotations", "adjudicate labels", or wants to measure/defend a de-identification gold standard. Local-only: synthetic or consented data only; annotators' names and transcript text stay on the machine — only labels/stats are collected, nothing PII is re-shared.

Repository Source folder

Details

Path
confide/skills/annotate
Bundled scripts
2
Dependencies
1

Bundled scripts

  • confide/skills/annotate/scripts/gold_to_labels.py
  • confide/skills/annotate/scripts/score_iaa.py

FAQ