DARK FIRESORIGINS UNKNOWN

Why Two UAP Reviewers Agreeing Is Not the Same as Being Right

Generated illustration of a hypothetical two-reviewer classification table with 40 agreements in each category and 10 disagreements in each direction

Written by

in

Independent reviewers can test whether a classification scheme is reproducible, but agreement alone does not identify an object. Jacob Cohen explained the distinction in his own 1986 retrospective on kappa: when both reviewers heavily favor one category, substantial agreement can occur even under independent random labeling.

For a hypothetical UAP exercise, give two reviewers the same 100 clips and predetermined categories without letting them see each other’s labels. Publish the complete cross-tabulation, not merely a percentage.

Suppose each uses the two labels equally and they agree on 80 clips: expected agreement under independent labels is 50%, and kappa = (0.80 − 0.50) / (1 − 0.50) = 0.60. This is an illustrative calculation, not a real dataset or a quality threshold.

Review the disagreements and category definitions; even perfect agreement cannot rule out a shared mistake.

Source: Jacob Cohen’s author-written retrospective on agreement and kappa (1986). The UAP example is our explanatory application, not a finding in Cohen’s paper.

Generated illustration

Featured diagram: Dark Fires. Hypothetical counts, not measured UAP data.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More on this topic