log inhelp

 | 
View
 

Confidence Interval

Page history last edited by Mike 3 months, 1 week ago

Confidence Interval System — Technical Requirements for Developers

Goal: Develop a system that assigns a confidence interval (CI) to every belief score in the Idea Stock Exchange (ISE). The CI reflects how much trust we should place in the current score, based on user evaluation patterns, argument stability, and knowability of the belief.

The CI is a 0-100 measure of reliability, not the belief's truth or score.
High CI = "We can trust this number."
Low CI = "Score may change significantly with new evaluations."

1. Overview of What the System Must Do

The CI must automatically rise or fall based on:

  1. Depth and breadth of user evaluation
  2. Stability of the belief's score over time
  3. Nature of the belief (knowability of underlying facts)
  4. Failed attempts to overturn the belief's supporting arguments

The confidence interval is computed independently of the belief's truth score.

2. Required Data Inputs

The system must gather or maintain these fields for each belief page:

  • Total reading time by verified users
  • Number of independent verified readers
  • Number of unique pro-arguments evaluated
  • Number of unique con-arguments evaluated
  • Number of arguments whose scores changed after review
  • Stability metric: how much the belief's score changed over time
  • Argument redundancy ratio: proportion of new arguments that repeat old ones
  • Knowability classification (Category 1-4)
  • Downvotes / contested-argument flags
  • Time since last major score change

3. Four Core Factors That Increase the Confidence Interval

(1) User Examination Depth

CI must increase as more verified users meaningfully examine the belief and its arguments.

Program needs to track:

  • Cumulative verified user time (minutes)
  • Number of unique verified reviewers
  • Number of user interactions (argument expansion clicks, evidence verification, etc.)
  • Number of new contested points found (lower = better CI)

CI rises when:

  • More users review the arguments
  • Reviewers spend more time reading
  • No new major objections are raised
  • No reason to agree or disagree is downvoted for low quality

CI falls when:

  • New contested points appear
  • Downvotes indicate weak argumentation
  • Low number of independent reviewers

(2) Score Stability Despite Deep Review

CI should increase when the belief score remains stable even after thorough evaluation.

Developers need:

  • A rolling 30-day score-change chart
  • Standard deviation of the belief score over time
  • Number of new sub-arguments added during this period
  • Whether evaluations changed sub-argument scores meaningfully

CI rises when:

  • High user activity + minimal score movement
  • Sub-argument scores stabilize
  • Repeated evaluation yields the same conclusion

CI falls when:

  • Large swings in score
  • New arguments overturn previous evaluations
  • Score volatility increases even with many readers

(3) Knowability of the Belief

Some beliefs are testable; others are not. CI must reflect how knowable a belief is.

Developers must assign or allow users to assign a Knowability Category:

Category Description CI Impact
1 Testable facts (economic stats, scientific claims) Highest possible CI
2 Partially testable (policy predictions, economic forecasting) Moderate-high CI
3 Value judgments with empirical components Moderate CI
4 Pure philosophy/speculation Maximum CI capped low

CI rises more quickly for:

  • Beliefs with measurable outcomes
  • Beliefs supported by Tier 1-2 evidence
  • Beliefs where historical or scientific tests exist

CI rises slowly or is capped for:

  • Moral claims
  • Hypothetical futuristic scenarios
  • Philosophical positions

The system must apply CI caps based on this classification.

(4) Resistance to Overturn Attempts

CI should increase when people try to defeat the belief but fail.

This reflects the "peer-review" effect of scientific consensus.

Program must track:

  • Number of new arguments proposed
  • Percentage of new arguments marked "redundant"
  • Number of unique reviewers attempting to add counterarguments
  • How often the belief score significantly changes after challenges

CI rises when:

  • New challenges repeat the same already-refuted points
  • New counterarguments fail to change the score
  • Attempts to challenge the belief decrease over time

CI falls when:

  • New arguments reveal weaknesses not previously considered
  • New challenges cause score shifts
  • Evidence-based objections remain unresolved

4. Combining the Four Factors Into a Unified CI Score

Developers should compute the CI as a weighted composite index:

CI = w1 × (User Examination) + w2 × (Score Stability) + w3 × (Knowability) + w4 × (Challenge Resistance)

Default suggested weights (tunable):

  • User Examination: 30%
  • Score Stability: 30%
  • Knowability: 20%
  • Challenge Resistance: 20%

The CI should update dynamically as new data arrives.

5. Expected Software Behaviors

A. Automatically Update CI as New Activity Occurs

  • Reviewers read arguments → CI rises
  • Score changes → CI adjusts
  • New counterarguments proposed → system evaluates redundancy

B. Display CI to Users

Display as:

  • High Confidence (85-100)
  • Moderate Confidence (50-84)
  • Low Confidence (0-49)

And show tooltips like:

  • "Score stable over last 60 days."
  • "High argument redundancy suggests consensus."
  • "New objections appeared; CI reduced."

C. Prevent High CI for Low-Knowability Beliefs

Philosophy topics must never reach 100, even if no disagreements appear.

D. Reward Expert-Reviewed Content

CI must rise more sharply when evaluation is done by verified:

  • Economists
  • Scientists
  • Legal experts
  • Statisticians
  • Subject matter experts

6. Data Requirements and Storage

Developers must store:

  • Per-user reading time
  • Argument evaluation logs
  • Score-change timestamps
  • Number and type of new objections
  • Argument redundancy calculations
  • Knowability category
  • Reviewer verification level

7. API / System Requirements

The system must provide endpoints or internal functions for:

  • getConfidenceInterval(beliefId)
  • updateConfidenceInterval(beliefId)
  • logUserReviewAction(beliefId, userId, duration, interactions)
  • logNewArgument(beliefId, argumentId, redundancyScore)
  • getScoreStabilityMetrics(beliefId)

8. Example CI Outcomes

High CI Belief Example (Climate)

"Solar power costs have decreased over the past 20 years."

  • Knowability: Category 1
  • 100+ verified readers
  • No new counterarguments
  • Score unchanged for 4 months

Expected CI: 95

Moderate CI Belief Example (Social Media Regulation)

"Restricting algorithmic feeds will reduce polarization."

  • Knowability: Category 2
  • Active debate
  • Some new evidence
  • Score fluctuates moderately

Expected CI: 65

Low CI Belief Example (Moral Philosophy)

"Universal basic income is morally mandatory."

  • Knowability: Category 3-4
  • Score swings
  • Many new arguments emerging
  • No clear consensus

Expected CI: 30

9. Final Requirement

The confidence interval must always represent how likely the belief's score would remain within ±10 points if 100 new experts reviewed it tomorrow.

This gives the CI a precise, interpretable meaning.

10. Integration with ISE Framework

Links to ISE components:

11. Next Steps for Implementation

Developers can:

  • Design database schema for storing CI metrics
  • Build JSON specification for CI data structure
  • Develop ranking algorithms for CI-weighted belief sorting
  • Create user-interface mockups showing how CI is displayed
  • Implement dynamic CI calculation engine
  • Build redundancy detection algorithms

Contact me to collaborate on implementation.

View the technical framework on GitHub for existing codebase and architecture.

 

Comments (0)

You don't have permission to comment on this page.

Printable version