Methodology
How we mark and rate what we publish
What each mark, ribbon and rating on the site means, and how we decide on it. This page is also our rubric and our credibility guidelines.
01
How to read this site
Every research claim carries one of four marks, each with its own glyph and line style.
From a sourceOur readAI-suggestedUnclear
Every milestone sits in one of three stages. On the timeline, color shows the stage and nothing else.
- Possible
- Widely available
- Actually helping
We never average forecasts into one date. Each one is shown on its own with who made it, when, and the exact quote, and a team range never hides them.
When two forecasts for the same milestone are more than 10 years apart, or one says it will never happen, the milestone gets a Sources disagree stamp.
Unclear means we can’t make a responsible call yet. It always gives a reason, such as no relevant evidence, conflicting evidence, a proposal too vague to assess, or a search we haven’t finished.
02
The four marks
Claims in milestones, forecasts, source summaries and scorecards all carry a mark. The glyphs and line styles keep the four apart even in black and white.
Taken straight from a cited source, with the exact quote.
The team's own interpretation.
Suggested by an AI model. The tag names the person who checked it, or says it hasn't been checked yet.
We don't know, and the tag gives the reason.
More on each mark
- From a source
- The quote is 300 characters at most and links to the source, so you can check it yourself.
- Our read
- If you read the evidence differently from us, tell us.
- AI-suggested
- Once a person checks the line, the dashed outline stays and the label names who checked it.
- Unclear
- The reason always comes from the list under the scorecard's rating scales below.
Nothing AI-written goes on the site until a person has checked it, with one exception while the site is a draft: the brain computer interface and neuromorphic AI pages are AI drafts the team hasn’t reviewed. They say so, and so does every excerpt from them elsewhere on the site, like the cards on the home page.
03
The three stages
Each stage asks a different question: can it be done, can people get it, and does it help? Claims about brain tech often run these together. Every milestone gets exactly one stage.
- Stage 1
Possible
Reached once someone shows it can be done at all.
A lab demo is enough to count. It tells you nothing yet about cost, safety or who can get it.
- Stage 2
Widely available
Reached once it's reliable and affordable enough for lots of people to get.
Most of the hurdles at this stage are practical ones: cost, safety, regulation and supply have to work out for people outside a handful of labs and trials.
- Stage 3
Actually helping
Reached once it measurably helps with alignment or human agency.
The proposals are aiming for this stage, and wide use alone doesn't get a technology there.
04
The timeline
Each milestone sits on the track at the years it's expected, if anyone has said. What it's drawn with depends on the estimates we have, and forecasts are never merged into one date.
- Solid ribbonThe team has set a range and written down why. The solid ribbon takes priority, and any forecasts still show next to it as thin lines.
- Faint ribbonThere's no team range, and at least two forecasts roughly agree. A faint ribbon covers the years they span, with each forecast as a thin line over it.
- Thin lineOne forecast. The milestone's panel says who made it and when, the window they gave, and the exact quote.
- Red thin linesSources disagreeForecasts that disagree. There's no ribbon at all, just each forecast on its own and the stamp.
- DotThe milestone has already happened, and the dot sits at the year it did.
- Cross in the Never zoneSomeone said the milestone will never happen. Their forecast is a cross in the Never zone near the end of the track, so it stays in view next to the forecasts that give years.
- Timing unknown zoneNo team range and no forecasts yet. The milestone waits in the Timing unknown zone at the end of the track until someone gives it a date.
- A milestone with forecasts says how many and from how many sources, so you can tell a well-studied milestone from a single guess.
- If a forecast’s window has passed and the milestone hasn’t happened, its line fades and it’s counted separately.
The disagreement rule in fullv0.2
Threshold: 10 years
- We compare every pair of forecasts, except two from the same forecaster in the same source, like one report's optimistic and pessimistic cases.
- The same name in two different sources counts as two different forecasters. Two surveys can both report a “median respondent”, and those answers get compared.
- A pair disagrees if one forecaster says it will never happen and the other gives a year.
- A pair also disagrees if their windows are more than 10 years apart, measured from the end of the earlier window to the start of the later one.
- An open end (“by 2045” or “not before 2045”) counts as unbounded, so it can never create a gap in that direction.
- If any pair disagrees, the milestone gets a SOURCES DISAGREE stamp and no faint ribbon. Even if the team decides the forecasts aren't really in conflict, we don't draw one, because it would blur them into one range.
- A faint ribbon needs at least two forecasts, no disagreeing pair and no team range. It runs from the earliest year any forecast allows to the latest. If any forecast is open on one side, the ribbon is open on that side too. If that leaves it open on both sides, we skip the ribbon and show the forecasts alone.
- The team can override the rule either way, but only with a published reason. The milestone then says whether the flag came from our rule or from the team, and what the rule alone would have said.
- A team range never hides a disagreement. The stamp and every individual forecast stay visible.
05
How sources reach the site
A weekly search
Every Monday a script searches arXiv and Semantic Scholar for new papers on these proposals. The team can also start it by hand from the admin.
AI triage, when it's switched on
When the pipeline has an Anthropic API key, an AI model reads each paper's title and abstract, suggests whether it's relevant and how to tag it, and quotes the lines it relied on. Our code checks that each quote really appears in the paper. Without a key, papers wait for review untriaged.
A person reviews every paper
Someone on the team approves, edits or rejects each one. The AI can't approve a paper or put anything on the site.
The AI's first answer is kept
Its original assessment is stored unchanged next to the reviewer's final version. That record shows us where the AI keeps getting things wrong.
Publishing
Approved sources appear on this site the next time the team publishes it.
06
Three scales for sources
We rate credibility, independence and evidence type separately and never combine them into one score. A source can be independent and still weak, or affiliated and still careful. Sources from the team's scouting database start out unrated, and a category marked Draft may still change.
CredibilityDraft4 values
How far we trust a source's claims, judged on its methods and on whether the claims match the evidence. We rate it separately from where the work appeared, since a peer-reviewed paper can be weak and a blog post can be strong.
- Strong
- The methods are transparent, the claims match the evidence, and the work has been reviewed or replicated.
- Adequate
- Reasonable methods, with some gaps or overreach.
- Weak
- The methods are opaque, or the claims go well beyond the evidence.
- Unclear
- We can't judge it yet. The reason is given with the rating.
Source independence3 values
Whether the author or funder has a stake in the result.
- Independent
- No stake in the outcome.
- Non-biased
- Some connection, but no obvious stake.
- Affiliated
- The author or funder has a stake in the outcome.
Evidence type5 values
What kind of support a source gives. An argument and an experiment can both count, but we weigh them differently.
- Argument
- Reasoning or analysis without new data.
- Simulation
- Results from a model or simulation.
- Experiment
- Results from a controlled study.
- Deployment
- Evidence from real-world use.
- Review
- A synthesis of other work.
07
The scorecard's rating scales
Each technology page's scorecard gives six ratings: the alignment case, the Intelligence Curse case, human welfare risk, artificial-mind welfare and rights risk, governance risk and safeguard adequacy. Each one says how confident we are and what it rests on. The risk scale is a draft until the team sets its cutoffs.
Positive-case rating5 values
How strong the case is that a technology helps. We rate the alignment case and the Intelligence Curse case separately.
- Strong
- The mechanism is clear, relevant evidence supports it, and the main alternatives and ways it could fail have been substantially addressed.
- Moderate
- The mechanism is plausible and has some relevant support, but important assumptions or evidence gaps remain.
- Weak
- The mechanism doesn't target the outcome well, depends on especially demanding assumptions, or faces substantial evidence against it.
- Unclear
- We can't make a responsible call either way from the evidence available or the proposal as written. The reason is always given.
- Not claimed
- The proposal doesn't claim this outcome. This is different from Weak or Unclear.
Risk ratingDraft4 values
Used for human welfare risk, artificial-mind welfare and rights risk, and governance risk. Each is judged in the scenario the proposal describes, weighing potential severity, who's exposed, reversibility and the safeguards available. A row that's still Unclear can also carry a “potentially high severity” warning. The warning isn't a rating, and the row stays Unclear.
- High
- The top of the scale.
- Medium
- The middle of the scale, judged on the same factors.
- Low
- The bottom of the scale. It needs evidence behind it and is never given just because nobody has studied a risk.
- Unclear
- We can't make a responsible call yet. The rating names its reason.
Confidence3 values
How sure we are of a rating or an estimate.
- High
- Several relevant, reasonably solid sources support the judgment, and we've considered the main evidence against it.
- Medium
- Some relevant support, but with real limits.
- Low
- The judgment rests mainly on indirect evidence, early results or contested assumptions.
Reason for Unclear5 values
Every Unclear rating names one of these.
- No relevant evidence found
- We looked and found nothing relevant.
- Conflicting evidence
- Credible sources point in different directions.
- Underspecified proposal
- The proposal isn't specific enough to assess.
- Search incomplete
- We haven't finished looking.
- Other
- See the note.
How we handle uncertain moral status
The team hasn’t written this section yet.
08
Other categories
Everything else we tag, from technology and milestone category to source type and claim direction, gets a value from a short named list. The lists live in one shared file, so changing one is a small edit and a new rubric version.
All 10 categories
Technology family4 values
- Whole brain emulation
- Scanning a brain and running it as software.
- Brain computer interfaces
- Devices that read from or write to the nervous system.
- Neuromorphic AI
- AI that borrows its architecture or learning rules from brains.
- Hybrid
- Proposals that combine more than one of the above.
Problem addressed3 values
- Alignment
- Getting advanced AI to reliably pursue what people actually want.
- Intelligence Curse
- AI eroding the economic and political value of human participation.
- Both
- Claims to address both alignment and the Intelligence Curse.
Milestone category8 values
What kind of hurdle a milestone is.
- Reliability and robustness
- Works consistently outside the lab.
- Technical capability
- The technology can do the main thing it's meant to do.
- Safety and welfare assurance
- We can check that it's safe, including for any minds involved.
- Scalable manufacture, compute or data
- Can be produced or run at the scale needed.
- Economic viability
- Affordable enough that people will pay for it.
- Legal and regulatory permission
- Allowed by the relevant law and regulators.
- Institutional capacity and public legitimacy
- Institutions can run it and the public accepts it.
- Equitable or broad access
- Available beyond a small privileged group.
Milestone status5 values
- Not demonstrated
- No one has shown this yet.
- Prototype
- Shown once, in limited conditions.
- Replicated
- Shown more than once, by more than one group.
- Mature
- Routine and dependable.
- Not claimed
- None of our current sources claims the proposal will do this.
Directness to scenario3 values
- Direct
- About this exact scenario.
- Adjacent
- About a closely related scenario.
- Remote
- Only loosely related.
Source type8 values
- Peer reviewed
- Published after peer review.
- Preprint
- Public but not yet peer reviewed.
- Blog
- A blog post or newsletter.
- Report
- A report from an organization.
- Forum post
- A post on a forum like LessWrong or the Alignment Forum.
- Talk
- A recorded talk or workshop presentation.
- Podcast
- A podcast episode or interview.
- Book
- A book or a chapter in one.
Claim directionDraft4 values
Which way a source argues about a proposal. A source can argue more than one way, so it can carry several of these.
- AI alignment
- The source argues the approach could help align AI.
- AI misalignment
- The source argues the approach won't help with alignment, or could make misalignment worse.
- Human empowerment
- The source argues the approach could strengthen people's agency or bargaining power.
- Human disempowerment
- The source argues the approach could weaken people's agency, privacy or bargaining power.
Source kind2 values
Arguments about these proposals are rare, and reports of technical progress are plentiful. We keep the two apart so the arguments don't get buried.
- Argument
- Proposes or criticizes neuroengineering as a response to alignment or the Intelligence Curse.
- Milestone evidence
- Technical progress that bears on a milestone.
RelevanceDraft4 values
Placeholder values until the team agrees on them.
- Core
- Bears directly on a claim or milestone on this site.
- Supporting
- Backs up a core source or adds useful context.
- Peripheral
- Related background, rarely cited.
- Not relevant
- Unrelated to this project.
Time horizonDraft3 values
- Near term, Medium term, Long term
- Definition to be agreed.
09
Versions
Current rubricv0.1-draft
When a category changes, the rubric version goes up. Each AI assessment and each review records the version it was made under, so an old rating can always be read against the rules at the time. Categories marked Draft are still being decided, which is what “draft” in the version number means.
Disagreement rulev0.2
Changing the 10-year threshold or the wording of the disagreement rule bumps this version. The stamps are worked out again every time the site is built, so every one you see follows the current rule.
If you think a rule here is wrong, tell us.