Advanced reasoning strategies

Beyond a single pass

Every technique so far has mostly been about getting one good answer from one call (or one chain). This lesson covers strategies that spend extra computation to improve reliability: sampling several answers, checking answers, exploring alternatives, and planning before acting.

None of these makes a model infallible. Each one costs more tokens, more time, or both — and each has specific failure modes. The skill is knowing which one fits your problem.

1. Self-consistency: sample, then compare

Idea: ask the same question several times (with some randomness, e.g. a non-zero temperature), then compare the final answers. For problems with a single correct answer — a number, a category, a yes/no — take the most common one.

Run 1: ... final answer: 42
Run 2: ... final answer: 42
Run 3: ... final answer: 38
Run 4: ... final answer: 42
Run 5: ... final answer: 42   -> majority: 42 (4 of 5 agree)

Why it can help: if a model sometimes slips on a step, independent attempts are unlikely to all make the same slip. Agreement across runs is also a useful signal of confidence; disagreement is a flag for human review.

Limits:

• Costs roughly N times as much as one call.
• Works best when answers can be compared exactly. For open-ended writing there's nothing simple to vote on.
• If the model has a systematic misunderstanding, it can be consistently wrong. Agreement is not proof.

2. Self-critique and verification passes

Idea: after generating an answer, run a separate step that checks it against explicit criteria, then revise if needed.

A vague "check your work" often produces a cheerful "Looks correct!". A good verification pass is specific: it names what to check, against what, and requires evidence (quotes, recalculations) for each verdict.

Practical tips:

• Run verification as a separate call with fresh context, rather than asking in the same breath as generation.
• Give the checker something external to check against: source text, test cases, a rubric, the original requirements.
• Where possible, verify with tools rather than words: run the code, execute the tests, recompute the figure, validate the JSON. A passing test is stronger evidence than a model saying the code "looks right".

Limits: models checking their own (or similar) output tend to share blind spots, and can either miss real errors or "fix" correct content. Verification lowers error rates; it doesn't eliminate them.

3. Considering alternatives

Idea: before settling on an answer, ask for several distinct candidate approaches or interpretations, evaluate them against criteria, and then choose.

Our web app's dashboard page has become slow to load (from ~1s to ~6s) since last week's release.

1. List 4 distinct hypotheses for the cause. Make them genuinely different (e.g. database, frontend bundle, network/API, caching).
2. For each, give: the evidence that would confirm or rule it out, and the cheapest check to run first.
3. Rank the checks by (likelihood x ease), and explain the ranking in one line each.

Don't pick a single cause yet — I'll run the checks and report back.

This counters a common failure: the model latching onto its first plausible idea. It's especially useful for debugging, design decisions, ambiguous questions and strategy, where the first answer is rarely the only reasonable one.

Limits: the model may produce alternatives that are superficially different but really the same idea, or pad the list. Ask for distinct options with explicit criteria, and keep the count small.

4. Plan, then execute

Idea: split a complex task into a planning step and an execution step. First ask for a plan (outline, steps, files to change, queries to run). Review or adjust it. Then execute it step by step.

Call 1 (plan):    "Propose a step-by-step plan to migrate this module from callbacks to async/await. List the files, the order, and the risks. Don't write code."
[You review and edit the plan]
Call 2..n (do):   "Implement step 2 of the approved plan below. Only change the files listed for that step."

Why it helps: problems in the approach are cheaper to catch in a plan than in finished work. It also gives you a natural human checkpoint and a structure for chaining.

Limits: plans can look tidy and still be wrong; the model may also drift from the plan during execution. Re-anchor each execution step to the approved plan, and check results against it.

Choosing a strategy

You can combine them — for example, plan-then-execute with a verification pass after each step — but every addition multiplies cost and latency. Measure the benefit on your test set (from the evaluation lesson) before making a strategy permanent.