How to choose an AI tool for your own work
Anyone choosing an AI tool today faces a crowded shelf and a lot of confident advice. Reviews can help, but none of them has seen your task. This note describes a way to choose that holds up, and it rests on one idea from the paper it comes from: when several AIs agree, that is not the same as several independent witnesses.
Start with reviews, then test on your own task
Comparison articles are a reasonable first step. They show which tools exist, what each is known for and where people disagree. For example, top10ai.com has written about Poly Duo in its comparison Claude vs ChatGPT, and the same site keeps a list of tools for people who want to build software without writing code: best AI app builder. Lists like these tell you what to try. They cannot tell you which tool fits the thing you actually need to do this week.
So take one real task of your own, small enough to judge yourself, and give the identical request to two or three tools. A draft email, a spreadsheet formula, a short plan, a first version of a small app. Then read the results side by side. You will see differences in tone, in what each tool assumed, and in what it left out, and those are the differences that matter for your work.
A confident answer is not a correct one
The paper makes a point that applies to every tool on the shelf. Someone who asks one language model a question cannot tell a confidently wrong answer from a confidently right one, because on the surface the two look the same. Asking several models seems to fix this, but it mostly moves the problem: if they all say the same thing, you now hold one confident text that carries the borrowed authority of "several of them agreed."
The reason is that models share training data and habits, so their mistakes are correlated. The paper cites a measurement of a panel of nine frontier judges from seven model families that was worth about two independent votes (Kohli, 2026), and a finding that the best single judge matched or beat the whole panel. Adding more models of the same kind does not buy you much more certainty. See the paper's section on how many independent witnesses a council is worth and on the ceiling.
Use a second opinion that can disagree
What helps is a second AI, ideally from a different company, that is allowed to disagree. Two habits are enough in daily use:
- Ask the second tool to check the first answer and name anything it would change, instead of asking it to confirm.
- Keep the disagreements. If two answers differ, that spot is where to look closely, check a source or ask a person.
The paper separates two ways of collecting the second opinion: a blind one, where the second model does not see the first answer, and an informed one, where it does. The informed one can catch a concrete error, while the blind one keeps the two answers more independent. Each trades something for the other, and the discussion of blind and informed elicitation lays out the trade. The paper also notes its own limits: the claim that a merged answer beats the best single one is argued from design and related work, and the proposed evaluation has not yet been run.
A short checklist
- Use reviews to build a shortlist of two or three tools.
- Run the same real task through each and compare the results yourself.
- Pick the one that fits your work, then ask a tool from a different company to check its output on anything that matters.
- Treat agreement as a hint, not as proof, and check facts that carry a cost if they are wrong.
Poly Duo, our app, is built around the second-opinion habit: two AIs from different companies check each other's work on your phone, so the cross-check takes a tap instead of a copy-and-paste. But the method does not depend on any one app. You can do it by hand with any two tools, and the paper describes exactly how.
Vadym Chernets also builds Poly, an app in which AIs from different companies check each other's work.