If you are working out how to choose an AI development company, you have probably noticed that most guides are lists of vendors written by vendors. They tell you who is good. They rarely tell you how to check. This post does the second thing: ten questions you can put to any company, including ours, and what a good answer sounds like. It is about AI work only: agents, automation, chatbots and AI features inside an existing app.
I am a vendor, so use this against us too. If another company answers these better, hire them.
Why the choice is harder for AI work
With a normal app, you can see the screens and click through them. With AI, a demo can look perfect and still fail on your real emails, your real invoices, your real customers. The demo ran on cases chosen to work.
The wider market shows what happens when that is not checked. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, citing rising costs, unclear business value and weak risk controls. Those three reasons map to the checklist below: cost, measurement and review.
How to choose an AI development company: the 10-question checklist
Print this, send it to three vendors and compare the written answers. Silence on a question is an answer.
1. How will you test it on my real cases?
Ask what the test set is, who builds it and how big it is. A good answer: you collect real examples together, agree what “correct” means for each, and the vendor runs the work against that set on every change. A weak answer: “our models are very accurate.”
Ask to see an example of an evaluation set from a past project, with the data removed.
2. What is the accuracy target, and when do we agree it?
The number should be written in the proposal before the build starts, measured on your documents. If the target is set after launch, it will be set to whatever the system achieves.
3. Where does a person review the output?
Every AI system makes mistakes. The question is who catches them. Ask which actions run alone and which wait for a person, and what the review screen looks like. Anything that moves money, deletes data or talks to a customer should be able to wait for approval, and every step should be logged.
4. How do you track cost per request?
The build is one cost. Running it is another: every request can mean several model calls and tool calls. Ask for an estimate of cost per request on your examples, and ask how it will be tracked after launch. Our own take on the build side is in what drives AI agent development cost.
5. Who owns the code, the prompts and the evaluation sets?
The answer should be you, in writing. The evaluation sets matter most. Without them, a new team can’t tell whether a change helps or breaks things, and you are locked in without anyone saying so.
6. What are the data terms?
Ask three things. Is my data used to train models? Whose accounts do the AI services run on? Who gets access, and can I revoke it? A good vendor uses business API plans that exclude training on your data, runs services on accounts you own, and gives each agent only the permissions its job needs. Check the model provider’s own terms too, for example OpenAI’s business data page, because the vendor’s promise sits on top of theirs.
7. Is the quote fixed or hourly?
Both can be honest. A fixed quote is easier to compare and puts the risk of unknowns on the vendor, but only if the scope is written down. Hourly billing suits open research, but ask for a cap and a weekly demo. Whichever it is, ask what the quote leaves out: model usage, hosting and paid APIs are often billed separately.
8. Do I talk to the people who will build it?
Many companies send a sales lead to the first call and a different team to the build. Ask who will be on the project, whether they will be on the first call, and whether you can ask them technical questions before signing.
9. What do I see every week?
Working software on your real cases, not slides. Weekly demos give you a chance to stop early if quality is not where it should be.
10. What happens when the model changes?
Models get replaced, repriced and retired. Ask how the system is built to swap models, and how the evaluation set tells you whether the swap is safe. A cheaper model is often the biggest lever on running cost, but only if you can test it.
Good answers and red flags
| Topic | A good answer | A red flag |
|---|---|---|
| Testing | Real cases from you, a written accuracy target | Demo on their own data, “very accurate” |
| Review | Named actions that wait for a person, a log | “The AI handles everything” |
| Cost per request | Estimated on your examples, tracked from version one | Only a build price, no run estimate |
| Ownership | Code, prompts and evaluation sets are yours in writing | Prompts or tests stay on their side |
| Data | No training on your data, accounts you own | Vague answer, shared keys |
| Quote | Fixed with a written scope, or hourly with a cap | Open-ended time and materials |
| Team | You meet the builders before signing | Sales on the call, strangers on the project |
What I would not compromise on
My opinion: ask to see how the vendor will measure quality on your cases before you sign, and walk away if they can’t show you. Everything else on the list can be negotiated. A budget, a timeline or a tool choice can change after the first demo.
The reason is simple. Without a measurement agreed up front, every later argument turns into opinion. The vendor says it works. You say it does not. Nobody has a number. An evaluation set turns that into a score you both can read, and it stays useful after the project ends, because the next team can use it.
I would also put the review step next to it. If a vendor can’t tell you which actions wait for a person, they have not thought about what happens when the system is wrong.
How to run the process
You do not need a long procurement round. This is enough:
- Write down one process. Pick work that happens daily or weekly, runs on text or documents, and that a person reviews today.
- Collect 20 to 30 real examples. Include the messy ones. Share them under an NDA.
- Send the ten questions to three vendors. Ask for answers in writing.
- Compare scope, not totals. A low price that leaves out evaluations or the review step has moved that cost to after launch.
- Talk to the engineers. Ask each vendor to put a builder on a call.
- Start small. A first phase with a fixed quote and weekly demos costs you little if it is wrong.
Not every process suits AI. High-stakes decisions where nobody can review the output, work that happens a few times a year, and data you can’t share or that does not exist yet are poor fits. A good vendor says so before taking your money.
How we answer these
Here are our answers, so you can compare them with everyone else’s. We start with a free 30-minute discovery call, and the engineers on it are the people who build the project. We map the process, look at real examples and check the data and tools. Then comes a scoped proposal: what the AI will do, where a person reviews it, how success is measured and a fixed quote. There is no hourly billing and no published price for AI work.
We build with evaluations, with weekly demos on your real cases. Agents ship with a review step, and every step is logged. Cost per request is estimated in the proposal and tracked from the first version. You own the code, the prompts and the evaluation sets. We use business API plans whose terms exclude training on your data, and AI services run on accounts you own. If you need an NDA, we sign it before the first call.
You can see the range of work on our AI services page. If you want to test us against the list, bring a few real examples to a call and ask the ten questions.
Questions people ask
How do I choose an AI development company?
What should an AI development company show me before I sign?
Is a fixed quote better than hourly billing for AI work?
What are red flags when hiring an AI development company?
Should the vendor own my prompts and evaluation sets?
