Questions to Ask an AI Agency Before You Hire

A document system for a law firm, accounting office, or logistics team can look impressive in a demo and still misread the number that matters. I'd use these questions to ask an AI agency whether it can prove the system works on your data, explain what failure looks like, and leave you in control after launch.

The choice isn't simply whether to hire an agency. You're deciding which model and workflow to pay for, how much human review to keep, and who carries the work if the system breaks. My first question would be: how did you choose the model for my specific documents, rather than the one that looked strongest in a vendor benchmark?

Ask how the agency tests models on your documents

A model benchmark is a test on a prepared set of examples. It can help compare models, but it doesn't tell you whether one will read your scans, invoices, contracts, or forms correctly. I'd ask the agency to show its test method and results on a sample of your own documents.

We faced that decision with ChatsControl, which reads scanned documents that go to embassies, courts, and immigration offices. We tested Gemini Flash against Gemini Pro in a blind test on 26 real scans comprising 44 pages, then chose Flash: Pro scored better on text, but cost $0.32 a page against $0.021 and took 147 seconds against 62. On one certified document, Pro read 8905 as 6905. The choice brought processing to about two cents per page, and the example shows why average accuracy alone isn't enough. (A document translation platform that turns scans and PDFs into editable Word files)

Those are results from that specific project, not a general ranking of models. The useful question for your agency is whether it can show a similar comparison for your documents, with cost per page, processing time, average performance, and the worst consequential error side by side.

Blind testing means the person judging the outputs doesn't know which model produced each one. That reduces the chance that a familiar model name or a confident-looking answer gets an easier pass. My advice is to agree on the test documents and what counts as an error before the agency starts.

For the decision rule, I'd start with a blind test on 20 to 50 of your own documents. Check what the most damaging error would mean in your workflow, not only how many answers look right. A model that is cheaper and slightly weaker on average may still be the sensible choice; a single wrong account number or legal reference may make that trade unacceptable.

Ask the agency	What a useful answer includes
Which models did you test?	The model names and why each fits the task
What documents did you use?	A representative sample from your workflow
How did you compare outputs?	Blind review and clear error criteria
What did the test find?	Cost, speed, average quality, and worst error
What did you choose?	A reason tied to your workload and the cost of mistakes

Ask who checks the AI's work

A production workflow needs a clear answer to one plain question: who checks the output before it reaches a customer, court, or business record? A model can draft, classify, or extract information, but your agency should tell you which steps need human approval and which actions the system can safely take on its own.

In Mata v. Avianca, a U.S. court found that lawyers submitted non-existent judicial opinions with fake quotations and citations generated with ChatGPT, and continued standing by them after questions were raised. The order explains that using an AI tool was not inherently improper, but attorneys still had a gatekeeping role and responsibility for checking filings. (The court's opinion and order in Mata v. Avianca)

I'd ask the agency to walk through a real output from your process: what information it relies on, what a person checks, and what the system does when it can't support an answer. A good answer names the reviewer and the approval point. "The model is accurate" doesn't tell you who catches an error before it causes trouble.

A human review step costs time, so it should match the risk. A draft for an employee to edit is different from an automatic payment, an external legal filing, or a message sent to a customer without review. Ask the agency to show how it blocks risky actions until someone approves them, and how it records what the reviewer changed.

Ask what happens when the system fails

An automation is a sequence of steps that moves data or work between systems. If one step fails, the next question is whether the system stops safely, alerts someone, and lets an authorized person retry the work without duplicating it.

Ask the agency to demonstrate a failed run, not only a successful one. For example: what happens if a document is unreadable, an account connection expires, or a destination system is unavailable? Who receives the alert, what does the alert include, and how does an operator know whether a retry is safe?

A July 2026 discussion from a small-business buyer raised questions about ownership, access controls, logs, tests, maintenance, and failure handling in AI-assisted automations. One practical question from that discussion was whether a failed automation alerts someone and can be safely rerun. Treat that as a sensible diligence prompt, not as evidence of a particular failure rate. (The July 2026 buyer discussion on AI business automation)

I'd also ask what the agency tests before launch. Tests are checks that verify the system behaves as expected for specified inputs. A useful plan covers ordinary cases and the awkward ones: unreadable files, missing fields, unexpected formats, and outputs that should be sent for a person to review. Ask to see the results and the agreed process for fixing failures.

A fast build doesn't prove the system is dependable, just as a polished demo doesn't show what happens when a job stops halfway through. The agency should explain how it checks the work, how you report a defect, and who responds after launch.

Ask what you will own and how handoff works

Ownership is more than a line in a contract saying you own the project. Ask whether you can access the code, data, prompts, and accounts that the system needs, and what the agency will hand over if you move support elsewhere.

I'd ask for a written list of the accounts the system uses, who controls each one, and what access you'll have during the project. If the agency creates an account in its own name, ask how it will transfer that account or replace it with one you control. Keep the same question for data storage and any third-party services.

Then ask for the handoff in practical terms: what documentation will you receive, where will the system run, and what would another engineer need to maintain it? "We can hand it over" is less useful than a list of deliverables, access rights, and steps to transfer operations.

Some agencies use AI coding tools to build software. That alone doesn't tell you whether the result is safe to maintain. Ask who makes the technical decisions, who reviews code before it ships, and how the system is tested. A July 2026 buyer discussion about AI-assisted development also raised these questions around architecture ownership, access controls, logs, tests, and maintenance. (The buyer discussion on AI-assisted business automation)

Ask for a running-cost estimate with its assumptions

An implementation quote is only part of the cost. Ask what you'll pay to run the system each month, what usage the estimate assumes, and which connected services are included. A forecast should name the model, expected volume, input and output assumptions, tool usage, and the date the rates were checked.

For example, Google's Gemini API pricing page lists Gemini 3.8 Flash standard paid rates of $0.75 per million input tokens and $3.75 per million output tokens through December 31, 2026, with rates scheduled to double on January 1, 2027. The same page lists Gemini 3.8 Flash batch rates at half the standard paid token rates through December 31, 2026, for work that can finish asynchronously rather than in real time. (Google's Gemini API pricing page)

Token costs are not the only possible charge. Google's pricing page says individual Google Search queries performed through Gemini grounding are chargeable, with 5,000 free monthly requests shared across Gemini 3-and-newer models and then $14 per 1,000 requests. If your system uses connected tools, ask the agency to include those costs rather than estimating only model usage. (Google's Gemini API pricing page)

I'd want the estimate to say what happens when usage rises, vendor prices change, or the agency recommends a different model. Ask whether the system can use a cheaper model for routine inputs and send difficult cases for review or a stronger model. That approach needs to be tested on your documents; it isn't a saving if it increases the errors that matter.

Cost item	What to ask
Model usage	Which model and what input, output, and volume assumptions?
Connected tools	Are search, storage, or other service charges included?
Price changes	When was the estimate checked, and what changes if rates move?
Support	What work is covered after launch, and what is billed separately?
Higher usage	How does the estimate change if your document volume grows?

Ask how your data is handled

Data privacy depends on the services and account setup behind the workflow, not just on what the agency says in a meeting. Ask where your documents go, which model provider receives them, what tier or account the system uses, and what the provider's terms say about that data.

Google's Gemini API pricing page distinguishes free, paid, and enterprise tiers. It says content in the free tier may be used to improve Google products, while content in the paid tier is not used for that purpose. Ask the agency to identify the tier and data terms in the proposed production setup, then have your lawyer or compliance officer confirm what applies to your business. (Google's Gemini API pricing page)

A data requirement can also change which model setup makes sense. If your rule is that confidential files must not go to a cloud API, ask whether a locally run model is an option and what trade-offs it would create for cost, speed, and quality. I'd want the agency to explain those choices before it builds around a service that conflicts with your requirements.

Compare your options and ask what support includes

An off-the-shelf tool may suit a familiar workflow you can configure without custom software. A freelancer can be a fit for a narrow integration if you can define the work and handle continuity yourself. An agency may suit a workflow that needs several systems connected, testing on your data, and a planned handoff. An in-house hire gives you someone inside the business, but you still need to define the problem and assess the build.

Route	Ask what you are getting	What to weigh
Off-the-shelf tool	Does it fit your documents and approval steps?	Setup effort, limits, and ongoing subscription
Freelancer	Who tests the system and supports it later?	Scope, availability, and handoff
Agency	Who owns testing, delivery, and maintenance?	Defined responsibilities and continuing costs
In-house hire	Who sets the workflow and reviews the system?	Internal time and the skills required

For a defined task, WebDaddy lists a fixed-price AI agent package starting at $500 and taking 7 days; it covers one routine task automated end to end. More involved packages connect the agent to email, a CRM, or documents, and have a person approve key steps. (WebDaddy's AI services and packages) I'd compare that kind of defined scope with the actual work you need, not assume a package covers a custom system or every ongoing support need.

Support terms should say how you report problems, what counts as a bug in the agreed scope, who maintains the system, and what happens when a model or connected service changes. Ask whether maintenance includes monitoring and repairs, or whether it only covers questions by email. A handoff plan matters even if you expect the original agency to stay involved.

Use this checklist before you sign

A useful AI agency vetting process turns broad promises into evidence you can review. I'd ask the agency to answer each item in writing and tie the answer to your documents, workflow, and operating budget.

1. Name the workflow. Write down what comes in, what the AI should do, what leaves the system, and which steps need approval.
2. Choose a document sample. Provide representative examples, including difficult scans or unusual formats, and agree on what counts as an error.
3. Run a blind model test. Compare candidate models on your documents, including cost, speed, average quality, and the worst consequential error.
4. Set the review point. Name who checks high-risk outputs and which actions the system must pause until a person approves them.
5. Test failure handling. Ask the agency to show how the system alerts an operator and how a failed task can be retried safely.
6. Confirm ownership and data terms. List the code, data, prompts, accounts, and service tiers, then confirm who controls and can access each one.
7. Review running costs and support. Get a dated estimate with usage assumptions and connected-tool charges, plus written terms for maintenance and handoff.

The larger model is not always the wrong choice. If the quality gap is exactly what your customer pays for and your document volume is low, paying more for the larger model can make sense. My rule is to test the models on your own documents first, then make the choice against the cost of the worst error your business can accept.