How to Choose an AI Automation Agency: 12 Questions to Ask Before You Sign
A buyer’s guide for owners and COOs comparing AI automation agencies: what to ask before you sign, what a good answer sounds like, how to test an agent on your own data and when to walk away.
Key takeaways
- Start from one process and one number, such as hours a week or time to first reply; an agency that asks for that number before it names a model is thinking about your business.
- Pay for a short pilot on your own past cases, with a test set and a pass mark agreed before the build; a demo on the agency’s data proves only that the demo works.
- Get in writing which model provider and region process your data, how long logs are kept and which agreement covers it; the UAE data law covers processing inside or outside the country.
- Keep every account in your company’s name and get the workflows, prompts, code and test set at handover, plus written terms for model updates and support.
- Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027 and warns of agent washing, so ask what the agent decides on its own.
Choose an AI automation agency by its answers to twelve questions about your process, your data and life after launch, not by its demo. A good agency starts from one process and one number, proves the agent on your own past cases in a paid pilot, keeps every account in your name and can tell you what it would not automate. The stakes are real: Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027 "due to escalating costs, unclear business value or inadequate risk controls" (Gartner, 2025).
This guide is for owners and COOs in the US and the UAE comparing agencies for support, sales or back-office work. Inside: the 12 questions with good answers and red flags, a pilot plan, data rules in both markets, handoffs, security, life after launch and agent washing. We build AI automation systems for clients, so we answer from the agency's side of the table.
Start with a process and a number, not with an AI agent
Before the first call, pick one process and write down the number it should move, as one sentence with a baseline and a target: "Leads that arrive after 6 pm wait until morning; we want a first reply within five minutes." An agency that asks for that number before it names a model is working on your business; one that opens with the model is selling technology.
The number separates results from activity. In McKinsey's 2026 survey of 1,719 participants, 37% attribute at least some EBIT impact to AI use, about the same share as a year earlier. Nearly three-quarters of AI high performers report fundamentally redesigning workflows because of AI, against one-quarter of other respondents (McKinsey, 2026). They change the process instead of bolting a model onto it.
Encore, a PR, media and events agency in Dubai, started from the process: what a client agreed to on a call depended on who took notes. Now a call goes in, AI drafts the notes and tasks with an owner, due date and priority, one click files them on the right project board in the agency's CRM, and the assignee gets a Telegram message. Within three weeks, 70 client calls were transcribed and searchable.
12 questions to ask an AI automation agency before you sign
Send the same twelve questions to every agency in writing and compare the answers side by side. Good answers contain names, numbers, documents and links; weak ones contain adjectives.
12 questions, good answers and red flags. NextGenSoft checklist from client projects, October 2026.
| # | Question | A good answer | Red flag |
|---|---|---|---|
| 1 | Which process will change, and which number will show it? | One process, today's baseline and a target | "AI will transform your operations," with no number |
| 2 | Will you run a paid pilot on our data, with a pass mark agreed up front? | Yes: a few dozen of your past cases and a written pass mark | Only a demo on the agency's sample data |
| 3 | Which provider and region process our data, for how long, and is it used for training? | The provider, the region, the retention period and the DPA, by name | "Everything is secure," with no names |
| 4 | When does the agent hand over to a person, and what does that person see? | Written rules, plus the conversation and the reason for the handoff | "The AI handles everything" |
| 5 | How do you defend against prompt injection, and what can the agent access? | Least privilege, approval for risky actions, tests with hostile inputs | Admin access "to keep it simple" |
| 6 | What does the agent log, and can we read the log? | Every action and tool call, alerts on failures, a log you can open | No log, or one only the agency can see |
| 7 | What will it cost to run each month, and who watches that cost? | An estimate at your volume, with usage alerts | Only the build price |
| 8 | Whose name are the accounts in, and what do we get at handover? | Your accounts from day one; workflows, prompts, code and the test set handed over | It all runs in the agency's accounts |
| 9 | Who answers when it breaks, and what happens when the model changes? | Response times in the contract and a test re-run after every change | "We'll fix it if something comes up" |
| 10 | Who will build it, and can we see one of your systems in production? | Named people, and a live system or a walkthrough of one | Slides and mockups only |
| 11 | Does this need an agent, a fixed workflow or a chatbot? | A reasoned choice, often a mix of the three | An "agent" for every task |
| 12 | What would you not automate here? | The steps that stay with people, and why | "Everything can be automated" |
How to run a paid pilot on your own data
A pilot is a short, paid build on one slice of the process, scored against your own past cases with a pass mark agreed before work starts. It answers the one question a demo cannot: does the agent work on your data, with your exceptions?
- Pick one slice: one channel, one type of request or one team, not the whole department.
- Build the test set: a few dozen past requests with the right outcome. In our projects, the agent has to pass it before it meets a real customer.
- Write the pass mark and the target before the build: the share of test cases handled correctly, the share handed to a person, the time to reply. A result defined afterwards can always be called a success.
- Set the limits: what the agent may do alone and what needs a person's approval.
- Close with a review: test results, failures, the measured cost per conversation and a go or no-go for the full build.
Your own cases matter more than any benchmark. NIST's Generative AI Profile warns that current testing approaches "often remain focused on laboratory conditions or restricted to benchmark test datasets" that may not carry over to real-world use (NIST AI 600-1, 2024). The pilot is also where you check value before you scale: Gartner recommends that agentic AI "only be pursued where it delivers clear value or ROI."
Where does your data go? What to ask under US and UAE rules
Ask for one page that names the model provider, the region where your data is processed, how long prompts and logs are kept, whether the provider may train on them and the data processing agreement (DPA) that covers it. Check it against the rules where you and your customers are. We agree this page before the build and keep card numbers, ID numbers and similar fields out of prompts and logs.
The UAE. The Personal Data Protection Law (PDPL), Federal Decree-Law No. 45 of 2021, came into force on 2 January 2022. It applies to processing of personal data "inside or outside the country" and sets rules for transfers abroad (UAE Government portal). So a provider abroad that processes your customers' data is covered too, and the contract should say on what basis the data leaves the country. Free zones such as the DIFC have their own data protection laws, and a federal law on information technology in health care adds rules for health data. Our guide to WhatsApp AI agents in the UAE goes through the PDPL articles that matter for a customer-facing agent.
The US. Rules depend on the state and the sector. California's CCPA, amended by the CPRA, covers for-profit businesses that do business in California and meet one of three tests, such as gross annual revenue over $25 million, which California's privacy agency raised to $26,625,000 from 1 January 2025, or buying, selling or sharing the personal information of 100,000 or more California residents or households (California Attorney General). Consumers can ask a business to delete their data and "tell their service providers to do the same," so ask how the agency would remove one customer from the logs, the agent's memory and the knowledge base.
If the agent touches patient data for a health plan or a health care provider covered by HIPAA, HHS treats the agency as a business associate: a business associate agreement (BAA) has to be in place before the data is shared, and the agency's subcontractors, cloud providers included, need one too (HHS).
When should an AI agent hand a conversation to a person?
When a request is outside its job, sensitive or stuck on a failed step. A good handoff passes the whole conversation and the reason for it, so the person can act at once and the customer does not repeat the story. Ask the agency for four things in writing:
- The triggers: the cases above, plus any customer who asks for a person.
- The context: the conversation, what the agent tried and why it stopped.
- The place: the chat, CRM or helpdesk your team already watches, not a new dashboard.
- The people: who is on duty, and what happens after hours.
McKinsey lists human-in-the-loop design among the practices that sustain AI deployment: deciding up front "how and when model outputs need a human in the loop."
Our Instagram agent, built for a client under NDA, runs 100+ Instagram accounts from one Telegram chat. When a run breaks or a DM needs judgment, a person gets a Telegram alert with the step, the account and the reason, so there are zero silent failures.
Instagram AI AgentClient under NDA
How should an agency secure an AI agent?
By limiting what the agent can do, not only what it is told. Ask how it resists instructions hidden in the content it reads, which systems and actions it may touch and where every action is logged. The OWASP Top 10 for LLM Applications 2025 is a good shared checklist, and prompt injection is item one.
Prompt injection is input that changes the model's behavior in ways nobody intended. The indirect kind is the one buyers miss: it occurs "when an LLM accepts input from external sources, such as websites or files" (OWASP LLM01), and for a support agent that means every customer email and uploaded PDF.
OWASP itself says "it is unclear if there are fool-proof methods of prevention for prompt injection," so the defense is to limit the damage. Its guidance on excessive agency comes down to four rules you can check in a proposal:
- Least privilege. The agent gets only the tools and permissions its job needs. A key that creates deals in your CRM should not be able to delete contacts or export the database.
- Approval for high-impact actions. Refunds, mass messages and changes to prices or contracts wait for a person.
- Checks outside the model. In OWASP's words, "implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not."
- Logs and limits. Every tool call is logged and monitored, with rate limits, in a log you can read.
Then ask how the agency attacks its own agent before launch: OWASP also recommends adversarial testing and attack simulations.
What happens after launch: running costs, ownership and maintenance
After launch you pay monthly for model usage, hosting and the people who take handoffs, and someone has to keep quality steady as models change. Before you sign, ask for a running estimate, the accounts the system will run in and written support terms.
Running costs. Market ranges for building and running an agent are in what an AI agent costs to build and run. Here, ask what it will cost per conversation or per month at your volume and how the agency will warn you when usage jumps. About 20% of McKinsey's 2026 respondents say AI-related operating costs, including token costs, constrained their AI use.
Ownership. Accounts belong in your company's name from day one: the model provider, the automation platform, the database and the channels. At handover you get the workflows, prompts and code, the test set, the documentation and an access list; our one-page brief template shows how to state that before anyone quotes. For lock-in, whose account the system runs in matters more than whether it runs on n8n or in custom code; n8n, Make or Zapier vs a custom AI agent covers when each pays off.
Maintenance. Models change after launch, and the contract should say who answers for that. NIST's Generative AI Profile suggests vendor contracts assign "liability and responsibility for incidents, GAI system changes over time (e.g., fine-tuning, drift, decay)" and include service level agreements that "address incident response, response times, and availability of critical support." In practice: a named person on support, response times in writing and a test-set re-run after every change to the model, the prompt or the tools.
Who will build it, and can you see a system in production?
Ask for the names of the people who will build and support your system, and to see one of their systems in production, live or in a recorded walkthrough. A system in daily use is the strongest proof an agency can offer.
Production is rarer than pitches suggest. In McKinsey's 2026 survey, 40% of respondents at organizations with annual revenue above $1 billion report scaling AI agents; at smaller organizations the share stayed flat at 22%.
An NDA does not stop an agency from showing its work. Ask to see the workflow, the logs and the alerts of a running system with the client's data hidden. Then ask who writes the prompts and the integrations, who tests, who reads the alerts and how much of each person's week your project gets. In our projects, a partner leads each one and brings in the specialists it needs.
Red flags when hiring an AI automation agency, including agent washing
Walk away if you see any of these five:
- The system runs in the agency's accounts, and you get access instead of ownership.
- No test on your data before the big contract.
- No answer on data: no provider, no region, no retention period.
- Every task is an "agent", and nobody can say what it decides.
- No logs and no alerts, so you hear about failures from your customers.
The fourth has a name: agent washing.
"Many vendors are contributing to the hype by engaging in 'agent washing' – the rebranding of existing products, such as AI assistants, robotic process automation (RPA) and chatbots, without substantial agentic capabilities." — Gartner, press release, June 25, 2025
Gartner estimates that only about 130 of the thousands of agentic AI vendors are real. That does not make an agent the only honest answer. Gartner analyst Anushree Verma advises starting "by using AI agents when decisions are needed, automation for routine workflows and assistants for simple retrieval."
Honest labeling looks like this. The content factory we built for a media publisher under NDA turns new videos into published articles, none of them hand-written. It runs on 40+ n8n nodes, and only two are AI agents: one picks the angle and title, the other writes. The rest are fixed steps on a schedule, from pulling transcripts to a review before anything goes live.
AI Content FactoryClient under NDA
An agency that clears these and answers the 12 questions with names and numbers is worth a pilot. Our AI automation services page shows how we map a process, test it on real cases and launch it with handoffs to your team.
Questions we get asked
How long should a pilot with an AI automation agency take?
Scope it by the test, not the calendar: one channel or one type of request, a few dozen past cases with the right outcome and a pass mark agreed before the build. Most of the time goes to integrations and tests rather than the AI itself. For Encore, a Dubai agency, the CRM that its AI call notes feed went from the first commit to production in 9 days. Ask for the end date in writing, together with the scope.
What is agent washing?
Agent washing is Gartner’s term for rebranding existing products, such as AI assistants, robotic process automation (RPA) and chatbots, as AI agents without substantial agentic capabilities. In June 2025 Gartner estimated that only about 130 of the thousands of agentic AI vendors are real. To check a proposal, ask what the system decides on its own and what it does when unsure. If it only follows fixed steps, it is a workflow, which may still be the right tool.
What should a contract with an AI automation agency include?
The process and the number it should move, the pilot scope and pass mark, the model provider and region, how long data and logs are kept and how they are deleted, handoff rules, the actions the agent may take without approval, whose name the accounts are in and what is handed over, and support terms: response times, re-testing after model updates and how incidents are reported. Have a lawyer in your state or emirate check it.
Do we need a business associate agreement for an AI agent in healthcare?
In the US, yes, if you are a HIPAA covered entity, such as a health plan or a health care provider that transmits health information electronically in covered transactions, and the agency handles protected health information for you. HHS treats a vendor that creates, receives, maintains or transmits that data on your behalf as a business associate, and its subcontractors, cloud providers included, as business associates too. Ask which providers in the agency’s stack will sign a business associate agreement.
How do we test an AI agent before it talks to customers?
Turn a few dozen past requests with the right outcome into a test set and agree a pass mark. The agent has to pass it before it meets a real customer, and the set is re-run after every change to the prompt, the model or the tools. Add hostile inputs, such as an email with hidden instructions, because OWASP ranks prompt injection first among risks for LLM applications. Check that every failed run alerts a person.
What happens to our AI agent if we part ways with the agency?
If the accounts are in your name, it keeps running, and a new team can take it over. Ask for a handover list in the contract: the workflows, prompts and code, the test set, the documentation, an access list and a call with your next team. If the agent runs in the agency’s accounts, leaving turns into a negotiation over your own system, which is why ownership belongs in the first conversation.
Sources
- Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 · Gartner, 2025
- The state of AI in 2026: On the road to ROI · McKinsey & Company, 2026
- OWASP Top 10 for LLM Applications 2025 · OWASP, 2025
- LLM01:2025 Prompt Injection · OWASP, 2025
- LLM06:2025 Excessive Agency · OWASP, 2025
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) · NIST, 2024
- Data protection laws · The Official Platform of the UAE Government, 2026
- California Consumer Privacy Act (CCPA) · State of California, Office of the Attorney General, 2026
- Updated Monetary Thresholds in CCPA · California Privacy Protection Agency, 2024
- Business Associates · U.S. Department of Health and Human Services, 2026
Share this article
About the author
Artem Zelenskii
Head of AI
Leads AI and ML: 5+ years in production. Apple Developer Academy graduate.
