Understand the words. Make better decisions.
24 terms you will encounter in an AI project. Each includes an operational definition, a limitation and a concrete question to ask a supplier. Linked guides explain the method and examples.
By Stolen Orbit. Reviewed 25 September 2026.
Workflow
A defined sequence of steps and conditions that takes an input to an outcome. It can include an AI model without delegating the entire sequence to it.
Ask: Which steps are fixed and which depend on a person’s judgement?
AI agents or workflows →AI agent
A system in which a model can choose tools or steps within a defined scope. Acting requires permissions, limits and verifiable stopping conditions.
Ask: Which actions can it perform, under whose credentials, and when must it stop?
AI agents for business →LLM
A language model that generates or transforms text from its input and learned patterns. Fluent output does not establish correctness or access to business data.
Ask: How will we check results on our work, including incorrect answers?
AI agents or workflows →RAG
An approach that retrieves information from selected sources and provides it to the model when answering. Retrieval, permissions and the answer can still be wrong.
Ask: Does the answer show a current source that the recipient is allowed to access?
RAG or fine tuning →Fine-tuning
Additional training on examples to adapt a model’s behaviour. It does not automatically replace an up-to-date knowledge store or permission-aware retrieval.
Ask: Is the problem the model’s behaviour or missing information?
RAG or fine tuning →Context
The instructions, messages and documents available to a model for a particular run. A large context window does not mean every detail will be used correctly.
Ask: Which sources enter the context, and what happens when one is missing?
RAG or fine tuning →Tokens
The units a model uses to represent text. They do not always correspond to words; input, output and other provider-specific charges can affect cost.
Ask: Does the estimate specify volume, retries and consumption limits?
AI implementation costs →Structured output
A result with expected fields and types, such as product code and quantity. A valid format does not prove the values exist in the source or are correct.
Ask: Do we check both the schema and agreement with the document?
Document AI →Hallucination
A generated statement or value without adequate support in the available information. It may sound plausible and perfectly match the requested format.
Ask: Does the system leave a field empty and request review when the value is absent?
Evaluating AI reliability →Evaluation
Comparing outputs with predefined criteria on representative cases. Include ordinary cases, failures and conditions in which the system must not proceed.
Ask: Which cases failed, and who decided the expected result?
Evaluation lab →Expected result
The approved reference answer or action for a test case. If the case is ambiguous, the reference should acknowledge that rather than invent certainty.
Ask: Do two process owners agree on the expected outcome and review cases?
AI evaluation dataset →Human review
A real check before an action, with the information, time and authority to correct or stop it. An Approve button alone does not establish useful oversight.
Ask: Can the reviewer see the source, changes and consequences before approving?
Human review for AI →API
An interface through which software reads data or requests operations. Its existence does not establish that the project’s required fields and permissions are available.
Ask: Which operations are available on our plan and within what limits?
System integration map →Webhook
A notification sent between systems when an event occurs. Provider behaviour may include delays, repeated delivery or failed attempts.
Ask: How do we verify the notification and handle repeated or missing events?
Automation failures and duplicates →Idempotency
The property that repeating the same request does not multiply its intended effect. It matters when a missing response leaves the outcome uncertain.
Ask: Could a retry create another order or send another email?
Automation failures and duplicates →Reconciliation
Comparing what should have been recorded with what the destination system actually recorded. Successful transmission alone does not prove the final state is correct.
Ask: Can we trace each input to the record created in the ERP?
ERP integration reconciliation →Least privilege
Giving each component only the access required for its task. Reading, writing and approving may need different roles and credentials.
Ask: Can we revoke one capability without disabling the entire process?
AI permission matrix →Shadow mode
A trial alongside real work that produces comparable outputs without performing the process’s operational actions. The no-write boundary needs technical verification.
Ask: What prevents the trial from changing records or contacting customers?
Rolling out AI →Rollback and recovery
Returning to a previous configuration and recovering work affected by a failure. Rolling back software does not automatically undo a sent email or order.
Ask: How will we find and correct effects that have already occurred?
AI rollout plan →Monitoring
Checks on failures, queues, consumption and quality, with an owner for alerts. Technical availability does not establish that outputs remain useful.
Ask: Which signal triggers intervention, and who acts on it?
AI operations runbook →Data drift
A change in inputs or operating context that can degrade a previously useful system: new formats, products, instructions or request types.
Ask: Which evaluations do we repeat when the work changes?
AI change log →Released capacity
Potentially available time after subtracting review, corrections and operational work. Multiplying it by an hourly rate does not prove that expense has disappeared.
Ask: How will the hours be used, and which realised benefit will we measure?
How much time could automation release? →Total cost of ownership
Project cost over the chosen period: setup, software, consumption, internal work, maintenance and changes. It depends on the assumptions and comparison boundary.
Ask: Does the comparison use the same period, volume and support scope?
AI implementation costs →Managed service
An agreement for recurring operational work: checks, failure handling, updates and agreed support. A recurring fee does not imply unlimited work or unspecified response times.
Ask: What is included, what triggers action and what counts as a scope change?
Automation operations →References and method
These are editorial definitions for business buyers. The questions are Stolen Orbit’s suggestions; they do not replace a technical specification or establish a product’s reliability.
- Anthropic — Building effective agents
- Microsoft Learn — Retrieval augmented generation
- AWS — Idempotent APIs