AI vendor risk assessment: a practical buyer’s checklist

An AI vendor risk assessment evaluates how a supplier’s AI uses your data, affects your systems and supports business decisions. It should connect each material risk to current evidence, a named reviewer and an approval condition. Assess the specific service and configuration you plan to buy, including any AI features added to an existing product.

This guide is for security, procurement and third-party risk teams reviewing an AI-enabled supplier. The 15 questions below are a practical starting point, not a certification test. Adapt them to your data, intended use and internal risk policy.

Start with the use case, not the questionnaire

Write a short intake statement before requesting documents: who will use the service, what information it can access, what outputs people will rely on and what actions it can take. A tool that drafts text for review needs a different assessment from an agent that updates customer records or changes production settings.

Separate three exposure routes: AI embedded in the product, AI the vendor uses to deliver its service, and AI-assisted development of the software you depend on. A supplier can create exposure through any of these routes even when the purchased product is not marketed as an AI tool.

NIST’s Generative AI Profile recommends adapting acquisition and vendor assessments to generative AI risks, including embedded technologies and ongoing monitoring. Its GOVERN 6.1 and 6.2 sections provide useful reference points for supplier oversight and contingency planning.

15 AI vendor assessment questions and evidence to request

1. Which AI features are in scope?

Ask the supplier to identify the exact product edition, enabled features, model providers and optional integrations. Request a dated service diagram and feature inventory. Record which features will remain disabled so approval cannot silently expand to a different use case.

2. What data reaches each provider?

Trace prompts, uploaded files, retrieved documents, outputs, logs and support access separately. Request a data-flow diagram that identifies storage locations and recipients. A statement that the application is hosted in one region does not answer where every downstream AI service processes information.

3. Can customer data be used for training?

Request the contractual position and the settings that apply to your purchased plan. Distinguish training, product improvement, abuse monitoring and temporary processing. Ask for evidence of the configured choice. Do not assume all AI services or account types use data the same way.

4. How long is information retained?

Ask about prompts, outputs, logs, backups and derived indexes individually. Request a retention schedule and an example deletion workflow. For a retrieval system, confirm how removal of a source document affects the stored index and future answers.

5. Which permissions does the integration require?

Request the proposed permission list before connection. Identify read, write, delete and administrative capabilities separately, including service-account access. Have the application owner explain why each permission is needed and how it can be revoked.

6. What can an AI agent do without approval?

List actions the system can execute and the approvals required for each. Request a demonstration of a blocked or escalated action. OWASP’s guidance on excessive agency highlights the risks of unnecessary functionality, excessive permissions and excessive autonomy.

7. How are unsafe instructions and untrusted inputs tested?

Ask for test results covering malicious instructions in documents, messages or other retrieved material. Request the test date, system configuration, observed failures and remediation status. A broad claim that the model is secure does not describe how your integration behaves.

8. How is output quality evaluated for your workflow?

Define representative tasks and unacceptable errors with the business owner. Request results for the relevant model and configuration, including limitations. For example, test whether a support assistant invents refund terms when an answer is missing from the approved knowledge base.

9. What can a human reviewer inspect?

Ask whether reviewers can see supporting sources, recognize uncertainty and override an output before it affects a customer or system. Request a workflow demonstration with an incorrect answer. Identify the team responsible for reviewing disputed outputs.

10. How is AI-assisted code reviewed?

Ask how generated code enters the vendor’s normal development process. Request a redacted release example showing review, dependency checks, security testing and exception handling. NIST’s Secure Software Development Framework provides a common vocabulary for discussing secure development with suppliers.

11. Who else supports the AI service?

Request the current list of relevant model, hosting and other service providers. Identify which party handles each control and support obligation. Record how the supplier communicates a provider change and who on your team assesses its effect.

12. What happens when the model or feature changes?

Request a sample change notice and release record. Agree which changes require reassessment, such as a new model provider, broader connector permissions or a feature moving from advice to execution. Keep the approved configuration with the assessment.

13. What evidence supports the security claims?

Review available assurance reports, testing summaries and remediation records for their scope and dates. Ask the vendor to map them to the purchased service. Log exclusions and unanswered questions instead of treating a report’s existence as proof that every AI feature was reviewed.

14. How will an incident be investigated?

Request the incident contact, notification process and available audit records. Walk through a hypothetical incorrect action or data disclosure. Establish who can suspend the feature and what evidence your team would need to understand the event.

15. Can you safely stop using the service?

Request an exit procedure covering access revocation, data export, deletion and business continuity. Test the most important step during the pilot. Identify a usable fallback if the AI feature becomes unavailable or no longer meets your requirements.

Turn answers into an approval decision

Use a simple assessment record with one row per material issue: question, vendor answer, evidence link, evidence date, reviewer, unresolved gap, required action and due date. Mark evidence as verified, incomplete or not supplied. Keep an unanswered question visible; it is not equivalent to a failed control or an accepted risk.

For example, a sales assistant might request permission to read contacts and send messages. If the pilot only needs contact summaries, the reviewer could require read-only access, restrict the pilot dataset and document a test proving that sending is blocked. Approval would apply to that configuration, with broader access triggering another review. This is an illustrative decision, not a universal policy.

Use your organization’s existing approval authority and risk criteria. Possible outcomes include approval for the defined scope, a restricted pilot with conditions, or deferral until evidence is supplied. Record the rationale and accountable owner rather than reducing every answer to one unqualified score.

When should an AI vendor be reassessed?

Set a review cadence based on the service’s importance, data sensitivity and ability to act. Also define event-based triggers: new integrations, permission changes, model substitutions, incidents, material retention changes or expansion into a new business process. Assign each trigger to an owner who can reopen the assessment.

The goal is an assessment that stays connected to the service in use. Findings’ AiVRM offering focuses on AI exposure across vendor code, infrastructure and productivity layers. Use the questions above to define the evidence and ongoing visibility your vendor-risk program needs.

Frequently asked questions

What is the difference between a questionnaire and an assessment?

A questionnaire collects supplier responses. An assessment evaluates those responses against the intended use, supporting evidence and the organization’s risk criteria, then records a decision and follow-up actions.

Does every vendor need the same AI review?

No. Start with a short intake to establish whether and how AI affects the service. Expand the review when the use case involves sensitive information, important decisions, broad system permissions or actions that are difficult to reverse.

What is the first document to request?

Ask for a current description of the AI feature and its data flow. This establishes the service, providers and information in scope, allowing you to request relevant evidence instead of a generic document bundle.