Start with the use case, not the model name

NIST's AI RMF is voluntary and risk-based. It organises work around Govern, Map, Measure and Manage rather than declaring a model universally safe. The practical consequence for procurement is that the same product can carry very different risks when it drafts public marketing copy, reads confidential employee files or triggers an operational action.

Write one use case in plain language: intended users, data, output, decision affected, human reviewer, prohibited uses and the consequence of a wrong answer. A vendor comparison without this boundary rewards broad claims that cannot be tested.

Request artefacts, not reassuring adjectives

The NIST Generative AI Profile identifies risks that may be novel or amplified by generative systems and suggests actions across the lifecycle. It is not a certification scheme. A vendor saying it is 'NIST aligned' should therefore map its claim to specific practices and evidence.

Ask for dated documentation: system and data-flow boundaries, evaluation design and results, known limits, access controls, retention and deletion behaviour, change logs, monitoring, incident handling, subcontractors and an exit path. Where disclosure is restricted, ask what can be demonstrated in a controlled review and who independently assessed it.

  • Scope: exact product, version, region and use case covered.
  • Evidence: report, test output, configuration or contractual control.
  • Owner: person accountable for the control and the exception path.
  • Freshness: date, planned re-test and material-change trigger.

Run a proof task before price decides the winner

Give shortlisted vendors the same representative tasks, including ordinary inputs, difficult edge cases and a deliberately unsafe request. Record accuracy criteria before testing. Measure refusal quality, citation or traceability where relevant, reviewer effort, latency, cost and the effect of configuration changes.

Do not average away a critical failure. A product may score well overall but still fail a mandatory privacy, security, accessibility or operational requirement. Mark those gates separately from weighted preferences.

Treat the scorecard as a decision record

Keep the question, vendor answer, evidence link, reviewer, score, exception and follow-up in one record. This makes disagreement inspectable and creates a baseline for renewal. It also prevents a later product update from inheriting an approval based on evidence for an older version.

This guide is general procurement education. NIST guidance is voluntary; applicable law, regulation, sector rules and contractual duties must be evaluated for the actual organisation and jurisdiction.