Check Your Readiness | The Exam-Ready Toolkit

0 of 25 answered

Section 1 of 5

1. Inventory and classification

Can you name every AI system you run, including the ones inside platforms you rent, and say how risky each is? This is where an exam starts.

20% of the score · Exhibit A; Exhibit C ref 1 to 7

1.1You keep a written inventory of every AI system in production, including vendor-embedded models and features inside platforms you rent.

Stand up an ai-system.yaml register in the repo, one entry per system, and add a CI check that fails on any model call with no entry. Fix: Control 1

YesPartlyNo

1.2Each system is tagged to an Exhibit A operational area (underwriting, claims, fraud, and the rest).

Add an operational_area field to every register entry, using the Supplement's fourteen areas. Fix: Control 1

YesPartlyNo

1.3Each system carries an autonomy level: support, augment, or automate.

Record autonomy per system in the register, using the Supplement's three definitions. Fix: Control 1

YesPartlyNo

1.4Each system carries a risk classification with a written rationale.

Add risk_class and a one-line risk_rationale to every entry; your definition of high risk is itself evidence. Fix: Control 1

YesPartlyNo

1.5You can produce version, implementation date, and a count of systems implemented in the past 12 months on demand.

Record version and implemented date per entry so the register answers the past-12-months count as a query. Fix: Control 1

YesPartlyNo

Section 2 of 5

2. Governance and accountability

Is there a written, owned AI program that someone actually reviews, not a PDF filed once for a carrier questionnaire and never reopened?

20% of the score · Exhibit B narrative Q1, Q5; checklist 1 to 3

2.1A written AI program exists, dated, with a review cadence.

Adopt a dated AI program with an explicit review cadence; scale the ceremony to your size, keep the substance. checklist 1

YesPartlyNo

2.2A named role owns the AI program.

Name the owning role, not a person, and record it on each register entry's owner field. Fix: Control 1

YesPartlyNo

2.3The program addresses checklist items 3a through 3n, at least by reference.

Map the program to each checklist item 3a to 3n; a reference is enough, silence is not. checklist 3a

YesPartlyNo

2.4AI risk is a step in your software delivery process, not a separate document.

Fold AI risk into the release checklist and SDLC so delivery itself produces exam evidence. Fix: Control 8

YesPartlyNo

2.5The program has been reviewed at least once since adoption, and the review is documented.

Run and minute one review of the program; an unreviewed policy PDF scores nothing. checklist 1

YesPartlyNo

Section 3 of 5

3. Validation and observability

The heaviest pillar. Are your models tested before launch and watched in production, with a last-test date you can produce on demand?

25% of the score · Exhibit B Q3; Exhibit C ref 8, 9

3.1Pre-deployment validation is documented per model, with the reference data source stated.

Commit a dated validation report per model from a CI evaluation suite, stating internal, external, or both. Fix: Control 6

YesPartlyNo

3.2Each high-risk model is monitored in production with drift detection and a threshold that triggers review.

Build production metrics from the decision log and set a drift threshold that opens a review ticket. Fix: Control 6

YesPartlyNo

3.3Every automate-level decision is logged with inputs, output, model version, and timestamp, so any single outcome can be reconstructed.

Write an append-only decision record on the same code path that returns the decision. Fix: Control 4

YesPartlyNo

3.4Vendor-supplied models have a validation procedure that does not rely on the vendor's attestation alone.

Run your own evaluation suite against the vendor model; treat attestation as an input, not a substitute. Fix: Control 6

YesPartlyNo

3.5You can produce a last-test date for every high-risk model without asking an engineer to go look.

Have the scheduled validation job write last_test into the register so the field is always true. Fix: Control 6

YesPartlyNo

Section 4 of 5

4. Data provenance

Can you trace every data element feeding a model to its source and vendor, and justify any sensitive inputs?

20% of the score · Exhibit D; checklist 3d

4.1Every data element feeding a model is mapped to an Exhibit D category.

Tag each column to an Exhibit D category in the schema documentation. Fix: Control 7

YesPartlyNo

4.2Source is recorded per data element: internal, or third-party with the vendor named.

Record source per element; vendor name is a required Exhibit D field. Fix: Control 7

YesPartlyNo

4.3Training and test data lineage is documented for internally developed models.

Document the lineage of training and test data for the models you build. Fix: Control 7

YesPartlyNo

4.4Sensitive categories are absent from model inputs, or their use has a written justification and a proxy-discrimination check.

Enumerate sensitive features; remove them, or add a written justification and a proxy check in the validation suite. Fix: Control 7

YesPartlyNo

4.5The data map is versioned and updated when a model or a vendor changes.

Version the data element map and update it on any model or vendor change. Fix: Control 7

YesPartlyNo

Section 5 of 5

5. Consumer outcome controls

When AI touches an adverse decision, is there a human path, a complaint trail, and the disclosures your states require?

15% of the score · checklist 3a, 3c, 3m, 3n; Exhibit C ref 12, 13

5.1A human review path exists for adverse consumer decisions, with written triggers.

Route adverse decisions (declination, non-renewal, adverse rating, claim denial) to human review with written triggers. Fix: Control 5

YesPartlyNo

5.2Complaints are tagged to the AI system involved, and the tag is queryable.

Write the decision_id onto the complaint record so complaints are queryable by system. Fix: Control 4

YesPartlyNo

5.3Consumer-facing disclosure of AI use exists where the state requires it.

Check the state tracker for disclosure duties and add disclosures where required. checklist 3n

YesPartlyNo

5.4Each consumer-impacting model has a documented compliance review against unfair trade practice and claims settlement rules.

Add a compliance and bias section to each model's validation report. Fix: Control 6

YesPartlyNo

5.5A regulatory-action log exists per model, even if empty.

Add a regulatory-action field to the register; empty is a valid and expected value. Fix: Control 1

YesPartlyNo

·/100

Your readiness

Answer the sections to see your score.

Clear all 25 answers?

By pillar

Inventory and classificationScore · · Weight 20%→

Governance and accountabilityScore · · Weight 20%→

Validation and observabilityScore · · Weight 25%→

Data provenanceScore · · Weight 20%→

Consumer outcome controlsScore · · Weight 15%→

What to fix

Book a 30-day readiness review Read the full methodology

Keep your results

A short, tailored memo from your answers: where you stand, the two or three gaps to close first, and what "exam-ready" looks like for your book. Built from your answers only, orientation not legal advice.

Email me this summary and occasional Nolte updates on AI governance in insurance. Unsubscribe any time. How we use this.

Self-assessment and orientation only. Nothing here is legal advice, a certification, or a compliance determination. Nolte is not affiliated with or endorsed by the NAIC. Exhibit references are paraphrased from Supplement draft 4.0; the wording that counts is the NAIC's. Corrections: j@nolte.io.