Invigil

Certification

A score from Invigil, certified by someone who is not Invigil.

A benchmark that only its author vouches for is a claim. Every Invigil holdout run is executed under the Antefacts protocol and carries an Antefacts certificate that the run was conducted exactly as it was registered in advance. That is what turns the score into evidence.

The protocol

What happens to every run

beforeThe problem set for the quarter, the scoring rule and the model list are written down, hashed and independently timestamped before any model is run. The hash is public before the results are.
duringThe harness that prompts, assembles, compiles and audits is the published harness at a named commit. Every step enters an append-only, hash-chained record.
afterA machine-checked certificate states that the published scores are the output of the published harness applied to the recorded transcripts, against the problem set whose hash was committed in advance. Chain roots are published to locations Antefacts does not control.
alwaysAny customer may rerun its own transcripts against the pinned toolchain. Any deviation from the pre-registration is recorded and published, including where results are confidential.

The protocol and its published harness are described on antefacts.com. Invigil is the first licensed instrument to run on it.

Disclosure

Invigil and Antefacts share an owner

Invigil and Antefacts are separate companies with separate management, infrastructure, customer contracts and data, owned by the same parent. Invigil licenses the protocol from Antefacts on arm's length terms and pays a royalty on holdout revenue. We state this plainly because a reader of any certificate is entitled to know it, and because the alternative, that a customer or a regulator discovers it, is the failure both companies exist to prevent.

What follows from it, in the protocol rather than in policy:

  • No held-out data, problems, credentials or infrastructure are shared between the companies. The Invigil problem library is never used as Antefacts test material and Antefacts items are never used by Invigil.
  • Every Antefacts certificate on an Invigil run carries a published relationship field stating the common ownership.
  • Where Antefacts is asked by a certification body or a procurement buyer to evaluate a system that is itself an Invigil customer, Antefacts publishes the methodology and produces no score. Invigil's customer list is disclosed to Antefacts for that purpose alone.
  • The challenge owner for Antefacts evaluations sits outside the group.

The Antefacts side of this disclosure is published at antefacts.com/licensees.

What we do not claim

A pass rate, not a verdict on the model

Invigil measures one thing: whether a model produced a machine-checkable proof of each problem it was set. We do not say a model is safe, aligned, generally intelligent or fit for a purpose. The certificate says the measurement was conducted as registered. What the number means for a deployment decision is for the customer, the board or the regulator to judge.