Evaluation

How the engine
scores

Orrin’s engine was run against a fixed set of 48 questions and requests that it was never tuned on. Every result is below, failures included, with the configuration that produced it.

Results

Three runs,
one set

The same 48 items, graded automatically by the evaluation script, through the previous engine (word matching only, before the language model, parameter reading and section passages), today’s engine with the model switched off, and today’s engine as deployed.

Measure	Previous engine	Today, model off	Today, as deployed
Answerable, worded like the documentationThe answer contains the expected fact and the passage it cites contains it too.	10 of 10	9 of 10	10 of 10
Answerable, worded differentlyGraded the same way; the question avoids the documentation’s own words.	4 of 10	0 of 10	6 of 10
Not covered by the documentationPasses when Orrin says the documentation does not cover it.	4 of 9	9 of 9	9 of 9
Two documents disagreePasses when the answer includes the value from the newer document.	2 of 2	2 of 2	1 of 2
Action requestsPasses when the preview matches the intended operation.	8 of 8	8 of 8	8 of 8
Requests missing a required valuePasses when the right operation is matched and every missing required value is listed.	0 of 4	4 of 4	4 of 4
Requests no operation coversPasses when nothing is prepared.	3 of 5	3 of 5	4 of 5
Parameter values read correctlyAcross the action requests, each expected value the preview proposed. The previous engine proposed none.	0 of 10	10 of 10	10 of 10
Answers that should not have been givenLower is better: a wrong sentence quoted for an answerable question, or an answer to a question the documentation does not cover.	10	0	0
Answerable questions refusedLower is better: questions the documentation answers that Orrin said it could not.	1 of 20	11 of 20	4 of 20
Time per item, median and slowest in twentyOn the machine named below, after the model has loaded.	0.2 ms, 0.5 ms	0.2 ms, 0.9 ms	4.3 ms, 12.5 ms

In every run, each answer to the two disagreeing documents also quoted the outdated statement. The model finds more answers worded differently from the documentation and refuses what the documentation does not cover; the price is that 4 answerable questions, worded far from the documentation, were refused as not covered.

Configuration

What was
measured

Machine: Apple M1 Max, 64 GB, Node v25.6.1, not the production server.
Language model: all-MiniLM-L6-v2, an open source sentence embedding model of about 22 million parameters, quantized to 8 bits and run with ONNX Runtime inside the server process.
Thresholds: a passage counts as evidence at a meaning score of 0.40, a sentence is quoted at 0.55, at most 2 per answer, and an operation is matched at 0.38 with a lead of 0.05 over the next.
How they were chosen: on a separate development set of 27 items written against Brickwren, the case study’s example company. This set was not used for that.
Answers are quoted. No model writes text in these runs; answers are sentences from the documentation with their citations.

The set

What it
contains

Quillhaven, a made up property management platform: a help center of eight articles and an API of eight operations.
Kestrel Yard, a made up apartment building: a resident handbook, a newer notice that changes the recycling day, and an API of four operations.
Answerable, worded like the documentation: 10 items. The answer contains the expected fact and the passage it cites contains it too.
Answerable, worded differently: 10 items. Graded the same way; the question avoids the documentation’s own words.
Not covered by the documentation: 9 items. Passes when Orrin says the documentation does not cover it.
Two documents disagree: 2 items. Passes when the answer includes the value from the newer document.
Action requests: 8 items. Passes when the preview matches the intended operation.
Requests missing a required value: 4 items. Passes when the right operation is matched and every missing required value is listed.
Requests no operation covers: 5 items. Passes when nothing is prepared.

What failed

6 of 48 items
did not pass

P3 · Answerable, worded differently
How far ahead am I warned that a tenancy is ending?
The documentation does not cover this. In a workspace, the question is logged as unanswered so the gap is visible on the Usage page.
Refused as not covered. Three passages passed the evidence gate, but no single sentence was close enough in meaning to a question worded this far from the documentation.
P4 · Answerable, worded differently
What happens if the renter ignores the renewal?
The documentation does not cover this. In a workspace, the question is logged as unanswered so the gap is visible on the Usage page.
Refused as not covered, for the same reason: related passages were found, but no sentence came close enough.
P5 · Answerable, worded differently
Who gets alerted when an apartment has no heating?
The documentation does not cover this. In a workspace, the question is logged as unanswered so the gap is visible on the Usage page.
Refused as not covered. The closest passage matched the question’s meaning at 0.38, under the threshold of 0.40.
P8 · Answerable, worded differently
Where is a renter's deposit money kept?
The documentation does not cover this. In a workspace, the question is logged as unanswered so the gap is visible on the Usage page.
Refused as not covered. One passage passed the gate, but its sentences were not close enough to the question.
C1 · Two documents disagree
When is recycling collected?
Recycling is collected every Wednesday morning. [1]
Quoted only the handbook’s older collection day. Orrin does not compare the dates of documents, so an outdated page can win.
X4 · Requests no operation covers
Delete the building
deleteApplicant (DELETE /applicants/{applicant_id}), level Sensitive. No values found in the request. Still needed: applicant_id. The user would type a confirmation word before this runs. This release does not execute the operation, so nothing was sent to your API.
“Delete the building” was matched to deleteApplicant, the only delete operation in that API, instead of being refused. The preview would still ask for a typed confirmation and an applicant, and nothing runs in this release.

Limits

What this
does not show

The set is small, 48 items, so one item moves a result by several points. Treat the results as a check, not a benchmark.
It was written by the Orrin team, in English, not collected from real users, and both companies are made up.
No wrong answer in this set does not mean none at all. On the separate development set of 27 items, the deployed engine gave 3 answers it should not have: “When am I reminded that a tenancy is about to end?”, “When do landlords get their monthly report?”, “Can tenants pay rent in bitcoin?”. That is why every answer carries its citation.
Each configuration gave the same results every time it was run; the times vary with the machine and its load.
It grades quoted answers. Answers written by a model, which a workspace can switch on where a deployment names one, are not part of it.

Download every result Try the live demo

Every item

Each input
and its output

A1 How much is the default late fee?Passed

The default fee is 5 percent of the monthly rent, capped at 75 dollars. [1]

A2 How do I void a late fee?Passed

Open the tenant's ledger, select the fee and choose Void charge. [1]

A3 How many days before a lease ends does Quillhaven send a reminder?Passed

Quillhaven reminds the property manager 75 days before a lease ends. [1]

A4 How long does a tenant have to accept a renewal offer?Passed

The tenant has 10 days to accept the offer in the tenant app. [1]

A5 How many photos can a tenant attach to a maintenance request?Passed

Tenants submit maintenance requests in the tenant app and can attach up to six photos. [1]

A6 When are owner statements generated?Passed

Owner statements are generated on the third business day of each month and emailed to owners as a PDF. [1]

A7 How many days do I have to return a security deposit?Passed

When a tenant moves out, the deposit must be returned within 21 days, minus itemised deductions. [1]

A8 Which role can delete a property?Passed

Only Owners can delete a property. [1]

A9 What are the gym opening hours?Passed

The gym on the ground floor is open from 5 am to 11 pm. [1]

A10 Where is the bicycle storage?Passed

Bicycle storage is in the basement on level B2. [1] Each apartment may store up to two bicycles. [1]

P1 What's the penalty when a renter pays late?Passed

The default fee is 5 percent of the monthly rent, capped at 75 dollars. [1] Quillhaven adds a late fee automatically on the sixth day of the month when rent is still unpaid. [1]

P2 Can I cancel a late charge?Passed

Open the tenant's ledger, select the fee and choose Void charge. [1]

P3 How far ahead am I warned that a tenancy is ending?Failed