Answer
Do you train on our data?
Customer documents and conversations are never used to train models.1
See real questions answered, with the page each answer came from.
See the case studies[ Case studies ]
Two case studies, run through the same engine every Orrin account uses. Every answer, score and preview on this page is exactly what that engine produced from the help pages when the page was built. Nothing in the results is typed in by hand.
[ How to read a run ]
Orrin’s own help pages are real, and the assistant answers from them on every page of this site.
[ Case study 1 ]
The assistant in the corner of every page on this site reads Orrin’s own pages and articles. It runs on the same engine as a customer’s account, and its small AI model that understands meaning runs on Orrin’s own server, so no text goes to an outside AI service.
Run 1
Answer
Do you train on our data?
Customer documents and conversations are never used to train models.1
What Orrin read · 6 parts of pages
Run 2
Answer
How long does a test token last?
The token lives ten minutes in the code shown on the Developers page, and never more than fifteen.1
What Orrin read · 6 parts of pages
Run 3
Not mentioned
Does Orrin integrate with Salesforce?
The documentation does not mention Salesforce. What it does say: It answers your users’ questions from your own help pages, shows the source of each answer, and fills in tasks for them to check.2
What Orrin read · 6 parts of pages
12/20 passed as deployed8/20 with the small AI model off
OD3What happens on the Free plan when the conversation limit is reached? Quoted the sentence on prices for extra conversations, then “It includes 200 conversations a month”, instead of the sentence saying what happens at the limit: on the Free plan, the assistant stops starting new conversations until the next month. The passage it cited first does contain it. It passed before the help pages were rewritten in plainer words.OD6How quickly are affected workspaces told after a confirmed incident? Refused as not covered. The question says affected workspaces and a confirmed incident; the help pages say “If we confirm a problem that affects your account, we tell you within seventy two hours.” No sentence was close enough in meaning. It passed before the help pages were rewritten in plainer words.OD7Which file types can I upload as documentation? Refused as not covered. The help pages do list the file types, “Markdown, HTML, text, JSON and CSV”, but no part holding that list was close enough to the question to count. It passed before the help pages were rewritten in plainer words.OP1If someone keeps asking more questions in the same chat, does each one count towards my monthly total? Refused as not covered. The help pages do say “Follow up questions in the same thread do not start a new conversation.” The question says chat where the page says thread, and shares few other words with it.OP3What does the assistant tell a user when my help center has nothing on the topic? Quoted a sentence about the assistant answering only from your documentation, and one about user permissions, not the one saying what it replies when nothing covers the question. The cited passage does not contain the expected words, “documentation does not cover”.OP4How do I keep the assistant away from an API operation I have not approved? Quoted “The assistant can only match a request to an operation you switched on.” That is a correct answer in substance. It is graded as failed because the grading looks for the documentation’s own words, “until you switch it on”.ON5Is there a discount if I pay for a year up front? Refused as not covered. The documentation does say that yearly billing gives two months free, but a discount for paying a year up front is worded far from that sentence. This item was first written expecting “not mentioned”, before the pricing page said yearly billing gives two months free; its expected answer was corrected to what the documentation now says, which turned a pass for the wrong reason into a miss.OM1What does the Free plan include, and what happens when it runs out? Quoted only that the Free plan costs nothing, then said the documentation does not say what happens when it runs out. It does: on the Free plan, the assistant stops starting new conversations until the next month. Neither part’s fact was in the answer.[ Case study 2 · Brickwren ]
We wrote Brickwren’s help center and API to test Orrin. Every answer, score and task shown here is what Orrin’s engine produced. Brickwren decides which tasks the assistant may fill in, and each one carries a level: Read (looks something up), Write (changes one record) or Sensitive (money, messages, or many records at once).
Run 4
Answer
How do I waive a late fee for a tenant?
To waive a late fee, open the tenant's ledger, select the fee and choose Waive fee.1 You need the Waive fees permission to do this.1
What Orrin read · 6 parts of pages
Run 5
Answer
Can I remove a late charge from a tenant's account, and what permission do I need?
To waive a late fee, open the tenant's ledger, select the fee and choose Waive fee.1 You need the Waive fees permission to do this.1
What Orrin read · 6 parts of pages
Run 6
Not mentioned
Can tenants pay rent with Bitcoin?
The documentation does not mention Bitcoin. What it does say: Tenants can pay rent in the portal by bank transfer or by debit or credit card.1
What Orrin read · 6 parts of pages
Run 7
Preview · Write
Schedule a move-out inspection for unit 4B on Friday at 10am
scheduleInspectionWritePOST /inspections · Schedule an inspection
unit_id 4Btype move-outdate FridayTasks compared
scheduleInspection · POST /inspectionsPreparedcreateWorkOrder · POST /work-ordersNot chosengetRentRoll · GET /reports/rent-rollUnder 0.38Run 8
Preview · Write
Create a work order for the broken heater in unit 3B
createWorkOrderWritePOST /work-orders · Create a work order
unit_id 3Bdescription the broken heatercategoryTasks compared
createWorkOrder · POST /work-ordersPreparedupdateWorkOrder · PATCH /work-orders/{work_order_id}Under 0.38listWorkOrders · GET /work-ordersUnder 0.38Run 9
Not prepared · switched off
Refund the deposit for unit 2C
The closest operation, refundDeposit, is switched off, so nothing was prepared and nothing was run. It can be switched on in Actions.
Tasks compared
getRentRoll · GET /reports/rent-rollUnder 0.38sendRentReminder · POST /remindersUnder 0.38getTenantLedger · GET /tenants/{tenant_id}/ledgerUnder 0.3825/27 passed as deployed22/27 with the small AI model off
dP3When am I reminded that a tenancy is about to end? Quoted a sentence about how many tenants will get a message. The cited passage does not contain the 90 days the documentation gives.dP5How many kinds of staff access are there? Refused as not covered. The documentation lists five roles, in words unlike the question.[ Setting up, recorded · internal exercise ]
A fresh Orrin account, recorded from start to finish: a help article uploaded, the API uploaded, tasks switched on, a signing key and roles, the settings for adding the assistant to a page and a test token, an answer with its source, a preview and a CSV download. Then three small tasks, done by hand and with Orrin, counted the same way.
| Step | Actions | Characters typed | Script time |
|---|---|---|---|
| Upload a document | 5 (3 clicks, 1 field, 1 file) | 23 | 3.3 s |
| Import the API | 8 (7 clicks, 0 fields, 1 file) | 0 | 3.9 s |
| Signing key and roles | 3 (3 clicks, 0 fields) | 0 | 2 s |
| Embed and test token | 5 (2 clicks, 3 fields) | 21, plus 139 pasted | 4.3 s |
| Ask a question | 3 (2 clicks, 1 field) | 69 | 9.3 s |
| Prepare an action | 3 (2 clicks, 1 field) | 60 | 8.4 s |
| Export the conversations | 1 (1 click, 0 fields) | 0 | 0.4 s |
| All seven steps | 28 | 173 | 31.6 s |
| Task | Path | Clicks | Characters typed | Fields filled in by hand | Pages opened |
|---|---|---|---|---|---|
| A: schedule a move-out inspection for unit 4B on Friday at 10am | By hand | 4 | 21 | 3 | 1 |
| With Orrin | 5 | 65 | 2 | 1 | |
| B: create a work order for a broken heater in unit 3B | By hand | 4 | 22 | 3 | 1 |
| With Orrin | 5 | 59 | 2 | 1 | |
| C: find out who may waive a late fee and how | By hand | 1 | 0 | 0 | 2 |
| With Orrin | 3 | 34 | 1 | 1 |
[ Measured ]
Four sets were written by the Orrin team, so they are checks, not benchmarks. The fresh set was written and checked by AI agents on 3 unfamiliar companies and locked before the engine ran on it. Every item, answer and miss is published on the test results page.
| Set | As deployed | Small AI model off |
|---|---|---|
| Fresh set: 61 items on 3 unfamiliar companies, locked before the engine saw it | 32/61 | 32/61 |
| Held out: two companies the engine was never tuned on | 45/48 | 39/48 |
| Review set, written after an outside review | 23/26 | 21/26 |
| Orrin’s own help pages | 12/20 | 8/20 |
| Brickwren practice set, used to choose the score cut offs | 25/27 | 22/27 |