See real questions answered, with the page each answer came from.

See the case studies
Start free

[ Case studies ]

Real questions,
real answers, sources shown

Two case studies, run through the same engine every Orrin account uses. Every answer, score and preview on this page is exactly what that engine produced from the help pages when the page was built. Nothing in the results is typed in by hand.

[ How to read a run ]

Each run, from
question to answer

  1. 01What went inA question or a task, exactly as a user would type it into the assistant, like how to waive a late fee.
  2. 02What Orrin readEvery part of the help pages it found, with its meaning score, or the tasks it compared.
  3. 03What came backA quoted answer with its source, a plain “not covered”, or a task filled in for you to check.
  • QuotedThe answer’s sentence came from this part
  • PassedClose enough to the question, so it went forward
  • Set asideNot close enough, so not used
  • Not neededPast the four parts that go forward

Orrin’s own help pages are real, and the assistant answers from them on every page of this site.

[ Case study 1 ]

Orrin answers
questions about Orrin

The assistant in the corner of every page on this site reads Orrin’s own pages and articles. It runs on the same engine as a customer’s account, and its small AI model that understands meaning runs on Orrin’s own server, so no text goes to an outside AI service.

14documents: pages, articles and developer notes
105parts of pages, each one looked up by meaning and by keywords
0.40meaning score a part needs to count as evidence

Run 1

Answer

Do you train on our data?

Customer documents and conversations are never used to train models.1

Orrin’s own help pages · 228 ms when this page was built

What Orrin read · 6 parts of pages

The number is each part’s meaning score. A part needs 0.40 to count as evidence, and at most four go forward.

  1. 0.69Pricing and plans › Do you train on our data?Quoted [1]
  2. 0.67Questions software vendors ask › Do you train on our data?Passed
  3. 0.39Security › No training on customer dataSet aside
  4. 0.33Security › No training on customer dataSet aside
  5. 0.38Pricing and plans › Who owns the data?Set aside
  6. 0.23Security › Encryption while data travels and while it is storedSet aside

Run 2

Answer

How long does a test token last?

The token lives ten minutes in the code shown on the Developers page, and never more than fifteen.1

Orrin’s own help pages · 451 ms when this page was built

What Orrin read · 6 parts of pages

The number is each part’s meaning score. A part needs 0.40 to count as evidence, and at most four go forward.

  1. 0.33The Orrin console and its five parts › Identity: every user with their own tokenSet aside
  2. 0.29What to prepare before adding an assistant to property software › A way to say who the user isSet aside
  3. 0.41Developers: tokens and the embedded assistant › Step 1: sign tokens on your backendQuoted [1]
  4. 0.30Questions software vendors ask › How do we give the assistant a token for each signed in user?Set aside
  5. 0.18Pricing and plans › The Free planSet aside
  6. 0.26How the AI works › Tested in the openSet aside

Run 3

Not mentioned

Does Orrin integrate with Salesforce?

The documentation does not mention Salesforce. What it does say: It answers your users’ questions from your own help pages, shows the source of each answer, and fills in tasks for them to check.2

Orrin’s own help pages · 546 ms when this page was built

What Orrin read · 6 parts of pages

The number is each part’s meaning score. A part needs 0.40 to count as evidence, and at most four go forward.

  1. 0.61Questions software vendors ask › Does Orrin run actions in our system?Passed
  2. 0.58What Orrin is and who it is for › What Orrin isQuoted [2]
  3. 0.56Developers: tokens and the embedded assistant › Help when you need itPassed
  4. 0.54The Orrin console and its five parts › OverviewPassed
  5. 0.51Questions software vendors ask › Which language model does Orrin use?Not needed
  6. 0.51Pricing and plans › Does Orrin execute actions?Not needed

Scored on 20 questions about Orrin

Questions the Orrin team wrote about Orrin’s own help pages, some worded like the pages and some not, with every miss below. A check, not a benchmark.

12/20 passed as deployed8/20 with the small AI model off

  • OD3What happens on the Free plan when the conversation limit is reached? Quoted the sentence on prices for extra conversations, then “It includes 200 conversations a month”, instead of the sentence saying what happens at the limit: on the Free plan, the assistant stops starting new conversations until the next month. The passage it cited first does contain it. It passed before the help pages were rewritten in plainer words.
  • OD6How quickly are affected workspaces told after a confirmed incident? Refused as not covered. The question says affected workspaces and a confirmed incident; the help pages say “If we confirm a problem that affects your account, we tell you within seventy two hours.” No sentence was close enough in meaning. It passed before the help pages were rewritten in plainer words.
  • OD7Which file types can I upload as documentation? Refused as not covered. The help pages do list the file types, “Markdown, HTML, text, JSON and CSV”, but no part holding that list was close enough to the question to count. It passed before the help pages were rewritten in plainer words.
  • OP1If someone keeps asking more questions in the same chat, does each one count towards my monthly total? Refused as not covered. The help pages do say “Follow up questions in the same thread do not start a new conversation.” The question says chat where the page says thread, and shares few other words with it.
  • OP3What does the assistant tell a user when my help center has nothing on the topic? Quoted a sentence about the assistant answering only from your documentation, and one about user permissions, not the one saying what it replies when nothing covers the question. The cited passage does not contain the expected words, “documentation does not cover”.
  • OP4How do I keep the assistant away from an API operation I have not approved? Quoted “The assistant can only match a request to an operation you switched on.” That is a correct answer in substance. It is graded as failed because the grading looks for the documentation’s own words, “until you switch it on”.
  • ON5Is there a discount if I pay for a year up front? Refused as not covered. The documentation does say that yearly billing gives two months free, but a discount for paying a year up front is worded far from that sentence. This item was first written expecting “not mentioned”, before the pricing page said yearly billing gives two months free; its expected answer was corrected to what the documentation now says, which turned a pass for the wrong reason into a miss.
  • OM1What does the Free plan include, and what happens when it runs out? Quoted only that the Free plan costs nothing, then said the documentation does not say what happens when it runs out. It does: on the Free plan, the assistant stops starting new conversations until the next month. Neither part’s fact was in the answer.

[ Case study 2 · Brickwren ]

A property software
help center and API

We wrote Brickwren’s help center and API to test Orrin. Every answer, score and task shown here is what Orrin’s engine produced. Brickwren decides which tasks the assistant may fill in, and each one carries a level: Read (looks something up), Write (changes one record) or Sensitive (money, messages, or many records at once).

14help center articles
50parts of pages Orrin made from them
14tasks in its API (the list of things its software can do)
10tasks switched on for the assistant

Run 4

Answer

How do I waive a late fee for a tenant?

To waive a late fee, open the tenant's ledger, select the fee and choose Waive fee.1 You need the Waive fees permission to do this.1

Brickwren help center · 110 ms when this page was built

What Orrin read · 6 parts of pages

The number is each part’s meaning score. A part needs 0.40 to count as evidence, and at most four go forward.

  1. 0.89Late fees › Waiving a late feeQuoted [1]
  2. 0.68Late fees › When a late fee is chargedPassed
  3. 0.68Late feesPassed
  4. 0.72Late fees › Setting up late feesPassed
  5. 0.43Online payments and autopay › Paying rent onlineNot needed
  6. 0.49Late fees › Undoing a waiverNot needed

Run 5

Answer

Can I remove a late charge from a tenant's account, and what permission do I need?

To waive a late fee, open the tenant's ledger, select the fee and choose Waive fee.1 You need the Waive fees permission to do this.1

2 of 2 parts answered. Part 1, “Can I remove a late charge from a tenant's account”: answered from [1]. Part 2, “what permission do I need”: answered from [1].

Brickwren help center · 285 ms when this page was built

What Orrin read · 6 parts of pages

The number is each part’s meaning score. A part needs 0.40 to count as evidence, and at most four go forward.

  1. 0.66Late fees › Waiving a late feeQuoted [1]
  2. 0.63Late fees › When a late fee is chargedPassed
  3. 0.56Late feesPassed
  4. 0.35Team roles and permissions › Changing permissionsSet aside
  5. 0.47Move a tenant out › The final accountPassed
  6. 0.49Security deposits › Refunding a depositNot needed

Run 6

Not mentioned

Can tenants pay rent with Bitcoin?

The documentation does not mention Bitcoin. What it does say: Tenants can pay rent in the portal by bank transfer or by debit or credit card.1

Brickwren help center · 284 ms when this page was built

What Orrin read · 6 parts of pages

The number is each part’s meaning score. A part needs 0.40 to count as evidence, and at most four go forward.

  1. 0.61Online payments and autopay › Paying rent onlineQuoted [1]
  2. 0.44Record a rent paymentPassed
  3. 0.47Security deposits › Collecting a depositPassed
  4. 0.32Online payments and autopay › AutopaySet aside
  5. 0.33Reports › Rent rollSet aside
  6. 0.38Rent reminders › Automatic remindersSet aside

Run 7

Preview · Write

Schedule a move-out inspection for unit 4B on Friday at 10am

scheduleInspectionWrite

POST /inspections · Schedule an inspection

Details filled in
  • unit_id 4B
  • type move-out
  • date Friday
Details still missing
None.
Details the user gave that no field can hold
  • 10am: this operation has no time field

Your product would show the user this preview to confirm before running it. A preview never calls your API.

Tasks compared

The switched on tasks closest to the request in meaning. A task needs 0.38 or more, and a clear lead over the next one, to be filled in.

  1. 0.68scheduleInspection · POST /inspectionsPrepared
  2. 0.38createWorkOrder · POST /work-ordersNot chosen
  3. 0.26getRentRoll · GET /reports/rent-rollUnder 0.38

Run 8

Preview · Write

Create a work order for the broken heater in unit 3B

createWorkOrderWrite

POST /work-orders · Create a work order

Details filled in
  • unit_id 3B
  • description the broken heater
Details still missing
  • category
Details the user gave that no field can hold
None.

Your product would show the user this preview to confirm before running it. A preview never calls your API.

Tasks compared

The switched on tasks closest to the request in meaning. A task needs 0.38 or more, and a clear lead over the next one, to be filled in.

  1. 0.53createWorkOrder · POST /work-ordersPrepared
  2. 0.37updateWorkOrder · PATCH /work-orders/{work_order_id}Under 0.38
  3. 0.33listWorkOrders · GET /work-ordersUnder 0.38

Run 9

Not prepared · switched off

Refund the deposit for unit 2C

The closest operation, refundDeposit, is switched off, so nothing was prepared and nothing was run. It can be switched on in Actions.

refundDeposit (POST /deposits/{deposit_id}/refunds, Sensitive) matched this request at 0.55, and the software company chose not to switch it on.

Tasks compared

The switched on tasks closest to the request in meaning. A task needs 0.38 or more, and a clear lead over the next one, to be filled in.

  1. 0.26getRentRoll · GET /reports/rent-rollUnder 0.38
  2. 0.25sendRentReminder · POST /remindersUnder 0.38
  3. 0.25getTenantLedger · GET /tenants/{tenant_id}/ledgerUnder 0.38

Scored on 27 questions and requests about Brickwren

The practice set, written by the Orrin team on Brickwren and used to choose the engine’s score cut offs, with every miss below.

25/27 passed as deployed22/27 with the small AI model off

  • dP3When am I reminded that a tenancy is about to end? Quoted a sentence about how many tenants will get a message. The cited passage does not contain the 90 days the documentation gives.
  • dP5How many kinds of staff access are there? Refused as not covered. The documentation lists five roles, in words unlike the question.

[ Setting up, recorded · internal exercise ]

From an empty account
to a preview and a download

A fresh Orrin account, recorded from start to finish: a help article uploaded, the API uploaded, tasks switched on, a signing key and roles, the settings for adding the assistant to a page and a test token, an answer with its source, a preview and a CSV download. Then three small tasks, done by hand and with Orrin, counted the same way.

Internal exercise by the Orrin team on a workspace loaded with Brickwren’s help center and API. Not a measurement from a company using Orrin. The private key and the test token are blurred.
StepActionsCharacters typedScript time
Upload a document5 (3 clicks, 1 field, 1 file)233.3 s
Import the API8 (7 clicks, 0 fields, 1 file)03.9 s
Signing key and roles3 (3 clicks, 0 fields)02 s
Embed and test token5 (2 clicks, 3 fields)21, plus 139 pasted4.3 s
Ask a question3 (2 clicks, 1 field)699.3 s
Prepare an action3 (2 clicks, 1 field)608.4 s
Export the conversations1 (1 click, 0 fields)00.4 s
All seven steps2817331.6 s

Script time leaves out the pauses added so the recording can be followed. Seconds are machine time of a Playwright script on a development server on one laptop, not the time a person would take.

The same jobs, by hand and with Orrin

TaskPathClicksCharacters typedFields filled in by handPages opened
A: schedule a move-out inspection for unit 4B on Friday at 10amBy hand42131
With Orrin56521
B: create a work order for a broken heater in unit 3BBy hand42231
With Orrin55921
C: find out who may waive a late fee and howBy hand1002
With Orrin33411
  • A: schedule a move-out inspection for unit 4B on Friday at 10am. Manual counts only the filling and submitting, with the form already open; the host app opens its forms only through its assistant, so a person would also have to find the form. With Orrin the sentence costs more keystrokes than the three short values do, because every value here is only a few characters. The preview captured the unit, type and day, and said the time of day has no field (10am), so it was added by hand. The preview wrote the type as move-out; the form's own list says move out.
  • B: create a work order for a broken heater in unit 3B. Manual counts only the filling and submitting, with the form open. With Orrin the sentence costs more keystrokes than the short values. The preview captured the unit and the description (the broken heater) and listed the required category as missing, so it was typed by hand. Orrin saved one field, not the typing.
  • C: find out who may waive a late fee and how. Manual is a help center built from the same fourteen Brickwren articles: the index page, then the Late fees article, where the answer sits near the top, so no scrolling was needed. A person must also pick the right article from fourteen titles and read to the right line, which these counts do not measure. With Orrin the question is typed and the answer quotes the sentence with its source. The count here is higher in clicks and keystrokes; what it saves is the search and the reading, which a script cannot time.

[ Measured ]

Every set,
model on and off

Four sets were written by the Orrin team, so they are checks, not benchmarks. The fresh set was written and checked by AI agents on 3 unfamiliar companies and locked before the engine ran on it. Every item, answer and miss is published on the test results page.

SetAs deployedSmall AI model off
Fresh set: 61 items on 3 unfamiliar companies, locked before the engine saw it32/6132/61
Held out: two companies the engine was never tuned on45/4839/48
Review set, written after an outside review23/2621/26
Orrin’s own help pages12/208/20
Brickwren practice set, used to choose the score cut offs25/2722/27