← Back to Home
Ezra is the Bible study assistant inside FaithFeed. We say Ezra answers from Scripture it actually looks up, and declines rather than guesses. Before saying that publicly, we tried to break it.
How we tested it
We wrote 323 automated tests that attack Ezra on purpose. They use stand-in AI models scripted to misbehave: inventing verses, agreeing with wrong corrections, taking doctrinal sides, giving advice. This shows what Ezra's own code catches, rather than what we merely ask the AI to do.
Every failure was double-checked by two independent reviewers, with a third breaking ties.
What held up
- Crisis messages. When the safety screen flags a message, Ezra calls no AI model and shows a fixed list of crisis and emergency lines. It doesn't store what you wrote.
- Nothing relevant found. When the scripture search finds nothing close to the question, Ezra declines instead of answering from general knowledge.
- Search breaks. When the scripture search fails, Ezra declines instead of guessing, unless a verse was typed (see below).
- Wrong verses. When an answer names a verse, written the standard way, that Ezra didn't look up, the whole answer is thrown out.
- Annotations. New notes that threaten people are blocked, while writing about violent passages like Nahum or Judges 19 stays allowed.
What didn't
- If the safety screen itself is down, a crisis message currently goes on to the AI. Only the latest message was screened, not earlier ones.
- Typing a verse reference, or even starting with “Ezra,”, could get an off-topic question answered.
- Citations written in other ways (lowercase, abbreviations, verse ranges), verses you typed yourself, quoted wording, and Ezra's short opening line weren't checked.
- Nothing in the code stopped Ezra from taking a doctrinal side, agreeing with a wrong correction, or writing a devotional. Only its instructions asked it not to.
- Editing an annotation after posting skipped the threat screen.
What we're doing about it
We're fixing each of these in code, crisis handling first. We're also adding a second, separate AI check that withholds answers that go beyond the passages or take a side.
Then we'll rerun every test, plus a capped round against the real AI models, before we repeat any of these claims.
We haven't yet measured how the real AI models behave against these tests. That's the next round, and we'll publish what it shows here.