Just Made Testing Faster, Cheaper, and Easier! - Scenario Testing Simulator

Much of SimulAite depends on each student’s unique history: the questions they have completed, their abilities, onboarding choices and progress across the app.

But as we built more features, testing every possible situation became increasingly difficult. Creating just one realistic scenario could require completing several previous activities, waiting for LLM responses and sometimes sitting through question timings lasting up to 45 minutes.

We initially worked around this by manually changing data in the database. However, once we introduced Guided Improvement Plans - with even more dense, deep histories and configurations - it became clear that this approach was no longer going to work.

So, I built a Scenario Testing Simulator.

Admins can now create custom scenario users with specific onboarding settings, progress histories, mission states and starting pages in just a few clicks. The simulator instantly generates the corresponding data, after which the admin can impersonate the user and experience the app exactly as the user would.

While building it, I realised the simulated questions and answers did not need to be generated by an LLM at all. They could use reusable templates instead, making scenario users almost instantaneous to create while avoiding a large number of unnecessary LLM calls, making testing much cheaper!

There was still one frustrating problem: testing anything time-dependent  (e.g. features depending upon how long a student takes for a question) meant waiting for real time to pass.

So I added timing controls that allow admins to fast-forward, rewind or jump directly to different stages of an attempt. A 45-minute question can now be tested in seconds! The admin controls panel also gave me another idea: allowing admins to set scores and feedback directly. This has removed the need for Marking LLM calls and manually writing answers during testing.

Building the simulator involved thinking through a surprising number of user flows, determining exactly which database records each situation required and securely implementing the impersonation system. It has honestly been one of the most interesting features I have worked on recently.

Customers will probably never see it, but it will make developing everything they do see dramatically faster, cheaper and easier.

Who knew an internal testing tool could become such an exciting engineering adventure?