Just Made Testing Faster, Cheaper, and Easier! - Scenario Testing Simulator
Much of SimulAite’s functionality revolves around the completion, generation, and marking of exam questions, as well as the unique state of each student’s abilities, configured specifically from all their activities on the app. As new features were implemented, it became really hard to thoroughly test them because of how many variations of scenarios could be created by users, with each depending on multiple previous actions taken. This was especially difficult because each action would take a long time to be processed by the LLM, and some question's timings could last even 45 minutes.
To test these scenarios, we often relied on directly manipulating the database to simulate events. However, with the introduction of the new concept of the Guided Improvement Plan, even that became infeasible due to the sheer scale of variations and the depth of history I would have to create for each scenario.
To conquer this problem, I implemented a new feature for admins to essentially create custom users with custom progress history, custom onboarding settings, and mission progress states with just a few clicks. Then, they can click “impersonate” to view exactly what a user with that exact configuration would experience.
Essentially, this UI quickly allows the admin to enter the exact scenario they want to test, and then the app fills in the data that would correspond with that scenario almost instantly. One advantage I thought of while building this was that the pre-filled data does not need to be LLM-generated. It can simply use template questions and answers, which means that many LLM calls do not have to be wastefully made just to set up testing.
The admin is then able to press “launch” to impersonate the user. This logs the admin into the new scenario user, in the exact situation described, and the Admin can view and impersonate this actual user. Admins can also create multiple different users, and each user can be used for several testing purposes, with the initial destination page being configurable.
To further aid testing, I implemented a timing control system that appears on attempt pages for scenario users. This allows admins to fast-forward time, jump forwards and backwards to different points, and test time-based behaviour without ever having to wait for real time to pass.
This is honestly one of the most interesting features I have implemented on SimulAite recently. It has been really fun making this impersonation system, thinking about the different user flows, and considering how each flow would impact what pieces of data would and would not be present in the database. Ensuring proper authentication for this impersonation feature has also been quite the journey.
I am really happy about this feature, as it is quite frankly going to revolutionise the speed and ease of testing features in the app, as well as save on LLM calls. Who would have known that such an exciting adventure would come from a feature that customers will never even see?
