As in any other game, playtesting is a vital tool for us to assure the quality of WHALIEN. We test our levels regularly and in different development stages - from the first greyboxed concepts to the final colorful rooms. Due to the ongoing pandemic, performing playtests in person has been widely impossible for us. Moreover, conducting playtests online has proven to be far less time-consuming than organizing multiple playtesters and watching them meticulously. 

For this reason, we created the Playtesting Data Collection Assistant, a tool to help us conduct remote playtests. The PDCA creates an interactive recording of the users’ play session. Unlike a video, it allows us to observe the players’ actions and changes in the level from any viewpoint and observe the state of any actor at any time, while also requiring significantly less storage (a 20+ minute replay requires only <5MB).

PDCA:

On a more technical level, the PDCA is based on the Replay System provided by UE4. At the start of each recording, a virtual server is created, which the client sends all relevant data to. This works using the replication system in UE4 - all actors that we want to be represented accurately in the replay need to be replicated (e.g. if we want to track the open/closed state of a door, we need to declare the corresponding function as replicated and multicast, so that the replay server takes notice of it).

During the replay we let the replay-character perform all original input actions as they happen, reconstructing the gameplay of the playtester. 

Recording a replay functions fully automatically once the playtester has given their consent. They don’t notice the recording process while playing, which helps prevent the Hawthorne Effect: testers behaving differently because they know they are being watched.

The replay files are automatically sent to our server after the play session has concluded to spare the player some effort. The small file size of our replays means that we do not strain the player’s bandwidth too much.

Events:

During recording we also take note of the player’s actions in the form of events. We visualize those events along the timeline when viewing the replay. This provides us with a neat overview of the player’s activities in the play session and it also allows skipping to specific actions/events in the replay.

We decided that it would be best to view the replays in the UE4 editor itself instead of creating a standalone replay-player, because this allows us to use the built in user interface for inspecting relevant actors’ positions and states.

Heatmaps:

We can analyze recorded replays to generate offline “3D heatmaps”, that display the paths and actions of various players in a level to quickly determine if they used the intended routes or got stuck at a certain section. Furthermore we created a custom editor tool, which allows us to control which information of the heatmap we want to display.

Caveat:

What’s to be noted is that the PDCA doesn’t function out-of-the-box in a non-replicated game, such as WHALIEN. Therefore, replicating our game required some effort to get the system to work initially. In order to reduce this effort, we decided to only replicate the character movement (which is already provided by the UCharacterMovementComponent), input actions and events/objects that are relevant for the progression in a level. Due to the non-deterministic physics engine of UE4, we cannot guarantee the behavior of actors that are not replicated. Moreover, skipping forwards and backwards in the recording does not guarantee the state/position of all non-replicated actors in the level and can lead to inconsistencies in the replay.

Conclusion:

We’ve already applied the PDCA in multiple playtests, where it helped us gain insights on the players’ challenges while playing our levels, which in return helped us adapt and improve the levels accordingly.

If you have any more questions about the PDCA or our playtesting practices, do not hesitate to ask!

- Andi, Ralf