A managing partner of Bitfold Online Games, Mike Turner knows his way around the design and development of mobile and social games. He also plays the role of analyst at times. He looks into how system performance data matters for game analytics in this article.
The social and mobile gaming revolution in 2009 transformed the usage of analytics in gaming from something mainly used in MMO games to a universal best practice for all games. In subsequent years, the analytical methods adopted to analyze and affect user behavior have become very advanced. Today’s game analytics focus on:
Data mining: Executing specialized queries into behavioral data to get a granular picture of user actions and preferences.
User segmentation: Segmenting users into meaningful cohorts and maximizing positive behavior of each player segment.
Predictive modeling: Using statistical models to predict which changes will increase retention, strengthen engagement, and maximize paid conversions and user spending.
Custom KPIs: Establishing actionable performance indicators custom to each game (e.g., paid user conversion at specific key events, churn rate after particular engagement goals).
Successful developers today are experts at employing these advanced methods to gain deep insight into what their users like and providing content and features that keep them engaged.
Game companies don’t often pay enough attention to how the system performance affects their numbers. Skilled usage of behavioral analytics enables developers to create targeted content and features which keep their users engaged. However, mishandled system errors can cause significant disruption to user experience and properly managing them is a weak point of many developers. Technical errors such as bugs, unexpected downtime, and latency happen a lot, and they are typically not noticed until they manifest in a game’s KPIs.
Optimally though, developers should be detecting and fixing system errors before they damage a game’s numbers. This article will focus on how to combine behavioral analytics with system performance analytics to avoid the negative effects of technical errors and maximize a game’s retention, engagement, and monetization numbers.
System Errors Create Noise in Behavioral Analytics Data
Once a system error begins, it’s often not noticed until it shows up as a negative trend in behavioral analytics. Sometimes, it is obvious that the negative trend is due to a technical error. But sometimes, it’s unclear whether it’s a system issue, an unbalanced gameplay feature, or content that users don’t like. To determine which possibility it is, analysts generally have to mine deeply into their data to determine the affected users and conditions correlated with the negative trend.
An example of this is portrayed below:
If it is determined that the negative trend has been caused by a technical issue, it means two things:
- The issue has been allowed to persist long enough to damage the game’s KPIs and has cost the developer money.
- Data about which individual features and content users prefer have been obfuscated by technical error, making it more difficult to establish clear conclusions on what changes should be made to the game to better engage users.
Diagnosing game error with behavioral analytics tools is not desirable. A better approach is to use log management tools specifically designed to monitor system health and error logs to diagnose and fix errors.
Below we examine how to implement log management alongside behavioral analytics to ensure that behavioral data collected is representative of a (mostly) bug-free gameplay experience, and that technical errors are eliminated before they can cause disruptions in gameplay experience or game revenues.
Using Log Management Tools to Detect System Errors Before Users Find Them
To find errors early, you want to use tools that provide you real-time system health data and help you detect and fix technical errors within a short timeframe (optimally less than a day). Most of this data can be surfaced by logging your application and the clients and servers it is installed on. Implementing logging is very similar to how you would implement behavioral analytics. You have to decide what data is important to track, but avoid inundating yourself with too much garbage data.
For system data, you want to log:
● Information about the performance and health (CPU usage, memory allocation, etc.) of the servers your game exists on
● Information about your server-side application’s performance and behavior
● Information about your database performance
● Information about your client’s behavior, the state of the device it’s being used on, client-side network conditions, and code that interacts with your server application. If your game is client-only, you should still log crashes, exceptions, and select information on the application’s behavior to help you quickly determine what’s causing bugs in your application, should they arise.
However, once you log this data, it’s unwise to simply dump it to file to be searched later. To work with this data deftly, you need a place that gives you tools for properly monitoring and analyzing it.
Centralizing Logs and Isolating Issues with Log Management Platforms
Sending logs to log management providers like Loggly provides developers with mature tools to monitor log data in real-time and quickly analyze it to find the causes of technical issues. These tools include automatic log centralization and organization, search tools that enable you to search log data with custom parameters, visualization tools which represent log data in a variety of chart types that help developers identify errors and performance trends, and system monitoring and alert tools that monitor logs for abnormal behavior and send your team email, SMS, or PagerDuty alerts when there are errors or deviations from optimal system health.
When combining these tools, you provide the capability for your live operations team to know of errors immediately via alerts and provide them tools to search logs quickly for the root cause. This allows many errors to be solved BEFORE they’re able to effect your game’s behavioral analytics. Even if some issues do escape early detection, and show up in your gameplay metrics, developers will be able to diagnose and fix them much faster with good logging tools than without.
How to Pair Log Data Management with Behavioral Analytics to Provide Better Gameplay Experiences for End-Users
Let’s re-examine the game-update example above assuming the availability of log management tools from programs such as Loggly. After the update, the live operations team notices retention suffering. With Loggly in place, the team can cross-reference both their behavioral data and log data to quickly determine if the cause is due to a technical error or to user dissatisfaction with the update’s content. If the cause is determined to be a system error, developers can use Loggly to diagnose and fix the error quickly.
In the case above, if the issue in retention was not due to system errors or bugs, the analysts would be able to use Loggly to rule out the possibility of system errors quickly and focus on re-balancing the gameplay.
Best Practices for Combining Logging and Behavioral Data
Be Predictive
You can use your behavioral analytics to tell which KPIs and user cohorts are most heavily affected by error. Once you have this data, identify all of the areas of system performance that can affect your KPIs. Send all logs surrounding those areas to your log management program, set up monitoring of those logs, and create alerts to monitor deviations from optimal performance.
For instance, our behavioral analytics might tell us that NON-hardcore gaming cohorts don’t accept much difficulty in the game or long waits at the loading screen. Therefore, we set up a saved search for SLA violations related to loading in Loggly, and set it to monitor these SLAs and alert Dev Ops teams when they happen.
Cross-Validate Issues
If an issue arises and it’s not clear whether it’s a system error or behavioral error, both pieces of data can be checked simultaneously to quickly rule out possibilities of system errors.
Use Logging to Check System State Before and After Release
At most successful game companies, releases happen almost daily. Use logging to check the system logs before and after releases to validate each release’s integrity. This will save the Live Ops team from having to spend time diagnosing and fixing technical issues down the line and prevent users from being affected by any bugs in a release.
Use Logging to Monitor Third-Party Vendor Integrations
Third-party integrations often have their own performance issues. For instance, ad impressions shown in your game will often take the form of interstitial video and rich media ads. However, if players are in areas of low connectivity or have slower devices, these ads can fail and interrupt the game experience. With logging, you’ll be able to tell when and how errors such as these are happening so that you can make corrections to your third-party integration and avoid future interruptions.
Conclusion
In game companies, most people on a Live Operations team have access to the game’s behavioral metrics and are always thinking about how their work affects them. By integrating a centralized log management into this process, the whole team will be encouraged to take the system’s health into account when analyzing user behavior and pushing new content and features. System issues have a significant and often hidden effect on your game’s key numbers. Proactively monitoring and fixing them frees up developer time and allows you to keep the money and users those issues would otherwise cost you.
Comments
