Revulytics Blog

How much effort should a startup invest into analytics and measurement?

April 18, 2013

Subscribe

Let's get one thing out of the way. Whether you have 10 users or 1 million users for your software product, there is definitely no arguing against the value of knowing HOW those users react to your product, what they think of it and whether they are willing to use it or pay for it.

Knowing such information can drastically change the way you build, market and sell your product, and essentially it can also define your roadmap to success (or failure). However how much of your limited time & resources should you steal out of your product to work on measurement, data collection & analytics?

Investing in data collection and analytics at an early stage (as soon as you deploy version 1.0) will give you the benefits of a feedback loop which can help you test your ground and stay on the right track without wasting time and resources building and marketing low ROI features. But how much effort you should put into building this feedback loop? The answer lies in how much resources you have available. Essentially you need to strike a balance between how much data you can handle, what reporting capabilities you have available and the ROI you expect.

Where do you draw the line?

Before you decide what data takes priority over the rest, it might be useful to read this blog post on: "Why measuring download statistics is useless" which outlines some core metrics you will need to consider. The diagram below shows how the various levels of analytics data collection relate to the overall expected ROI.

Analytics-ROI-chart

Relating to the diagram, here is an explanation of each of the points along the ROI curve:

  • A: I Know Nothing - when you ship the product with zero metrics and no data collection
  • B: Bare Essentials - this is where you collect the most basic information such as downloads, installs, uninstalls and conversions. It is the bare minimum required to gauge your basic product performance.
  • C: Advanced Analytics - this is where you get your hands on more advanced metrics such as detailed usage patterns and segmenting data in a way that allows you to understand how usage and conversions vary across the various user groups or product versions.
  • D: More data than required - this is where the amount of data you are collecting starts to get overwhelming and you do not have the capacity to consume it all. At this point, the extra effort and expense to collect the 'unused' data will not add any benefit, but rather it will start reducing your overall ROI.
  • E: Follow the Big Data Hype - this is where you follow your developer instinct and end up collecting everything that is technically possible, just in case, one day, someone dreams of something. This tends to increase your cost of data collection and storage and over time will potentially lead to your reporting becoming unncessarily slow, bulky or expensive. At this level if the data is big enough you might start hitting scalability issues.

Move the line as you grow

For a startup with limited resources and minimal product traction, minimal investment to collect the bare essentials might be enough to provide answers for the basic business decisions. It would be nice to select a framework that can provide you with an easy transition from (B) Bare Essentials to (C) Advanced Analytics without having to change your data collection framework or your reporting infrastructure.

However keep in mind that after a few months of deploying your product, you will start getting a clearer picture of what it is you want to know more about, and only then you will be in a position to define the core requirements of what your advanced analytics solution should look like and how much you will be ready to pay for those answers.

Do not waste too much time and effort worrying about advanced analytics before you actually deploy your first product, however, if you have the option, choose a solution that can grow with you.

Big Data: a need or a hype?

Just because everybody is talking about it, it does not mean you should do it. Once you reach  point (D) in the diagram, you risk falling into an unsafe spiral which can easily lead you to a situation whereby you are collecting way too much data that gets out of hand and you do not have the capacity and resources to consume most of this data, let alone use it to improve your product or business. Ignore the media hype and stick to data that is actionable. Anything you cannot consume is simply noise.

Build vs Buy

During the past year, over 50% of the companies I talked to have claimed that at some point in their product lifetime, they tried building their own data collection system to track product installations, usage or conversions.  Around 90% of these companies also claimed that product managers or decision makers within their company would very rarely get the answers they wanted from their home-built system, and this was mostly due to the limitations or lack of flexibility in the reporting framework, rather than the data being collected.

Therefore don't be fooled by wearing your developer's hat and looking at how easy it might seem to collect the data. It's useless dumping a bunch of data in a database unless you have a method to easily and efficiently extract DIGESTIBLE reports from the data. Having big data inside a database does not give you any answers. You need a usable and flexible reporting framework that can scale with your product. Building and maintaining such an infrastructure that is both flexible and easily usable by the non-geek in your company, can become a long-term nightmare for your developers. What you need to ask yourself is how easy would it be for a sales person or product manager to build new reports and extract ACTIONABLE answers from the system without involving the developers?  If it becomes too much hassle to extract the right reports, over time nobody will use it, and hence you will have lost all the development effort which could have been better used on your product.

If you consider investing in a custom-built call-home system, you need to keep in mind that your own physical or virtual server will need administration time, apart from the general running cost of the server. Hence why a hosted 3rd party solution might be a faster and cheaper to setup.

Break it in steps

In your startup days, you should find tools that help you get close to point (B) Bare Essentials without any risk and without too much effort. If budgets are tight you can find very good quality freeware or pay-per-use cloud services such as Google Analytics for your web analytics and Trackerbird for your software analytics that will only cost you a couple of hours of development time to implement, without any damage to your pocket. Once you have some real traction and are ready for more, then you can consider switching to premium analytics that will allow you to do cooler and more advanced stuff.

Conclusion

When resources are tight, you should still take some time to invest in the basic analytics to collect at least the bare essential KPIs. However don't lose focus on your product. Once you start getting product traction you can identify more advanced metrics that you can use to unlock the power of advanced analytics tools.

Get Started with Usage Analytics

Register a free account and start touring analytics immediately. Then, simply integrate the SDK into your app to start your free trial. Start making data-driven decisions.

Keith Fenech

Post written by Keith Fenech

Keith is Revulytics’ VP, Software Analytics and was the co-founder and CEO of Trackerbird Software Analytics before the company was acquired by Revulytics in 2016. Following the acquisition, Keith joined the Revulytics team and is now responsible for the strategic direction and growth of the Usage Analytics business within the company. Prior to founding Trackerbird, Keith held senior product roles at GFI Software where he was responsible for the product roadmap and revenue growth for various security products in the company's portfolio. Keith also brings with him 10 years of IT consultancy experience in the SMB space. Keith has a Masters in Computer Science from the University of Malta, specializing in high performance computing.