TLDR
We launched and ran our app āAI Keysā for over a month. This article covers the entire story.
Start by identifying the key metrics of success to make sure you arenāt wondering in the dark.
Understand the difference between issues with growth and retention.
Look at the metrics to make changes to the product and strategy accordingly.
Analyze the results that the changes made on the metrics and decide to either continue re-iterating the product/strategy or give up.
Well it happened - weāve successfully failed again. For those who donāt remember, my brother and I were creating a keyboard for AI image generation that could be used in any messenger app that allows copy-pasting. You can read more about it here.
And I donāt mean it in a sense of āfailure = successā which was popularized in the startup community and became a buzzphrase, but rather in a sense of āquick failure = harmless failure = successā. No harm no foul as they say.
I take pride in creating content that goes straight to the point and has direct applications for my readerās work. So instead of talking about my past (and future) failures and their importance, Iāll just straight to digesting what happened and how you, too, can fail successfully. Of course I want you to succeed but failure is frequently a part of it.
1. Identify key metrics that will measure your success.š
Without any metrics to look at, you are just wondering in the dark doing random things and hoping that something ~feels~ like an indicator of success. Make sure that you know what parameters you need to keep an eye on to measure the success of your venture. Our key metrics were:
Retention rates - how many users continue keeping the keyboard 3, 7 and 30 days after downloading. The most important metric for ANY product (even if itās not SaaS) - if retention rates are low, you are doomed.
Conversion rates. At various times we were providing 3 and 10 free credits that users can use before having to pay a $1 fee for the next 20 credits. For this metric we were simply looking at the number of people who ended up buying credits, divided by the total number of downloads. This allowed us to calculate the future financial viability of the venture.
Number of downloads on Appstore. Your retention and conversion rates can be at a 100% if only you and your parents use the product so to have a good sample size we were looking to onboard at least 100 real users.
2. Analyze the metrics and adapt.šÆ
Iām a strong believer in doing things fast and then adapting on the fly so when it comes to the analytics of metrics in ideas that youāre trying to convert into a real company, I say - go with your gut feeling. But donāt set up hard expectations for these metrics right off the bat because they might be god-awfully off and quite frankly depressing. This in turn, wonāt allow you to adapt to the realities and change your product to the best. Letās say you see a 7-day retention rate of 3% and you think to yourself āWow 3 people out of 100 are still using my app 7 days later, thatās awesome but certainly need to improve!ā itāll definitely help you more in the long run rather than saying āWell the benchmark for my industry is 15%, Iām done hereā.
In our case, weāve set up tracking for retention rate and activity but had 0 expectations of the numbers we wanted to see. And here are the numbers that we got after about a month of running:
If you donāt want to trust your gut feeling, just use industry standards as your benchmark. With chatGPT being the genius it is, simply write out what your company does and then add āwhat is the average 7 day user retention rate for startups like mine?ā and youāll have your benchmark.
And on the side note, chatGPT is absolutely amazing for brainstorming so if youāre stuck - just go there and talk to the AI and chances are, itāll have an answer for you or at least help you find an answer.
Alternatively speaking, if you have developed an app as your final product simply go to your App Store Connect app analytics and click on āBenchmarksā:
3. Repeat the cycle or give up.š¦¾/ā ļø
Itās important to know how to change the product/growth strategy based on the metrics that you use to track your success.
- Low retention rates = look into the actions that users are taking in your app/website (we used Mixpanel).
- Not enough people are even looking at your product = change your growth strategy, you tried emailing, then try running ads, if you tried running ads then try cold-calling etc. When it comes to onboarding new users the number of strategies is basically unlimited so usually it comes down to product-related metrics that make or break the initial stages of a startup.
After looking at the numbers from App Store Connect, Mixpanel user stories and most importantly - after having a number of calls with our users, weāve identified three key issues:
Users were confused about setting up this AI keyboard and a lot of them dropped off straight after downloading the app without ever installing the keyboard plugin.
Those who started using the keyboard were struggling with the image generation. The AI frequently created designs that werenāt a good fit for the user and that forced user to change the prompt in order to create a new image.
It took too much effort to create the desired image. Creating a good image = creating a good prompt. When inspecting some users on Mixpanel we saw people who with ease completed the tutorial, opened the keyboard in their messengers and generated a couple of images - but never copied any of them.
Here are the solutions that we came up with to address those issues and increase retention rate:
Step-by-step interactive guide explaining how to use the keyboard.
Weāve provided users with a few prompts and results of the said prompts as examples of how to āspeakā to AI. Weāve also allowed users to practice image-generation right within the app without having to add the keyboard through their settings.
Havenāt addressed this issue since it takes ~30 seconds to come up with and type the prompt.
Here is an example of how we used Mixpanel to analyze user actions (start from the bottom of the screenshot to the top) -
This type of in-depth analysis of user actions can be extraordinarily insightful and life-changing for a company. I HIGHLY recommend to start paying very close attention to how customers use your product and not focus on generalist metrics shown in the first screenshot of this article.
The āprocess of giving upā in our case was very simple - we saw very little improvement in metrics and knew that it would take A LOT of effort to create some real improvement, but what really got us to give up was the realization that we ourselves, the core team, werenāt using the productā¦ It was too much effort to generate a custom image for a conversation and quite frequently it wasnāt even a good one since the AI drawings arenāt ideal. So you had to type a variety of texts to generate one image that you will eventually like and share in the conversation. By the time you share it, the conversation would already change. With the metrics listed above in mind and with the knowledge that even we as founders donāt actively use the product, we wrapped up the project and moved on with our lives.
4. Wrapping up.š«”
Total losses: ~$190 and a total of a few days of our free time. Check the numbers of downloads and usage of our tool and judge the outcome for yourself but if you ask me, Iāll say: āCanāt complainā.
Essentially, the summary of this article would be - keep your head up and try new things but know when to stop and move on.
Apologies for not sharing every step of this journey while it was going on. Iāll touch onto that and MANY other things in my next post.
And as always, have a good day and donāt forget to let go of ideas easily if they donāt stick;)