My hypotheses sources
- DataI used the results of previous experiments. I also analysed, segmented data, looked for deviations or trends in user behaviour. I analysed the points of decline in conversion and ways that lead to targeted actions.
- FeedbackI read what users write to us in the AppStore, GooglePlay and on the website. I observed the messages coming to the support service and the novice engagement system.
- AppsI improved my well-watching skills and explored activation tricks every day, especially for companies from Top Grossing. I observed their experiments in detail to understand which practices work or do not work for others.
- Analysis of user psychologyMore than half of the paying users are novice musicians. I talked to music teachers and explored services that help beginners take their first steps. I communicated with potential users of our service to identify their goals, motives and tasks that they would like to solve through Ultimate Guitar
- Decomposition and combinatoricsI looked closely at the current design, defined the current logical connections and how they affected perception. After that, I proposed hypotheses based on compositional changes in accents on the screens.
- FunctionalVerification of potential features that are difficult to develop. If there was a faith in them but there was no certainty that they were worth doing, the team and I tested the user's interest in them by creating fake-doors.
Main user motivations
UG will help me to learn guitar
I have a musical instrument and I want to learn how to play it
UG has the song I'm looking for
I want to learn how to play the song "Wish you were here" using simple chords and sound similar to the original song
In UG I will definitely find the original chords and tabs
I am a master of the instrument and I want to find the original chords or tabs
From ideation to creation
Generation of ideas
I think about, analyse, and explore the information from research.
In my experience, the most interesting and successful experiments are when you can find evidence of the same idea from two different sources.
As a result, I make a description which consists of: Which part of the product is the change? What is before? What is after? How can the user get to the screen? What are we going to change? What are we going to leave? What will be the text? Why will this work?
And I make a quick design that shows the main idea.
In my experience, the most interesting and successful experiments are when you can find evidence of the same idea from two different sources.
As a result, I make a description which consists of: Which part of the product is the change? What is before? What is after? How can the user get to the screen? What are we going to change? What are we going to leave? What will be the text? Why will this work?
And I make a quick design that shows the main idea.
Discussion and score of hypotheses
A team process of hypothesis discussion which selects the strongest hypotheses from the ICE scoring system.
Hypotheses are discussed for logicality before they are scored. The description of the hypothesis must contain all the answers to the questions from the previous part step.
Hypotheses are discussed for logicality before they are scored. The description of the hypothesis must contain all the answers to the questions from the previous part step.
Hypotheses design
Before starting the hypothesis, I create a document that will explain my thought process. The idea is supposed to be clear to anyone who has never heard of it before.
The description includes:
The results of the research and the sources on which the hypothesis is based. Arguments why the hypothesis should work and why it might not.
At this step, in the process of deeper analysis, the hypothesis is sometimes rethought and denied.
The description includes:
The results of the research and the sources on which the hypothesis is based. Arguments why the hypothesis should work and why it might not.
At this step, in the process of deeper analysis, the hypothesis is sometimes rethought and denied.
A/B testing
It is important to plan the experiment correctly. So I wrote down what metrics to observe and what results were expected, and the analyst helped me with that. And he also calculated how many users we would need for the experiment and how many days it would work.
Summarizing
After the launch, the team and I followed the experiment. Sometimes additional data and details of the results were required, so I set the task for the analyst.
If the hypothesis achieved good a result, either a decision was made to release it, or another iteration was launched if there were ideas on how to increase the metric.
If the hypothesis had a bad result, we either retested the experiment or planned the next iteration or closed it.
Then I wrote down the conclusions and shared the results with the company. In it, I described the suggestions why the hypothesis worked or why it did not, and what can be improved next time.
If the hypothesis achieved good a result, either a decision was made to release it, or another iteration was launched if there were ideas on how to increase the metric.
If the hypothesis had a bad result, we either retested the experiment or planned the next iteration or closed it.
Then I wrote down the conclusions and shared the results with the company. In it, I described the suggestions why the hypothesis worked or why it did not, and what can be improved next time.
As a result
34% growth ARPU
70+ A/B experiments
70+ A/B experiments
