Surveys and A/B tests

Surveys

A survey asks one question or several, inside your app. Answers are kept one at a time, so a survey left half way still counts.

1. Press + New › Message and choose Survey.
2. On Content, add the questions. Each can be a rating, a choice of answers, or words the person types, and can be required or not. Add an intro and an end screen if you want.
3. Choose Who, When and a Goal as for any message (Your first in-app message). A survey is usually Service.
4. Press Try it on my test phone, then Go live.

A small survey opens in a sheet over the app; a large one takes the whole screen. Once someone has answered, the survey is not swiped away by accident: Back goes back a question, and leaving asks first.

The survey’s Report tab shows each question’s answers, and how many started and finished. Finishing a survey is recorded as an event on the person, so a segment or a workflow can follow up.

Ask at the right moment

Use When they do something to ask straight after the moment you care about, such as order_delivered. The Ask for feedback workflow template asks people who have opened the app ten times in a month.

A/B tests

An A/B test sends two or more versions of a message, each to a share of people, so you can see which works better. It works for in-app messages, pushes, emails and chats.

1. Open the message and, on Content, press A/B test and control group.
2. Add a version. Each version has its own words; give each a share in percent. The shares always add up to 100.
3. Optionally keep a control group who get nothing, to measure what the message itself does.
4. Choose how to pick a winner:
   • judge by the goal rate (for a message with a goal) or the click rate;
   • decide at the end of the test, or as soon as one is clearly ahead;
   • if none is clearly ahead by the end, keep the split or pick the best so far.

A test lasts from an hour to 30 days. The report says where it stands: “clearly ahead”, “ahead, not yet clearly enough to decide early”, or “not clear yet”. When a winner is picked, everyone who has not had the message yet gets the winner, and the decision says how sure it was, such as “Version B won: significant at 97% after 3 days”. You can also pick a winner yourself.

How sure is sure

A winner is called at 95% confidence at the end of the test. Called early, it needs 99.99%, because a test that is looked at every minute would otherwise find winners that are not there. A small difference is decided at the end.