Hypothesis testing in Statistics:

Reading :

• Hypothesis Testing

• Effect of sample

• Sample Distribution

Techniques:

• Bootstraping

• Multiple Comparison

  • Bonferroni Correction

    • Boole-Boneferroni

    • Close Testing procedure

    • Holm-Boneferroni

    • p-hacking

    • Šidák correction

  • Tukey’s Correction

  • Q value Correction

Common hypothesis tests include:

• Testing a population mean One sample t-test.

• Testing the difference in means Two sample t-test

• Testing the difference before and after some treatment on the same individual Paired t-test

• Testing a population proportion One sample z-test

• Testing the difference between population proportions Two sample z-test

You can use one of these sites to provide a t-table or z-table to support one of the above approaches:

• t-table

• t-table or z-table

Generalized Stats Models:

• Generalized

A/B Testing :

• Drawbacks :

  • Can help you compare two options, but it can’t tell you about an option you haven’t considered.

  • Can only compare can’t tell the increment in outcome.

  • Bias results on existing users :

    • Change Aversion: Existing users may give an unfair advantage to the old version, simply because they are unhappy with change, even if it’s ultimately for the better.

    • Novelty Effect: Existing users may give an unfair advantage to the new version, because they’re excited or drawn to the change, even if it isn’t any better in the long run.

• Difficulties in A/B Testing

  • Novelty effect and change aversion when existing users first experience a change

  • Sufficient traffic and conversions to have significant and repeatable results

  • Best metric choice for making the ultimate decision (eg. measuring revenue vs. clicks)

  • Long enough run time for the experiment to account for changes in behavior based on time of day/week or seasonal events.

  • Practical significance of a conversion rate (the cost of launching a new feature vs. the gain from the increase in conversion)

  • Consistency among test subjects in the control and experiment group (imbalance in the population represented in each group can lead to situations like Simpson’s Paradox)

Credits:

Udacity