Skip to content

Q: How to get the bucket of combined_experiments ? #500

Closed

Description

@cmantas

When you, eg.

  Split.configuration.experiments = {
        :button_color_experiment => {
          :alternatives => ["blue", "green"],
          :combined_experiments => ["button_color_on_signup", "button_color_on_login"]
        }
      }

thing =  ab_combined_test(:button_color_experiment)

After you run ab_combined_test how can you know whether you are in the blue or green bucket (and decide what color to actually draw the button) ?
In this snippet, thing gets the value of ["button_color_on_signup", "button_color_on_login"] , which is actually the result of the each method here

Activity

  1. semanticart commented on Sep 19, 2017

    @semanticart
    Contributor

    It looks like this should be just a matter of returning alternative after this block:

    experiment[:combined_experiments].each do |combined_experiment|
    if alternative.nil?
    if control
    alternative = ab_test(combined_experiment, control, alternatives)
    else
    normalized_alternatives = Split::Configuration.new.normalize_alternatives(experiment[:alternatives])
    alternative = ab_test(combined_experiment, normalized_alternatives[0], *normalized_alternatives[1])
    end
    else
    ab_test(combined_experiment, [{alternative => 1}])
    end
    end

    That + a test is a pretty quick PR.

    Unfortunately from my usage, it doesn't look like ab_combined_test works properly. Using ab_combined_test locally will generate the first combined experiment in the combined_experiments array properly but will generate new versions of subsequent experiments for each participant.

    So in your example, you'd get a consistent test for button_color_on_signup but each participant in button_color_on_login would cause the version for that test to bump.

    It isn't immediately clear to me and it is certainly possible I'm doing something wrong but I don't see what that would be.

    Maybe @daviddening has some thoughts on the functionality since he originally wrote it?

  2. semanticart commented on Sep 19, 2017

    @semanticart
    Contributor

    Aha. the issue is line 18 above. Only the selected alternative is passed in (with weight of 1) but since not all alternatives are represented, it appears to be a new test and the version is bumped.

    I'll try to get a PR together to fix this.

  3. added a commit that references this issue on Sep 20, 2017
    d69048a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions