Recruitment Via Prolific¶

In this tutorial, you will learn how to set up a random dot motion (RDK) experiment, sample experimental conditions, collect data via the recruitment platform Prolific and use linear regression to analyse the data. We will use the following components:

• theorist: linear regression
• experiment-runner: serving the experiment via Firebase and recruiting participants via Prolific
• experimentalist: random sampling

Prerequisites¶

In this example, we assume that you have set up a RDK experiment on Firebase. For example, following the cookiecutter tutorial Here, we focus on how to integrate recruitment via Prolific.

!!! Warning: The firebase-prolific-runner will automatically set up a study on Prolific and recruit participants. It is highly recommended to test the experiment before recruiting participants, to have approval from an ethics committee, and to adhere to the ethics guidelines. For example include a consent page.

Workflow¶

In [ ]:

import numpy as np
import pandas as pd

import numpy as np import pandas as pd

Variables And State¶

We set up an Autora experiment by defining variables and a state. Here, we will use the coherence of the random dot motion kinematogram as the dependent variable. This is the ratio of dots moving in a coherent direction as opposed to moving randomly.

In [ ]:

from autora.state import StandardState, on_state, Delta
from autora.variable import VariableCollection, Variable

# *** Set up variables *** #
# independent variable is coherence in percent (0 - 100)
# dependent variable is accuracy (0 - 1)
variables = VariableCollection(
    independent_variables=[Variable(name="coherence", allowed_values=np.linspace(0, 1, 101))],
    dependent_variables=[Variable(name="accuracy", value_range=(0, 1))])

# *** State *** #
# With the variables, we can set up a state. The state object represents the state of our
# closed loop experiment.

state = StandardState(
    variables=variables,
)

from autora.state import StandardState, on_state, Delta from autora.variable import VariableCollection, Variable # *** Set up variables *** # # independent variable is coherence in percent (0 - 100) # dependent variable is accuracy (0 - 1) variables = VariableCollection( independent_variables=[Variable(name="coherence", allowed_values=np.linspace(0, 1, 101))], dependent_variables=[Variable(name="accuracy", value_range=(0, 1))]) # *** State *** # # With the variables, we can set up a state. The state object represents the state of our # closed loop experiment. state = StandardState( variables=variables, )

Theorist¶

We use linear regression as a theorist and wrapt the regressor in on_state functionality to use it on the state

In [ ]:

from sklearn.linear_model import LinearRegression
# ** Theorist ** #
theorist = LinearRegression()

# To use the theorist on the state object, we wrap it with the on_state functionality and return a
# Delta object.
# Note: The if the input arguments of the theorist_on_state function are state-fields like
# experiment_data, variables, ... , then using this function on a state object will automatically
# use those state fields.
# The output of these functions is always a Delta object. The keyword argument in this case, tells
# the state object witch field to update.


@on_state()
def theorist_on_state(experiment_data, variables):
    ivs = [iv.name for iv in variables.independent_variables]
    dvs = [dv.name for dv in variables.dependent_variables]
    x = experiment_data[ivs]
    y = experiment_data[dvs]
    return Delta(models=[theorist.fit(x, y)])

from sklearn.linear_model import LinearRegression # ** Theorist ** # theorist = LinearRegression() # To use the theorist on the state object, we wrap it with the on_state functionality and return a # Delta object. # Note: The if the input arguments of the theorist_on_state function are state-fields like # experiment_data, variables, ... , then using this function on a state object will automatically # use those state fields. # The output of these functions is always a Delta object. The keyword argument in this case, tells # the state object witch field to update. @on_state() def theorist_on_state(experiment_data, variables): ivs = [iv.name for iv in variables.independent_variables] dvs = [dv.name for dv in variables.dependent_variables] x = experiment_data[ivs] y = experiment_data[dvs] return Delta(models=[theorist.fit(x, y)])

Experimentalist¶

Here, we use a random pool and use the wrapper to create an on state function Note: The argument num_samples is not a state field. Instead, we will pass it in when calling the function

In [ ]:

from autora.experimentalist.random import pool
# ** Experimentalist ** #
@on_state()
def experimentalist_on_state(variables, num_samples):
    return Delta(conditions=pool(variables, num_samples))

from autora.experimentalist.random import pool # ** Experimentalist ** # @on_state() def experimentalist_on_state(variables, num_samples): return Delta(conditions=pool(variables, num_samples))

Experiment Runner¶

Here, we will serve the experiment and store data via Firebase and recruit participants via Prolific. We assume that you already set up an experiment on Firebase. It is recommended to use the autora-firebase-runner and manually test the setup before using the autora-firebase-prolific-runner.

In [ ]:

# We will need json to parse the date from the runner
import json
from autora.experiment_runner.firebase_prolific import firebase_prolific_runner

# We will run our experiment on firebase and need credentials. You will find them here:
# (https://console.firebase.google.com/)
#   -> project -> project settings -> service accounts -> generate new private key

firebase_credentials = {
    "type": "type",
    "project_id": "project_id",
    "private_key_id": "private_key_id",
    "private_key": "private_key",
    "client_email": "client_email",
    "client_id": "client_id",
    "auth_uri": "auth_uri",
    "token_uri": "token_uri",
    "auth_provider_x509_cert_url": "auth_provider_x509_cert_url",
    "client_x509_cert_url": "client_x509_cert_url"
}

# Sleep time (seconds): The time between checks to the firebase database and updates of the prolific experiment
sleep_time = 30

# Study name: This will be the name that will appear on prolific, participants that have participated in a study with the same name will be
# excluded automatically
study_name = 'my autora experiment'

# Study description: This will appear as study description on prolific
study_description= 'RDK experiment'

# Study Url: The url of your study (you can find this in the Firebase Console)
study_url = 'www.my-autora-experiment.com'

# Study completion time (minutes): The estimated time a participant will take to finish your study. We use the compensation suggested by Prolific to calculate how much a participant will earn based on the completion time.
study_completion_time = 5

# Prolific Token: You can generate a token on your Prolific account
prolific_token = 'my prolific token'

# Completion code: The code a participant gets to prove they participated. If you are using the standard set up (with cookiecutter), please make sure this is the same code that you have providede in the .env file of the testing zone.
completion_code = 'my completion code'

# Exclude Studies

# simple experiment runner that runs the experiment on firebase
experiment_runner = firebase_prolific_runner(
    firebase_credentials=firebase_credentials,
    time_out=100,
    sleep_time=5)

experiment_runner = firebase_prolific_runner(
            firebase_credentials=firebase_credentials,
            sleep_time=sleep_time,
            study_name=study_name,
            study_description=study_description,
            study_url=study_url,
            study_completion_time=study_completion_time,
            prolific_token=prolific_token,
            completion_code=completion_code,
        )


# We need to wrap the runner to use it on the state. Here, we send the raw conditions.
@on_state()
def runner_on_state(conditions):
    data = experiment_runner(conditions)
    # We parse the return value of the runner. The return value depends on the specific
    # implementation of your online experiment (see testing_zone/src/design/main.js).
    # In this example, the experiment runner returns a list of strings, that contain json formatted
    # dictionaries.
    # Example:
    # data = ['{'coherence':.3, accuracy':.8}', ...]
    result = []
    for item in data:
        result.append(json.loads(item))
    return Delta(experiment_data=pd.DataFrame(result))

# We will need json to parse the date from the runner import json from autora.experiment_runner.firebase_prolific import firebase_prolific_runner # We will run our experiment on firebase and need credentials. You will find them here: # (https://console.firebase.google.com/) # -> project -> project settings -> service accounts -> generate new private key firebase_credentials = { "type": "type", "project_id": "project_id", "private_key_id": "private_key_id", "private_key": "private_key", "client_email": "client_email", "client_id": "client_id", "auth_uri": "auth_uri", "token_uri": "token_uri", "auth_provider_x509_cert_url": "auth_provider_x509_cert_url", "client_x509_cert_url": "client_x509_cert_url" } # Sleep time (seconds): The time between checks to the firebase database and updates of the prolific experiment sleep_time = 30 # Study name: This will be the name that will appear on prolific, participants that have participated in a study with the same name will be # excluded automatically study_name = 'my autora experiment' # Study description: This will appear as study description on prolific study_description= 'RDK experiment' # Study Url: The url of your study (you can find this in the Firebase Console) study_url = 'www.my-autora-experiment.com' # Study completion time (minutes): The estimated time a participant will take to finish your study. We use the compensation suggested by Prolific to calculate how much a participant will earn based on the completion time. study_completion_time = 5 # Prolific Token: You can generate a token on your Prolific account prolific_token = 'my prolific token' # Completion code: The code a participant gets to prove they participated. If you are using the standard set up (with cookiecutter), please make sure this is the same code that you have providede in the .env file of the testing zone. completion_code = 'my completion code' # Exclude Studies # simple experiment runner that runs the experiment on firebase experiment_runner = firebase_prolific_runner( firebase_credentials=firebase_credentials, time_out=100, sleep_time=5) experiment_runner = firebase_prolific_runner( firebase_credentials=firebase_credentials, sleep_time=sleep_time, study_name=study_name, study_description=study_description, study_url=study_url, study_completion_time=study_completion_time, prolific_token=prolific_token, completion_code=completion_code, ) # We need to wrap the runner to use it on the state. Here, we send the raw conditions. @on_state() def runner_on_state(conditions): data = experiment_runner(conditions) # We parse the return value of the runner. The return value depends on the specific # implementation of your online experiment (see testing_zone/src/design/main.js). # In this example, the experiment runner returns a list of strings, that contain json formatted # dictionaries. # Example: # data = ['{'coherence':.3, accuracy':.8}', ...] result = [] for item in data: result.append(json.loads(item)) return Delta(experiment_data=pd.DataFrame(result))

Closed Loop¶

After setting up all components, we can use the runner just as other runners (see https://autoresearch.github.io/autora/)

In [ ]:

# Now, we can run our components
for _ in range(3):
    state = experimentalist_on_state(state, num_samples=2)  # Collect 2 conditions per iteration
    state = runner_on_state(state)  # This will collect data from two participants
    state = theorist_on_state(state)

# Now, we can run our components for _ in range(3): state = experimentalist_on_state(state, num_samples=2) # Collect 2 conditions per iteration state = runner_on_state(state) # This will collect data from two participants state = theorist_on_state(state)