Double Sweet Experimental Design

Colab tip: Open the left sidebar (☰) -> Table of Contents to navigate this notebook.

from graph_scheduler import WithNode
%%capture

%pip install sweetpea
%pip install sweetbean

SweetBean: Creating A static Experiment

To create a simple experiment with SweetBean, we define a sequence of stimuli. This get added to a block, which in turn gets added to an experiment. Finally, we export the experiment to an HTML file.

Each stimulus is defined by a class, which can be found in the sweetbean.stimulus module.

SweetBean supports a variety of output-formats for your experiment. The most convenient to test your experiment locally is the html format. Here, we also add a path for local download of the data so we can look at what data is collected during the experiment.

from sweetbean import Block, Experiment
from sweetbean.stimulus import Text

seq = [
    # TODO: add RED in red for 2000ms
]

block = Block(seq)
experiment = Experiment([block])
experiment.to_html('experiment.html')

Reveal

seq = [
    Text(text='RED', color='red', duration=2000)
    ]

SweetBean: The Timeline Feature

The most prominent feature of SweetBean is the timeline feature. Experiments are characterized by two objects:

• A Stimulus Sequence: A sequence of stimuli that are presented to the participant over and over again.

• A Task Timeline: A sequence of parameters that parametrize the stimulus sequence.

from sweetbean.variable import TimelineVariable

timeline = [
    # TODO: add a timeline that parametrizes each sequence
]

seq = [
    # TODO: add RED in red for 2000ms
]

block = Block(seq, timeline=timeline)
experiment = Experiment([block])
experiment.to_html('experiment.html')

Reveal

timeline = [
    {'word': 'RED', 'color': 'red'},
    {'word': 'GREEN', 'color': 'green'},
]

seq = [
    Text(text=TimelineVariable('word'), color=TimelineVariable('color'), duration=2000),
]

A Perfectly Balanced Experiment:

Disproving the Myth of the Stroop Effect and Cognitive Control

import random
from sweetbean.stimulus import Blank

timeline = ...

seq = [
    Text(text=TimelineVariable('word'), color=TimelineVariable('color'), duration=1000),
    Blank(duration=400)
]

block = Block(seq, timeline=timeline)
experiment = Experiment([block])
experiment.to_html('experiment.html')

Reveal

incongruent_stimuli = [
    {'word': 'GREEN', 'color': 'red'},
    {'word': 'BLUE', 'color': 'red'},
    {'word': 'YELLOW', 'color': 'red'}
]
congruent_stimuli = [
    {'word': 'PURPLE', 'color': 'purple'},
    {'word': 'ORANGE', 'color': 'orange'},

]

timeline = random.choices(incongruent_stimuli, k=32) + random.choices(congruent_stimuli, k=4)
random.shuffle(timeline)
timeline = random.choices(incongruent_stimuli, k=4) + timeline

A More Perfectly Balanced Experiment:

Disproving the Myth of the Switch Costs

import random
from sweetbean.stimulus import Blank

timeline = [
    {'task': 'color naming', 'word': 'RED', 'color': 'red'},
    {'task': 'color naming', 'word': 'RED', 'color': 'green'},
    {'task': 'word reading', 'word': 'GREEN', 'color': 'blue'},
    {'task': 'word reading', 'word': 'BLUE', 'color': 'red'},
    {'task': 'color naming', 'word': 'BLUE', 'color': 'blue'},
    {'task': 'color naming', 'word': 'ORANGE', 'color': 'purple'},
    {'task': 'color naming', 'word': 'GREEN', 'color': 'red'},
    {'task': 'word reading', 'word': 'GREEN', 'color': 'green'},
    {'task': 'color naming', 'word': 'GREEN', 'color': 'green'},
    {'task': 'color naming', 'word': 'YELLOW', 'color': 'purple'},
    {'task': 'color naming', 'word': 'BLUE', 'color': 'red'},
    {'task': 'word reading', 'word': 'BLUE', 'color': 'blue'},
    {'task': 'word reading', 'word': 'RED', 'color': 'blue'},
    {'task': 'color naming', 'word': 'GREEN', 'color': 'red'},
    {'task': 'word reading', 'word': 'RED', 'color': 'yellow'},
    {'task': 'color naming', 'word': 'ORANGE', 'color': 'red'},
    {'task': 'word reading', 'word': 'RED', 'color': 'green'},
    {'task': 'word reading', 'word': 'PURPLE', 'color': 'blue'},
    {'task': 'word reading', 'word': 'RED', 'color': 'green'},
    {'task': 'color naming', 'word': 'RED', 'color': 'red'},
    {'task': 'word reading', 'word': 'RED', 'color': 'green'},
    {'task': 'color naming', 'word': 'BLUE', 'color': 'red'},
]

seq = [
    Text(text=TimelineVariable('task'), duration=1000),
    Text(text=TimelineVariable('word'), color=TimelineVariable('color'), duration=2000),
    Blank(duration=400)
]

block = Block(seq, timeline=timeline)
experiment = Experiment([block])
experiment.to_html('experiment.html')

A Reasonable Approach to Creating the Timeline: SweetPea

Regular Factors

from sweetpea import CrossBlock, synthesize_trials, experiments_to_dicts, Factor

### SweetPea ###
# TODO: implement factors here

design = []
crossing = []
constraints = []

cross_block = CrossBlock(design, crossing, constraints)
_timelines = synthesize_trials(cross_block, 1)
timelines = experiments_to_dicts(cross_block, _timelines)

### SweetBean ###
# TODO: implement stimulus sequence here
seq = [

]

blocks = []
for timeline in timelines:
    blocks.append(Block(seq, timeline=timeline))
experiment = Experiment(blocks)
experiment.to_html('experiment.html')

Reveal

### SweetPea ###
word = Factor('word', ['RED', 'GREEN'])
color = Factor('color', ['red', 'green'])
task = Factor('task', ['color naming', 'word reading'])


design = [word, color, task]
crossing = [word, color, task]
constraints = []

cross_block = CrossBlock(design, crossing, constraints)
_timelines = synthesize_trials(cross_block, 1)
timelines = experiments_to_dicts(cross_block, _timelines)

### SweetBean ###

seq = [
    Text(text=TimelineVariable('task'), duration=1000),
    Text(text=TimelineVariable('word'), color=TimelineVariable('color'), duration=2000),
    Blank(duration=400)
]

Derived Factors

from sweetpea import CrossBlock, synthesize_trials, experiments_to_dicts, Factor, DerivedLevel, WithinTrial

### SweetPea ###
word = Factor('word', ['RED', 'GREEN', 'BLUE', 'YELLOW', 'ORANGE', 'PURPLE'])
color = Factor('color', ['red', 'green', 'blue'])
task = Factor('task', ['color naming', 'word reading'])

## Congruency

# TODO: implement congruency derived factor here

congruency = ...

design = [word, color, task, congruency]
crossing = [congruency, task]
constraints = []

cross_block = CrossBlock(design, crossing, constraints)
_timelines = synthesize_trials(cross_block, 1)
timelines = experiments_to_dicts(cross_block, _timelines)

### SweetBean ###

seq = [
    Text(text=TimelineVariable('task'), duration=1000),
    Text(text=TimelineVariable('word'), color=TimelineVariable('color'), duration=2000),
    Blank(duration=400)
]

blocks = []
for timeline in timelines:
    blocks.append(Block(seq, timeline=timeline))
experiment = Experiment(blocks)
experiment.to_html('experiment.html')

Sampling 1 trial sequences using NonUniformGen.
Encoding experiment constraints...
Running CryptoMiniSat...

Reveal

## Congruency

### Functions
def is_congruent(_word, _color):
    return _word.lower() == _color.lower()

def is_incongruent(_word, _color):
    return not is_congruent(_word, _color)


congruent_level = DerivedLevel('congruent', WithinTrial(is_congruent, [color, word]))
incongruent_level = DerivedLevel('incongruent', WithinTrial(is_incongruent, [color, word]))

congruency = Factor('congruency', [congruent_level, incongruent_level])

Transition Factors

from sweetpea import CrossBlock, synthesize_trials, experiments_to_dicts, Factor, DerivedLevel, WithinTrial, Transition

### SweetPea ###
word = Factor('word', ['RED', 'GREEN', 'BLUE', 'YELLOW', 'ORANGE', 'PURPLE'])
color = Factor('color', ['red', 'green', 'blue'])
task = Factor('task', ['color naming', 'word reading'])


## Congruency

### Functions
def is_congruent(_word, _color):
    return _word.lower() == _color.lower()


def is_incongruent(_word, _color):
    return not is_congruent(_word, _color)


congruent_level = DerivedLevel('congruent', WithinTrial(is_congruent, [color, word]))
incongruent_level = DerivedLevel('incongruent', WithinTrial(is_incongruent, [color, word]))

congruency = Factor('congruency', [congruent_level, incongruent_level])

## Task Switching

# TODO: implement task switching derived factor here

task_switch = ...

design = [word, color, task, congruency, task_switch]
crossing = [congruency, task, task_switch]
constraints = []

cross_block = CrossBlock(design, crossing, constraints)
_timelines = synthesize_trials(cross_block, 1)
timelines = experiments_to_dicts(cross_block, _timelines)

### SweetBean ###

seq = [
    Text(text=TimelineVariable('task'), duration=1000),
    Text(text=TimelineVariable('word'), color=TimelineVariable('color'), duration=2000),
    Blank(duration=400)
]

blocks = []
for timeline in timelines:
    blocks.append(Block(seq, timeline=timeline))
experiment = Experiment(blocks)
experiment.to_html('experiment.html')

Reveal

## Task Switching
def is_switch(_task):
    return _task[0] != _task[-1]

def is_repeat(_task):
    return not is_switch(_task)

switch_level = DerivedLevel('switch', Transition(is_switch, [task]))
repeat_level = DerivedLevel('repeat', Transition(is_repeat, [task]))

task_switch = Factor('task_switch', [switch_level, repeat_level])

Constraints

In addition to counterbalancing, you can impose constraints on your design. For example, you might want to avoid that more than two incongruent trials of switch trials follow after each other. There are many predefined constraints in SweetPea with simple interfaces:

AtMostKInARow(factor, level, k), ExactlyKInARow(factor, level, k), …

Additional Features (new)

• MiniBlock Designs

• cContinuous factors (coming soon…)

SweetBean: Adaptive Parameters

Function Variables

from sweetpea import CrossBlock, synthesize_trials, experiments_to_dicts, Factor
from sweetbean.variable import FunctionVariable


### SweetPea ###
word = Factor('word', ['RED', 'GREEN'])
color = Factor('color', ['red', 'green'])


design = [word, color]
crossing = [word, color]
constraints = []

cross_block = CrossBlock(design, crossing, constraints)
_timelines = synthesize_trials(cross_block, 1)
timelines = experiments_to_dicts(cross_block, _timelines)

### SweetBean ###

## Add Function Variable

correct_key = ...


seq = [
    Text(text=TimelineVariable('word'), color=TimelineVariable('color'), choices=['j', 'f'], correct_key=correct_key, duration=2000),
    Blank(duration=400)
]

blocks = []
for timeline in timelines:
    blocks.append(Block(seq, timeline=timeline))
experiment = Experiment(blocks)
experiment.to_html('experiment.html')

Sampling 1 trial sequences using NonUniformGen.
Encoding experiment constraints...
Running CryptoMiniSat...

Reveal

## Add Function Variable

def correct_key_fct(cl):
    return 'f' if cl == 'red' else 'j'

correct_key = FunctionVariable('correct_key', correct_key_fct, [TimelineVariable('color')])

Data Variables

from sweetpea import CrossBlock, synthesize_trials, experiments_to_dicts, Factor
from sweetbean.variable import FunctionVariable, DataVariable

### SweetPea ###
word = Factor('word', ['RED', 'GREEN'])
color = Factor('color', ['red', 'green'])

design = [word, color]
crossing = [word, color]
constraints = []

cross_block = CrossBlock(design, crossing, constraints)
_timelines = synthesize_trials(cross_block, 1)
timelines = experiments_to_dicts(cross_block, _timelines)


### SweetBean ###

def correct_key_fct(cl):
    return 'f' if cl == 'red' else 'j'


correct_key = FunctionVariable('correct_key', correct_key_fct, [TimelineVariable('color')])


## Add Data Variable and get feedback from it
is_correct = ...

feedback_text = ...

seq = [
    Blank(duration=400),
    Text(text=TimelineVariable('word'), color=TimelineVariable('color'), choices=['j', 'f'], correct_key=correct_key,
         duration=2000),
    Text(text=feedback_text, duration=2000),
]

blocks = []
for timeline in timelines:
    blocks.append(Block(seq, timeline=timeline))
experiment = Experiment(blocks)
experiment.to_html('experiment.html')

Sampling 1 trial sequences using NonUniformGen.
Encoding experiment constraints...
Running CryptoMiniSat...

Reveal

## Add Data Variable and get feedback from it
is_correct = DataVariable('correct', 1)


def get_feedback(was_correct):
    return 'CORRECT' if was_correct else 'INCORRECT'


feedback_text = FunctionVariable('get_feedback', get_feedback, [is_correct])

Side Effects

from sweetpea import CrossBlock, synthesize_trials, experiments_to_dicts, Factor
from sweetbean.variable import FunctionVariable, DataVariable, SharedVariable, SideEffect

### SweetPea ###
word = Factor('word', ['RED', 'GREEN'])
color = Factor('color', ['red', 'green'])

design = [word, color]
crossing = [word, color]
constraints = []

cross_block = CrossBlock(design, crossing, constraints)
_timelines = synthesize_trials(cross_block, 1)
timelines = experiments_to_dicts(cross_block, _timelines)


### SweetBean ###

def correct_key_fct(cl):
    return 'f' if cl == 'red' else 'j'


correct_key = FunctionVariable('correct_key', correct_key_fct, [TimelineVariable('color')])

is_correct = DataVariable('correct', 1)


def get_feedback(was_correct):
    return 'CORRECT' if was_correct else 'INCORRECT'


feedback_text = FunctionVariable('get_feedback', get_feedback, [is_correct])

num_correct = ...

update_num_correct = ...

update_score_side_effect = ...

seq = [
    Blank(duration=400),
    Text(text=TimelineVariable('word'), color=TimelineVariable('color'), choices=['j', 'f'], correct_key=correct_key,
         duration=2000,
         side_effects=[update_score_side_effect]),
    Text(text=feedback_text, duration=2000),
    Text(duration=2000, text=num_correct)
]

blocks = []
for timeline in timelines:
    blocks.append(Block(seq, timeline=timeline))
experiment = Experiment(blocks)
experiment.to_html('experiment.html')

Sampling 1 trial sequences using NonUniformGen.
Encoding experiment constraints...
Running CryptoMiniSat...

Reveal

num_correct = SharedVariable("num_correct", 0)

update_num_correct = FunctionVariable(
    "update_num_correct", lambda score, value: score + value, [num_correct, is_correct]
)

update_score_side_effect = SideEffect(num_correct, update_num_correct)

SweetBean Data

Download CSV File on GitHub

Rows: One Row corresponds to one stimulus sequence Columns: the stimuli in sequence are indexed (0, 1, 2, …). Relevant data often has the prefix bean_

Additional Features

• Creating a Stimulus Image

• LLM as synthetic participants

Together: Create Our Stroop Experiment For the AutoRA Loop:

...