GDC 2005 - Cornell University

GDC 2005 - Cornell University

Game Optimization through Large-Scale Experimentation Erik Andersen Ph.D. student Center for Game Science University of Washington Computer Science Dept.

Music Art Sound effects Gameplay

? engagement retention revenue

A/B Testing A/B Testing A B

Metrics Levels completed

Time played Return rate

Aesthetics Secondary Objectives Tutorials

Refraction Hello Worlds Foldit Refraction

Rating: 3.8 / 5 400,000 plays Hello Worlds! Rating: 4.2 / 5

1,385,000 plays Foldit 240,000 players Player Tracking

Flash New cache / login name players only

Statistical significance 95% confidence level (p<0.05)

Experiment #1: Audio Music Sound Effects Result:

Music and sound effects did not matter Experiment #2: Animations Result:

Animations improved engagement Refraction 12 10 8 6

4 2 0 No Animations No Animations

Animations Levels Completed Refraction 600 500

400 300 200 100 0 No Animations

No Animations Animations Time Played (s) Refraction

No Animations 15 10 No Animations

Animations 5 0 Return Rate (%)

Hello Worlds 500 400 300 200 100 0

No Animations No Animations Animations Time Played (s)

Hello Worlds No Animations 20 15

No Animations Animations 10 5 0

Return Rate (%) Experiment #3: Secondary Objectives (Super Mario Bros., Nintendo 1985)

100 % of players 0 Levels

Hypothesis 100 With Coins % of

players 0 Levels Reality 100

With Coins % of players 0

Levels What about time played? 100 % of players

0 Time Hypothesis 100

With Coins % of players 0 Time

Hypothesis 100 With Coins % of

players 0 Time Reality: Many Players Quit Sooner 100

With Coins % of players 0

Time 100 % of players 0

Time Result: secondary objectives harmed engagement

(Assassins Creed, Ubisoft 2007) Maybe easier is better? (Super Mario Bros., Nintendo 1985) 100

Off-path Coins % of players 0

Time 100 On-path Coins % of

players 0 Time Result:

secondary objectives were good if they supported the primary objectives Experiment #4: Tutorials Foldit 8

No Animations 6 No Tutorials Tutorials

4 2 0 Levels Completed Foldit

800 No Animations 600 No Tutorials Tutorials

400 200 0 Time Played (s)

Refraction and Hello Worlds: no effect Result: text tutorials helped only in

the most complex game Number of concepts 11 13

24 Context-sensitivity Context-sensitive

Context-insensitive Result: context-sensitive help was better (if tutorials helped at all)

Foldit 8 6 Contextinsensitive Contextsensitive

4 2 0 No Animations Levels Completed

Foldit 800 600 Contextinsensitive Contextsensitive

400 200 0 No Animations

Time Played (s) Lets gather more data Let us A/B test your games!

Coauthors Yun-En Liu Richard Snider Roy Szeto Seth Cooper Eleanor ORourke Jeff Lowdermilk

David Truong Zoran Popovi Acknowledgements Kongregate: August Brown and Anthony Pecorella Logging: Kefan Xu Refraction Team: Erik Andersen, Yun-En Liu, Marianne Lee, Eric Butler, Brian Britigan,

Stephen Sievers, Roy Szeto, Mai Dang, Christian Lee, Ethan Apter, Emma Lynch, Happy Dong, Zorah Lea Fung, Justin Irwen, Seth Cooper, Franois Boucher-Genesse, Zoran Popovi Hello Worlds Team: Rich Snider, Michael Eng, Marianne Lee, Blake Thompson, Jeff Flatten Foldit Team: Seth Cooper, Adrien Treuille, Firas Khatib, Janos Barbaros, Joshua Snyder, Alex Cho Snyder, Jeff Flatten, Jeff Lowdermilk, Dun-Yu Hsiao, Jeehyung Lee, David Salesin, David Baker, Zoran Popovi

[email protected]

Recently Viewed Presentations

  • WISCONSIN UNIVERSITY OF WISCONSIN - MADISON Broadening the

    WISCONSIN UNIVERSITY OF WISCONSIN - MADISON Broadening the

    Variable. Uses . Train. Variable. penguin. constant. ILP : Background Knowledge. Logical statements expressing what is known. ... Advice-Taking via Human-Computer Interface (HCI) [ Walker et al., 2011 ] Parameter Tuning via the . Onion [ Walker et al., 2010...
  • Snímek 1

    Snímek 1

    Brno University Of Technology [email protected] Lukáš Burget, Michal Fapšo, Valiantsina Hubeika, Ondřej Glembek, Martin Karafiát, Marcel Kockmann, Pavel Matějka, Petr Schwarz and Honza Černocký
  • History of Safety Evaluation and Regulatory Status of

    History of Safety Evaluation and Regulatory Status of

    Post-ingestive effect on in digestive processes, nutrient absorption and metabolism, energy hemostasis via vagal nerve activation. Tori et al. J. Gastroenterol (2013) 48:442-451 . Studies Reporting Adverse Effects . Injection into animal brain to overcome BBB.
  • 6.9 Curve Fitting with Polynomial Models - Weebly

    6.9 Curve Fitting with Polynomial Models - Weebly

    6.9 Curve Fitting with Polynomial Models. Use finite differences to determine the degree of a polynomial that will fit a given set of data. Use technology to find polynomial models for a given set of data.
  • Chapter 11 Section 2 - rl063.k12.sd.us

    Chapter 11 Section 2 - rl063.k12.sd.us

    Chapter 11 Section 2 State of Atmosphere Temperature vs. Heat Temperature: measures the movement of molecules Faster = Warmer Slower = Colder Measured in oF, oC, K Heat Heat: transfer of energy from one object to another due to difference...
  • Title for presentation, Arial 20 pt., bold, gray

    Title for presentation, Arial 20 pt., bold, gray

    Including Patrick Remington, Bridget Catlin, David Kindig, Amanda Jovaag, Angela Russell. ... Now, I'm going to turn it back over to Jan to talk a little more about how communities are taking action and how you community can use the...
  • CEC  Natural Gas Public Interest Research Program Large

    CEC Natural Gas Public Interest Research Program Large

    Industrial / Commercial Burners Area with least interchangeability data GTI is conducting a survey with California focus for 11-member sponsor group of Work is currently underway Expected completion of Jan. 31, 2006 Data to be input into CEC project CEC...
  • Diapositiva 1 - NHS Confederation

    Diapositiva 1 - NHS Confederation

    NHS European Office. Who are we? Established 2007. Part of the NHS Confederation. Funded by national NHS bodies. What do we do? Represent NHS organisations in EU policy-making and legislation. Assist the NHS with the implementation of EU law. Offer...