IAT 800 - Simon Fraser University

IAT 800 - Simon Fraser University

IAT 334 Experimental Evaluation ______________________________________________________________________________________ SCHOOL OF INTERACTIVE ARTS + TECHNOLOGY [SIAT] | WWW.SIAT.SFU.CA Evaluation g g Evaluation styles Subjective data Questionnaires, g Interviews Objective data Observing Users

Techniques, Recording g Usability Specifications Why, How March 10, 2011 IAT 334 2 Our goal? March 10, 2011 IAT 334 3

Evaluation g Earlier: Interpretive and Predictive Heuristic evaluation, walkthroughs, ethnography g Now: User involved Usage observations, experiments, interviews... March 10, 2011 IAT 334 4

Evaluation Forms g Summative After a system has been finished. Make judgments about final item. g Formative As project is forming. All through the lifecycle. Early, continuous. March 10, 2011 IAT 334 5 Evaluation Data Gathering g

Design the experiment to collect the data to test the hypothesis to evaluate the interface to refine the design Information we gather about an interface can be subjective or objective g Information also can be qualitative or quantitative March10, 2011 IAT 334 Which are tougher to measure? g 6 Subjective Data g

Satisfaction is an important factor in performance over time g Learning what people prefer is valuable data to gather March 10, 2011 IAT 334 7 Methods g Ways of gathering subjective data Questionnaires Interviews Booths (eg, trade show)

Call-in product hot-line Field support workers March 10, 2011 IAT 334 8 Questionnaires g g Preparation is expensive, but administration is cheap Oral vs. written Oral advs: Can ask follow-up questions Oral disadvs: Costly, time-consuming g

Forms can provide better quantitative data March 10, 2011 IAT 334 9 Questionnaires g Issues Only as good as questions you ask Establish purpose of questionnaire Dont ask things that you will not use Who is your audience? How do you deliver and collect questionnaire? March 10, 2011

IAT 334 10 Questionnaire Topic g g Can gather demographic data and data about the interface being studied Demographic data: Age, gender Task expertise Motivation Frequency of use Education/literacy March 10, 2011

IAT 334 11 Interface Data g Can gather data about screen graphic design terminology capabilities learning overall impression ... March 10, 2011 IAT 334

12 Question Format g Closed format Answer restricted to a set of choices Characters on screen hard to read 1 2 March 10, 2011 3 4 IAT 334

5 easy to read 6 7 13 Closed Format g Likert Scale Typical scale uses 5, 7 or 9 choices Above that is hard to discern Doing an odd number gives the neutral choice in the middle March 10, 2011 IAT 334

14 Closed Format g Advantages Disadvantages Must Clarifycover alternatives whole range All Easily should quantifiable be equally likely Dont Eliminates get interesting, useless answers different

reactions March 10, 2011 IAT 334 15 Issues g Question specificity Do you have a computer? g Language Beware terminology, jargon g g

Clarity Leading questions Can be phrased either positive or negative March 10, 2011 IAT 334 16 Issues g Prestige bias People answer a certain way because they want you to think that way about them g

g g Embarrassing questions Hypothetical questions Halo effect When estimate of one feature affects estimate of another (eg, intelligence/looks) March 10, 2011 IAT 334 17 Deployment g Steps Discuss questions among team

Administer verbally/written to a few people (pilot). Verbally query about thoughts on questions Administer final test March 10, 2011 IAT 334 18 Open-ended Questions g g g Asks for unprompted opinions Good for general, subjective information, but difficult to analyze rigorously

May help with design ideas Can you suggest improvements to this interface? March 10, 2011 IAT 334 19 Ethics g g g People can be sensitive about this process and issues Make sure they know you are testing software, not them

Attribution theory Studies why people believe that they succeeded or failed--themselves or outside factors (gender, age differences) g Can quit anytime March 10, 2011 IAT 334 20 Objective Data g Users interact with interface You observe, monitor, calculate, examine, measure,

g g Objective, scientific data gathering Comparison to interpretive/predictive evaluation March 10, 2011 IAT 334 21 Observing Users g Not as easy as you think g

One of the best ways to gather feedback about your interface Watch, listen and learn as a person interacts with your system g March 10, 2011 IAT 334 22 Observation g Direct Indirect Video In same

recording room Reduces Can be intrusive intrusiveness, but doesnt it of eliminate Users aware Cameras your presence focused on screen, face & keyboard Only see it one Gives time archival record, but can spend a time reviewing it lot Mayofuse semitransparent mirror to reduce IAT 334

March 10, 2011 23 Location g Observations may be In lab - Maybe a specially built usability lab Easier to control Can have user complete set of tasks In field Watch their everyday actions More realistic Harder to control other factors March 10, 2011 IAT 334

24 Challenge g In simple observation, you observe actions but dont know whats going on in their head g Often utilize some form of verbal protocol where users describe their thoughts March 10, 2011 IAT 334 25

Verbal Protocol g One technique: Think-aloud User describes verbally what s/he is thinking and doing What they believe is happening Why they take an action What they are trying to do March 10, 2011 IAT 334 26 Think Aloud g Very widely used, useful technique

Allows you to understand users thought processes better g Potential problems: g Can be awkward for participant Thinking aloud can modify way user performs task March 10, 2011 IAT 334 27 Teams g

Another technique: Co-discovery learning Join pairs of participants to work together Use think aloud Perhaps have one person be semiexpert (coach) and one be novice More natural (like conversation) so removes some awkwardness of individual think aloud March 10, 2011 IAT 334 28 Alternative g g

What if thinking aloud during session will be too disruptive? Can use post-event protocol User performs session, then watches video afterwards and describes what s/he was thinking Sometimes difficult to recall March 10, 2011 IAT 334 29 Historical Record g In observing users, how do you capture events in the session for later analysis?

March 10, 2011 IAT 334 30 Capturing a Session g 1. Paper & pencil Can be slow May miss things Is definitely cheap and easy Task 1 Time 10:00 10:03 10:08 10:22 March 10, 2011

Task 2 S e IAT 334 Task 3 S e 31 Capturing a Session g 2. Audio tape Good for talk-aloud

Hard to tie to interface g 3. Video tape Multiple cameras probably needed Good record Can be intrusive March 10, 2011 IAT 334 32 Capturing a Session g 4. Software logging Modify software to log user actions Can give time-stamped key press or

mouse event Two problems: Too low-level, want higher level events Massive amount of data, need analysis tools March 10, 2011 IAT 334 33 Assessing Usability g Usability Specifications Quantitative usability goals, used a guide for knowing when interface is good enough Should be established as early as possible in development process

March 10, 2011 IAT 334 34 Measurement Process g If you cant measure it, you cant manage it g Need to keep gathering data on each iterative refinement March 10, 2011 IAT 334

35 What to Measure? g Usability attributes Quantitative Initial performance Long-term performance Learnability Retainability Advanced feature usage First impression Long-term user satisfaction March 10, 2011 IAT 334 36

How to Measure? g Benchmark Task Specific, clearly stated task for users to carry out g Example: Calendar manager Schedule an appointment with Prof. Smith for next Thursday at 3pm. Users perform these under a variety of conditions and you measure performance March 10, 2011 IAT 334 37 g

Assessment Technique Usability Measure attribute instrument Initial perf Benchmk task First Quest impression March 10, 2011 Value to Current Worst be measured level acc level Length of

15 secs 30 secs time to (manual) success add appt on first trial -2..2 ?? 0 IAT 334 Planned Best poss target level level 20 secs 10 secs

0.75 1.5 Observ results 38 Summary g Measuring Instrument Questionnaires Benchmark tasks March 10, 2011 IAT 334 39

Summary g Value to be measured Time to complete task Number of percentage of errors Percent of task completed in given time Ratio of successes to failures Number of commands used Frequency of help usage March 10, 2011 IAT 334 40 Summary g

Target level Often established by comparison with competing system or non-computer based task March 10, 2011 IAT 334 41 Ethics g g Testing can be arduous Each participant should consent to be in experiment (informal or formal) Know what experiment involves, what to expect, what the potential risks are

Must be able to stop without danger or penalty g All participants to be treated with respect Nov 2, 2009 IAT 334 42 g Consent g Why important? People can be sensitive about this process and issues Errors will likely be made, participant may feel inadequate May be mentally or physically strenuous g

What are the potential risks (there are always risks)? Examples? g Vulnerable populations need special care & consideration (& IRB review) Children; disabled; IAT pregnant; students (why?) 43 334 Nov 2, 2009 Before Study g g Be well prepared so participants time is

not wasted Make sure they know you are testing software, not them (Usability testing, not User testing) g g g g Maintain privacy Explain procedures without compromising results Can quit anytime Administer signed consent form Nov 2, 2009 IAT 334

44 During Study Make sure participant is comfortable Session should not be too long Maintain relaxed atmosphere Never indicate displeasure or anger Nov 2, 2009 IAT 334 45 After Study

State how session will help you improve system Show participant how to perform failed tasks Dont compromise privacy (never identify people, only show videos with explicit permission) Data to be stored anonymously, securely, and/or destroyed Nov 2, 2009 IAT 334

46 One Model March 10, 2011 IAT 334 47

Recently Viewed Presentations

  • News Reports - Peel District School Board

    News Reports - Peel District School Board

    Lead sentence addresses 5Ws. Supporting details are specific and well organized. Quotations with sources included. Follows the proper structure of the news report - with the most important information listed first. Proper spelling and grammar. Factual tone
  • What Does the Next Generation System Look Like?

    What Does the Next Generation System Look Like?

    What Does the Next Generation System Look Like? Marshall Breeding. Independent Consult, Author, Founder and Publisher, Library Technology Guides
  • Kaizen Approach to Closing Achievement Gaps for First-Year ...

    Kaizen Approach to Closing Achievement Gaps for First-Year ...

    1.Much more about mindset than skillset!...CC - MVP!!. Single greatest thing one can do to increase opportunities in life is the pursuit of higher education!. Two-thirds of jobs require postsecondary credentials. Culture of quality not determined by what is preached,...
  • L'énergie hydrolienne - Le blog de M. Jalenques

    L'énergie hydrolienne - Le blog de M. Jalenques

    Hammerfest-Strom. Open-Hydro OceanEnergyInc Atlantic Florida University. CONCLUSION. Bien que l'énergie hydrolienne ait tardé à apparaître, elle est devenueaujourd'hui une sourced'énergierenouvelable à partentièredotéed'un potentiel énergétique non négligeable.
  • Clinical Implications of WHI or, what's a doctor to do?

    Clinical Implications of WHI or, what's a doctor to do?

    Number of and Percentage Change in US Prescriptions for Hormone Therapy Between January-Jane 2002 and January-June 2003 by Formulation Grady D, et al. Obstet Gynecol 2003; 102: 1233-9. Troublesome symptoms
  • Motivation and Emotion - Los Angeles Mission College

    Motivation and Emotion - Los Angeles Mission College

    For example, we pursue sex in order to reproduce to propagate the human species. It is an innate biological need. Instinctual Behaviors Reproduction and social dominance Human beings are territorial "this is our space" by nature. Some animals have instinctual...
  • Metric Conversions and Calculating Density

    Metric Conversions and Calculating Density

    Now you try… A sample of water has a mass of 14 g. When placed in a graduated cylinder, the volume of the water reads 14 ml. What is the density of water?
  • Name of Presentation - test.aorn.org

    Name of Presentation - test.aorn.org

    Assign a traffic director. Show the fire response team or fire department personnel to the location. These responsibilities must be individualized to each fire and are for the fire that is rapidly spreading or is large and is on the...