COS2000 1-70004 Topic 13 Evaluation Data Analysis PDF

Preview

Summary

Download COS2000 1-70004 Topic 13 Evaluation Data Analysis PDF

Description

COS20001-70004 User Centred Design

13 Evaluation Data Analysis

Karola von Baggo

Photographic images used in this presentation:

mostly from Microsoft Clipart (unless otherwise specified)

Usability Evaluation Data Analysis Different types of data" • Demographic data" • Performance data" • Attitude (satisfaction) data" Severity ratings" Generalisation, validity and bias"

Required Reading Stone, Jarrett, Woodroffe & Minocha (2005) User Interface Design and Evaluation Chapters 25" CIF (see Blackboard)

After studying this topic you should be able to: Analyse and present demographic data." Analyse and present performance data according to CIF conventions." Analyse and present attitude data." Develop severity rating and recommendations for change based on data collected from evaluation.

After studying this topic you may be able to: Identify potential threats to validity such as bias, lack of generalisation and ecological validity.

Why do data analysis? ‣ Raw data can be difficult to interpret ‣ Analysis helps summarise, report and interpret the raw data ‣ Develop an understanding of what data can and can not say ‣ limitations in drawing conclusions from data ‣ Helps to identify problems ‣ Establish in requirements have been met Karola von Baggo Swinburne University of Technology

Common Industry Format for Reporting Usability Evaluation Results (CIF) ‣ ISO standard ‣ Specifies how to report evaluation results for an usability evaluation ‣ Must include details about the method of testing e ‣ Must include measures of: ativ m sum ‣ effectiveness g in ts ort p su l e e ‣ efficiency r r n For tio a ‣ satisfaction u val e

Karola von Baggo Swinburne University of Technology

Effectiveness

Efficiency

Tasks

Example usability requirements for sandwich ordering website

Satisfaction

Summative evaluation ‣ Check if usability requirements have been met " Karola von Baggo Swinburne University of Technology

Data analysis, interpretation and presentation ‣ Data types (qualitative vs quantitative, subjective vs objective) ‣ Data pre-processing ‣ Demographic data ‣ Performance data ‣ Attitude data Karola von Baggo Swinburne University of Technology

Data analysis, interpretation and presentation ‣ Threats to validity ‣ bias ‣ generalisation ‣ ecological validity

Karola von Baggo Swinburne University of Technology

Quantitative data ‣ single observations, ‣ represent an amount or a count, ‣ can be represented by a number. Qualitative data ‣ single observations ‣ represented by words or categories, ‣ can not be represented directly with a number.

On average it took participants 30 seconds to customise the shape. ‘I couldn't figure out how to manipulate the Bezier handles. It made me feel very frustrated.’ Karola von Baggo Swinburne University of Technology

Objective data ‣ based on something that exists in the world ‣ can be directly observed and verified ‣ scientific ‘gold standard’ Subjective data ‣ based on personal opinion or judgements ‣ cannot be directly observed, difficult to measure Karola von Baggo Swinburne University of Technology

I really like this...

I find this confusing...

Subjective data is very important in usability!! Karola von Baggo Swinburne University of Technology

Data analysis (pre-processing):

‣ clean up data ‣ look for outliers (e.g., some one choosing all extreme values,

someone who takes a lot longer than everyone else, makes more errors…)

‣ something to look into in more detail? ‣ consider removing if large number of participants

Karola von Baggo Swinburne University of Technology

Data analysis (pre-processing):

‣ missing data (e.g., a participant did not complete some or large parts of a survey, prototype ‘broke’)

‣ data entry error (e.g., 33 instead of 3 entered) Karola von Baggo Swinburne University of Technology

Data analysis: Traps for the unwary

‣ the importance of sample size ‣ 100% of people... sounds impressive, unless the sample size is only 4 (i.e., 4/4)

‣ always report the number of participants ‣ avoid % where number of P is less than 10

Karola von Baggo Swinburne University of Technology

Demographic data ‣ required to demonstrate that participants are members of user group ‣ must relate to user characteristics as per user profile ‣ Common data collected ‣ age, gender ‣ computer and internet experience ‣ job experience ‣ ownership and usage patterns Karola von Baggo Swinburne University of Technology ‣ task related

Example Data (Type of Internet connection at home):

‣ Broad band: 85 ~ 85/100 = 85% ‣ Dial up: 5 ~ 5/100 = 5% ‣ No Internet: 10 ~ 10/100 = 10% ‣ Pie chart display

Note: Figure number and title all figures and tables must be labeled and referred to in the text (e.g., Figure 6 shows...) Karola von Baggo Swinburne University of Technology

Example Data (Type of Internet connection at home):

‣ Broad band: 85 ‣ Dial up: 5 ‣ No Internet: 10 ‣ bar graph display (avoid fancy 3D etc)

Note: Use “Figure” to title images, graphs and diagrams Karola von Baggo Swinburne University of Technology

o Swinburne University of Technology

Karola von Baggo Swinburne University of Technology

Performance data ‣ Effectiveness ‣ completeness and accuracy ‣ Examples ‣ task completion ‣ assists ‣ errors ‣ goal completion

Karola von Baggo Swinburne University of Technology

for a i r rite on of c the leti p e r a co m at l u Wh f e ss c c ? su ask t s th i

Example Data Analysis: Lester’s Deli (circa 2009) Karola von Baggo Swinburne University of Technology

Task completion criteria Participant must:" • place an order for required sandwich, and" • report total cost of order

Example Data Analysis: Lester’s Deli (circa 2009) Karola von Baggo Swinburne University of Technology

Effectiveness

‣ unassisted task completion

‣ task completion ‣

‣ for individual participants ‣ “P1 completed the task” task completion rate ‣ for group of participants ‣ “The task completion rate for Task 1 was 50%”

‣

‣ for individual participants ‣ “P1 completed the task without help.” unassisted task completion rate ‣ for group of participants ‣ “The unassisted task completion rate for Task 1 was 50%” Karola von Baggo Swinburne University of Technology

nt ipa c i lete art p p com the y d l i l D sfu s e c ? suc ask t the

Lester’s Deli (circa 2009): P1

Karola von Baggo Swinburne University of Technology

Task 1 Unassisted Task Completion P1

Task completion: CIF convention" • write ‘100’ for completed

100

P2 P3 P4 P5 P6

Example Data Analysis: Reporting data

Karola von Baggo Swinburne University of Technology

nt ipa c i lete art p p com the y d l i l D sfu s e c ? suc ask t the

Lester’s Deli (circa 2009): P2

Karola von Baggo Swinburne University of Technology

Task completion criteria Participant did not:" • report total cost of order" • => task not completed

Lester’s Deli (circa 2009): P2

Karola von Baggo Swinburne University of Technology

Task 1 Unassisted Task Completion P1

100

P2

0

Task completion: CIF convention" • write ‘0’ for not completed

P3 P4 P5 P6

Example Data Analysis: Reporting data

Karola von Baggo Swinburne University of Technology

Effectiveness

‣ assist

‣ when a participant receives help from the evaluation team ‣ does not include participant accessing system help documentation ‣ i.e., if the participant was at home on their own they would have failed to complete the task (hence unassisted task completion rate)

Karola von Baggo Swinburne University of Technology

ant p i ic art lete p p the com y d l i l D u ssf e c ? suc ask t the

Lester’s Deli (circa 2009): P3

Karola von Baggo Swinburne University of Technology

Assists: • Asked for help, but did not receive help (OK)

Assists: • Asked for help, and received help => assume would not complete without help => not complete

Lester’s Deli (circa 2009): P3

Karola von Baggo Swinburne University of Technology

Task 1 Unassisted Task Completion

Assists

P1

100

0

P2

0

0

P3

0

1

Task completion: CIF convention" • write ‘0’ for not completed or completed with assistance Assistance • count number of times assistance provided

P4 P5 P6

Example Data Analysis: Reporting data

do s elp e h o d ng is ts u si s s P A by e d d e u l li inc upp s es pag . tem s y s Karola von Baggo Swinburne University of Technology

Task 1 Unassisted Task Completion P1

100

P2

0

P3

0

P4

100

P5

100

P6

100

Task completion summary: Task Completion Rate" • proportion of people who complete the task" • number of P who complete divided by number who attempt" ‣ 4/6 x 100 => 67%" ‣ “The unassisted task completion rate was 67%”

Example Data Analysis: Reporting data

Karola von Baggo Swinburne University of Technology

Task 1 Unassisted Task Completion

Assists

P1

100

0

P2

0

0

P3

0

1

P4

100

0

P5

100

0

P6

100

0

Assists summary: • the number of people who were provided with assistance" • the number of times assistance provided" ‣ “One participant requested and received help on Task 1”

Example Data Analysis: Reporting data

Karola von Baggo Swinburne University of Technology

Errors: • define what counts as an error before evaluation" ‣ navigation error: user deviates from optimal path, back button, home page button or main menu link used to return to original position" ‣ selection error: user chooses wrong option on interaction control such as a drop down box or check box etc" ‣ miss: did not do required action" ‣ miss typing not counted as an error(?)

Example Data Analysis: Reporting data

Karola von Baggo Swinburne University of Technology

Goal Achievement Task components:" • smoked meat " • fatty meat" • swiss cheese" • mustard" • price of sandwich" • delivery price

Example Data Analysis: Lester’s Deli (circa 2009) Karola von Baggo Swinburne University of Technology

Assists: • Asked for help, but did not receive help (OK)

Assists: • Asked for help, and received help => assume would not complete without help => not complete

Lester’s Deli (circa 2009): P3

Karola von Baggo Swinburne University of Technology

Task 1 Unassisted Task Completion

Goal Achievement (%)

P1

100

100

P2

0

83

P3

0

100

P4

100

100

P5

100

100

P6

100

100

Example Data Analysis: Reporting data

Goal Achievement: ‣ the percent of work completed successfully" ‣ P2 = 5/6 x 100" ‣ => 83%

Karola von Baggo Swinburne University of Technology

Performance Data ‣ Efficiency ‣ resources expended ‣ Examples ‣ task time ‣ ‘number of clicks’ and deviations from optimal path ‣ mental effort Karola von Baggo Swinburne University of Technology

0 Starts reading task

Orients to interface task

Rereading task

Working on task

Asks for help

Task time: ‣ can be divided up by activity" ‣ have a clear definition of task time (when it starts/finishes, what is included)" ‣ accuracy and units of measurement (min or sec)

Lester’s Deli (circa 2009): Task Time

Writes answer

4m 43s Says 'Finished

k tas e th es o d ? nd ere e h W and t r sta

Karola von Baggo Swinburne University of Technology

Task 1 Task Completion Time

Task Completion Time (min)

P1

1m 29s

1.48

P2

1m 32s

1.53

P3

4m 43s

1.72

P4

2m 19s

2.32

P5

2m 03s

2.05

P6

1m 53s

1.88

Example Data Analysis: Reporting data

Task Completion Time: ‣ convert to seconds or minutes (not minutes and seconds)" ‣ label column with units (i.e., min)

Karola von Baggo Swinburne University of Technology

Task 1 Unassisted Task Completion

Assists

Task Completion Time (min)

P1

100

0

1.48

P2

0

0

1.53

P3

0

1

1.72

P4

100

0

2.32

P5

100

0

2.05

P6

100

0

1.88

Example Data Analysis: Reporting data

Task Completion Time Summary: ‣ Mean Task Completion time includes all completed tasks:" (1.48 + 1.72 + 2.32 + 2.02 + 1.88)/5" ‣ Mean Unassisted Task Completion time includes only task completed without assistance: " (1.48 + 2.32 + 2.02 + 1.88)/4"

Karola von Baggo Swinburne University of Technology

Attitude Data

‣ Satisfaction ‣

c spe s a S h U d S E: tho T e O m N ng i r sco

ial

‣ comfort and acceptability Examples ‣ Likert scale ratings - frequency of rating categories, averages ‣ semantic differential ratings - frequency of rating categories, averages ‣ recommendation score - proportion of P who recommend ‣ open question responses - content analysis (positive vs negative comments) ‣ comments made during evaluation - content analysis

Karola von Baggo Swinburne University of Technology

System Usability Scale (SUS) - 10 item questionnaire

ggo Swinburne University of Technology

Scoring the SUS

Add up the scores and multiply by 2.5 for a score out of 100

Karola von Baggo Swinburne University of Technology

Please circle the number that describes your experience using YourEmail in relation to the following adjectives: hard 1 2 3 4 5 easy confusing 1 2 3 4 5 understandable s exciting 1 2 3 4 5 boring m y n nto a stressful 1 2 3 4 5 relaxing e us

Semantic Differential

• simple question format • good for getting peoples feelings about a system

Baggo Swinburne University of Technology

Please circle the number that best represents how you feel about the following statements. 1. I enjoy working with computers. Strongly 1 2 3 4 5 Disagree

Strongly Agree

2. Learning to use new software is stressful. Strongly 1 2 3 4 5 Disagree

Strongly Agree

Likert Scale • rating of strength of agreement with statements • good for measuring opinions and feelings • do not combine ratings for ratings of different questions unless questionnaire standardised (i.e. don’t take average of 1 and 2) ggo Swinburne University of Technology

Task 1 Difficulty Rating

Task 2 Difficulty Rating

Task 3 Difficulty Rating

SUS score

Recommend to friend

P1

3

2

1

77 yes

P2

2

1

2

81 yes

P3

2

2

3

67 no

P4

4

3

3

62 no

Example satisfaction results for iPhone gym exercise recording app (where Task Difficulty was rated on a scale of 1 = Very Easy and 5 = Very Hard

Karola von Baggo Swinburne University of Technology

Interpretation of data ‣ What does it all mean? ‣ Have we met our usability requirements? ‣ What problems did participants have (if any)? ‣ How bad were the problems? ‣ Do we need to make changes? If so, what kind of changes? Karola von Baggo Swinburne University of Technology

Interpretation of data ‣ Look for convergence of data (all metrics pointing in the same direction) ‣ Look for patterns in different sub-user groups (if you have the numbers) ‣ do low experience users show different satisfaction rating to high experience users? ‣ do different age groups have different problems? Karola von Baggo Swinburne University of Technology

Compare obtained data with usability requirements... ‣ Poor usability: ‣ Lower than expected task completion rates ‣ Higher than expected number of assists ‣ Higher than expected number of errors ‣ Longer than expected completion times ‣ Lower than expected satisfaction ratings ‣ Identify which parts of interface appear to be responsible

Karola von Baggo Swinburne University of Technology

Interpretation of data ‣ assign severity rating to possible problem areas ‣ use data to help justify severity ratings

Karola von Baggo Swinburne University of Technology

Severity Rating 0 = This is a not usability problem " 1 = Cosmetic problem only: need not be fixed unless extra time is available on project" 2 = Minor usability problem: fixing this should be given low priority" 3 = Major usability problem: important to fix, so should be given high priority" 4 = Usability catastrophe: imperative to fix this before product can be released "

Assign severity ratings Frequency

‣ common ‣ rare

Impact ‣ difficult to overcome ‣ easy to overcome

Persistence ‣ ongoing problem ‣ one time only

Karola von Baggo Swinburne University of Technology

Severity Rating Frequent task One time problem (if used frequently?) Difficult to overcome (if charged more than expected) -> 3: major problem?

Karola von Baggo Swinburne University of Technology

Recognise the limitations of your results. Karola von Baggo Swinburne University of Technology

General Population Target Population Accessible Population

Sample Population

Limitations in your ability to generalise to other samples.

Karola von Baggo Swinburne University of Technology

Issue of sample size Faulkner (2003)

Limitations in your ability to generalise to other situations - ecological validity.

Karola von Baggo Swinburne University of Technology

Biases can also distort the results. Karola von Baggo Swinburne University of Technology

...you'll never have all the information you need to make a decision. If you did, it would be a foregone conclusion, not a decision - David Mahoney...