CI6315 User-Experience Design Thinking

Usability Test Report

This coursework is part of your TB2 assessment.

Learning Outcomes

Aims

The aim of this coursework is:

• To conduct a CIF standard usability test of an existing system.

What you need to do

1. i) usability tests a real-world, data-intensive desk-top or cloud-hosted application, such as the following:

1.systems development tools (diagrammers, modellers, programming environments; database configuration tools);

2.retail auction sites and marketplaces.

3.network management tools (modelling, monitoring, analysing).

4.games and digital media tools (image editors, renderers; games development tools, tools for ux design).

5.project management tools.

6.fantasy sports websites.

Examples include: www.prezi.com; Gimp2.0; www.basecamp.com; www.mindmeister.com. www.aptana.comfantasy.premierleague.comI

You may also conduct a comparative usability test of your whole group prototype from TB1 vs a real-world competitor.  This option is only feasible if the whole group prototype is large, robust, and realistic enough to support CIF-style testing – your group probably obtained an A for TB1.  But it is a good way of finding out how good your TB1 prototype really was, and it is a real-world exercise – actual products get tested this way, if the developers can afford to.

I SUGGEST YOU AVOID: craigslist, lingscars, and other examples, which are bizarre or whose brand is ‘old skool’, or ‘poor ease of use’. a tv remote controls, a cash point machine, and other common textbook cases are not sufficiently complex, and ‘cliche-ed’.

1. ii) apply the CIF standard method and report format presented in the lectures. Following this correctly defines what is “sound – ‘B’ grade” practical work and reporting. Top grades are achieved when you adapt parts of the CIF method to address the unique characteristics of the issues more fully at hand, for example, to assess:

user experience (trustworthiness, engagement, brand perception, persuasiveness), not just usability.

Interaction on a mobile device, rather than a desktop.

Interaction in a specific context of use (a domestic environment, or shop) rather than dedicated performance in an ‘office’.

Using additional indicators of effectiveness – counting new behaviours, using different questionnaires etc

Iii) test between 3 and 5 participants performing 3-5 tasks, and taking about 40mins to complete the session.   This might be, for example, 2 groups of two individuals (there are two user types), attempting to make a team, check this week’s points, and transfer a player in fantasy football league, on a mobile and desktop device (one device each). This size of test is usually sufficient to both state how usable the existing system is, and also to identify a range of usability issues with it – you should be discovering unexpected ways of improving the current design.

This is not an endurance exercise! Scope your test to answer specific research questions – you cannot test every user type, every task, and every context of use, and get ‘the ultimate answer’. If the usability test session lasts more than 1hr, then you might want to reduce the scope

1. iv) ask your friends and family to participate in the usability test, or play ‘participant’ for each other during workshop hours –no need to approach strangers! Or test unsafely (whilst driving, or in underpasses after midnight)

Suggested Structure

Please follow the structure suggested – it maps onto the marking scheme. The details of each section, however, may ideally be adapted to meet the needs of your unique problems, and the way you need to address it.

1. Introduction and Background (approx..1 –2 sides)

1.1. Social and Economic Trends   What real world changes in business or society set the context for this usability test? Why are these trends important and of interest to many people?

1.2. What organisation ‘owns’ the software? What are the business goals of the software? How does usability and user experience enable these business goals? Why is it timely to evaluate usability and user experience now? (Why evaluate it now)?

Remember to back up any claims with facts and figures (evidence) from credible, cited sources

2 Aims. (approx. ½ side) Which system /software /website are you going to test?   Statement of Aim the over-arching purpose is, at least, …to evaluate and enhance …

List of Objectives (Deliverables)the tangible, intermediate ‘products’ that will be produced by the process that finally outputs a Usability Test Report, for example, …. method … raw data … data analysis …

Table of Research Questions.  State the ‘questions about actual usability and user experience’ that your study will answer (aka Problem Statements). Give reasons for focussing upon these research questions

3 Method (approx. 2-3 sides) Throughout this section, remember to explain your decisions., and to support your argument with evidence, and citations to credible sources.

3.1. Method overview (approx. ½ side) You are expected to conduct a CIF-based usability test for this coursework. Describe the kind of method that CIF is, and explain why you need to apply e.g., how rich and representative is the dataset obtained by a usability test, and why is observation preferred to subjective reports o answer most research questions about usability, a ‘one-shot’ experimental design is sufficient (‘How usable is …’). Some research questions require a comparison e.g. ‘Is website A more usable than website B’? ‘Which is preferred?’. Research questions about learnability, may need a repeated measures design (before vs after).

3.2 Participants.  Characterise the target user group and the sample of participants that you need to recruit to answer your research questions.  Profile the actual participants anonymously in a table. How and why were participants recruited and selected? Give a reason for the decisions you made. In which respects is the sample not fully representative of the target user group? Why is this sample, nevertheless, representative enough?

3.3 Tasks Identify and outline the tasks you set participants. How and why were these tasks selected and defined? How were task instruction sheets laid out and why? Give a reason for the decisions you made and argue for their importance and ‘realism’.

3.4 Metrics Identify and define the UX criteria you will be using in the evaluation, and state how each criterion will be measured.  If you are going beyond standard CIF metrics, state the additional evaluation criterion (‘construct’), its indicator (directly observable correlate),

3.5 Materials & Tools Identify the ‘worksheets’ you will be using to conduct the test sessions and gather data (questionnaires, task instruction sheets, observation sheets etc). State the content and format for these documents and give reasons for these decisions. Include blank copies in an Appendix. Identify any

important tools you will be using (online surveys, screen recorders) and explain the advantages of using them.

3.6 Procedure       Describe exactly how each session was carried out, so that the reader could replicate your study if they wanted to.  Where necessary, explain how details of this procedure reduced, controlled, or measured, potential bias

Please keep your raw data (video recordings, completed questionnaires etc.) safe, secure, and backed up. You will need them for the presentation. You do not need to hand them in.

4. Results& Recommendations

4.1 Written summary of overall findings (approx. ½ side) Written as if you were talking to the project manager in a lift. Overall, how good is usability and user experience, in the scheme of things? Outline the most important problems, what needs to be done to resolve each, and what would the potential benefit be?

4.2 Performance Data (approx. 2-4 sides) This section focusses upon Tables of Quantitative Data –outcomes, times, counts and ratings etc. There is one Table per task, one row for each participant, and a row for the ‘average’. Use charts to highlight selected findings. Add a short commentary to tell me what the data shows. Additional qualitative data, such as observations and quotes, make performance data ‘come to life’.

4.3 User Experience Issues (approx. 2-4 sides) This section focusses upon the Table of User Experience Issues. One row for each Issue. Group the Issues by UI feature, and take care to state each issue fully, by following the template suggested in the lecture. For each issue, include a text outline of a redesign recommendation, and assign a priority for redesign.

4.4 Redesign Recommendations (approx. 1 side) Close the results section with design implications. Consolidate the high priority changes recommended in 4.3 into a ‘To Do’ List, illustrated with sketches and screenshots if necessary.   You do not need to implement these changes or create a mock-up.

5. Discussion (1/2 –1 side)

Consider, for example, unrepresentative samples of users/tasks/contexts, confounding variables and biases, inaccurate or unreliable indicators and their implications. Identify areas of uncertainty, and outline further possible work, and related evaluation studies.

6. References

Appendices

One for each Screener, Task Instruction Sheets, Observation sheets, Post-Test Questionnaire, and any customisation of the Moderator-Participant protocol.

NB** An Optimal path, Screening script, and List of Possible Errors are *not*requested in the report you hand-in, although you completed them in the workshops. They are for the moderator’s benefit.

Common Questions

What system /software / website should I test? The application, system, or website that you choose should be:

1. Of interest to you. You will spend the rest of the year working on it, so pick sites and applications that are interesting to you. What software do you use to pursue your interests or hobbies?
2. Add to your track record. What do you want a future employer to see on your CV? Mobile phone retail, banking, education, games, data analysis, science, programming, charities?
3. Requires resources you have. Where will you find participants for your usability test? (Friends and family, other students, work colleagues?), Can you take screen shots and/or record the existing interface? How will you give usability test participants access to it? Do you have sufficient domain expertise?

technically challenging. Often this means:

1. Innovative, leading-edge systems, that have not been perfected by anyone yet
2. Data intensive systems. Conveying large amounts of information clearly and intuitively, through a small screen is difficult

iii. non-core features. Developers typically spend most time perfecting the frequently used, business critical features (such as checkout pages, editing tables), and have no time left to fix mistakes with less frequently used, non-critical features (such as terms and conditions pages, organising references)

1. Open-source software, that need volunteers to finish it
2. Raise User Experience and Visualisation Issues. The majority of contemporary user interface are not just about ‘effectiveness’ -usefulness, usability, clarity and learnability etc. –but also about ‘experience’ –user perceptions (of trust, brand value, security etc) engagement, persuasion -and include significant amounts of data and content –most contemporary systems do.
3. Access. Avoid limited duration licenses; or limited download licenses –you will need to access the system until the hand-in the themes are intended to steer you away from simple, narrow problems with self-evident solutions that any person in the street could design, or out of date amateurishness. These are tempting as it is easy to get started, but at the end of the day, you will not have enough to report. Be ambitious and try it out something new and interesting –I like map-based holiday planners.

How many words can the report be?  about 4000 words

The target word length stated does not include captions, annotations, tables, illustrations, insert panels, quotes, and Appendices. If your draft is looking “too long”, could you use images, diagrams, tables, charts, inserts, or create Appendices? A picture with a caption and an annotation can say 1000 words. The word length stated is a target not a maximum i.e., once you reach the target, review your work.  Are you responding to the brief? Are you presenting your work concisely? Please continue –better to include all your answer than stop prematurely. There is no automatic penalty for exceeding the target word length, but the target length is enough to get an A so why are you still writing?  Write a full draft, leave it for a few days, and then revisit it with a critical eye.  Does each word, each line, provide new, valuable information and so deserve its place in your report? Or could you cut it, and get to the point sooner? or could you convey the point more clearly another way?  If in doubt, it is much better to include relevant material, and exceed the word limit, than to omit a section (and get an automatic zero for some part).  There is a penalty for verbosity, irrelevance, repetition of textbook material, or bland generalisations that a member of the public could have guessed.  It is harmful to hide or bury important information –bring it to the surface