[Abstract]
In a world where technology is readily available to us in all shapes and sizes, how does device size affect our ability to accomplish certain tasks? Processing speed, input method and screen size have all been studied as factors that affect the user. However, taken altogether, how will these factors affect a search and selection task, and why? This paper will specifically examine the differences between a mobile device and a computer with a large monitor. The differences in task performance on the different devices could have important implications for which tasks we choose to perform on which devices.
1. Mobile device users will use more category heuristics than large-screen users.
This is to be measured based on how many times the user refines his search on a given task. For example, if a user searches for "Summer Dresses > Floral" under Dresses, that is a value of '2' for the number of heuristics used to refine the basic search.
2. Mobile device users will be more satisfied with their selection because they had fewer choices.
The reasoning behind this hypothesis is that the mobile device users will be able to view fewer dresses, both because the processing/network speed on the mobile device is slower and because the screen is smaller. The large screen user will be able to view many more options. Thus, based on the paradox of choice, the mobile device user will be more satisfied choosing from X options whereas the large screen user will experience decision paralysis choosing from Y options (where Y >> X). There is one confounding variable in examining the paradox of choice in practice with this particular experimental set up: time. Users on the mobile device may be frustrated by the fact that though there are Y potential choices, he can only access X of them. This is to be measured based on pre and post survey questions.
3. Large-screen users will perform better, as rated by independent raters.
This is to be measured by having mechanical turkers rate the subjects' selections based on how well they think the selection fits the criteria presented in the prompt. I am also considering having 'fashion experts' rate all of the selections. This would give me multiple modes of measurement with which to confirm this hypothesis.
4. Mobile device users will feel like they need more time more than large-screen users.
I have selected 5 minutes per task, for a total of 10 minutes, as the allotted time for this experiment. This seems to be appropriate based on my pilot experiments in which some participants ran out of time and some did not. I am keeping track of how how long it takes each participant in each task to complete. I plan on comparing on a binary scale (needed more time---did not need more time) but I will have the data to do more involved time analysis if this becomes desirable.
[Methods]
1. I have altered the participant instructions to just leave the tab open to his/her selection when done and I will record this later. This allows me to have the participant complete the survey on the computer but just perform the task on a mobile/large device.
2. I also updated the instructions to include a more specific description of Jamie and Matt and what they are looking for:
Excerpt
[Task 1]
Read below for what kind of dress Jamie would like:
"I'm going to my friend's wedding in July and I want to find a fun, flirty dress. I am 5'8" with a pear-shaped build. I have brown hair and olive complexion. I want something that will look hot, but not distract from my friend's big day, nothing too flashy! Help me!"
[Task 2]
Read below for what kind of shirt Matt would like:
"I'm a little bit anxious about this pool party. I want to wear a shirt that will look casual, but make me seem in style. I don't usually get too hot in the sun, so I think I'd rather cover up than bare my arms. I'm 5'10" with dirty blond hair and freckles. I have the lean build of a distance runner. Make me look good!"
3. I am offering prizes as follows: "There will be a prize for the best, most appropriate dress selection and shirt selection. The winners will get their choice of a $5 Starbucks, Jamba Juice, itunes or Philz gift card. If the same person wins both dress and shirt, he/she will get a $15 card. "
My reasoning for giving more if someone wins both shirt and dress is to incentivize someone to try hard on both tasks and overcome task fatigue.
[Materials]
I am using a iphone 4G as my mobile device and 27" monitor as my large-screen.
I am using safari web browser for both conditions.
[Further Questions] **If anyone is commenting on my blog, advice on this area would be particularly desired!
1) I'm still not sure how I can watch/observe my participants on their mobile devices. I would have to be a huge creeper!
2) I am not sure exactly how I want mechanical turkers to rate the selections. Rank order? Rate each individual on a scale or multiple scales? A 'which is better' A/B comparison?
Looks like you've made a lot of progress in terms of ironing out details. Great!
ReplyDeleteFor 1), I think this is tough. Is it possible to recover their search refinement steps by just using the "back" arrow on the browser when they're finished? (However, you'd have to tell both conditions they couldn't use the back arrow to undo refinement steps.) You may just have to look over their shoulders, which is probably acceptable as long as you're doing it in an equally creepy way for both conditions.
2) When we've talked about this in class, I think there's some argument for Likert scales with a warmup period. This is a good thing to bring up in class next week when we're talking about AMT.
Are both mobile and computer tests being conducted in the same settings? same room, environment etc? Since mobile devices are typically used "on the go" or in multiple locations, users may have a bias towards the types of tasks that they already do on such devices, and so they may not want to spend as much time completing a task they do on a computer. I would also look into providing a user with an android or windows7 phone. It is likely most people are already used to the web browser interface on a computer, but not everyone has iphones and more specifically safari on iphone.
ReplyDeleteAnother thing that you might to measure is the efficiency of the tasks. Sure, maybe users will spend less time on mobile devices, but maybe the overall efficiency per time is higher output than in the desktop case.