Page 293 -

282 CHAPTER 10 Usability testing

USER NIGHTS AT YAHOO!—CONT’D
Yahoo conducts several User Nights per quarter, each focused on a
particular product or product feature, like Yahoo Search, Yahoo Finance, or
Yahoo Fantasy Sports. The product manager (PM) will partner with their
embedded UX researcher to determine the goals of the event (e.g. user
reaction to a Yahoo product or feature), the number, and demographics of the
participants and outline the tasks users will be asked to complete during User
Night.
The UXRA researcher works with the project manager before the User
Night to create a script based on the task list. The scripts are very detailed
and comprehensive including methods of setting user expectations, “do’s and
don’ts,” reminders for the Yahoos conducting 1:1 interactions, and the step-
by-step way in which the specific tasks are to be conducted. An accessibility
specialist will also be consulted to determine if assistive technology users
will need to be recruited. The featured product and the specific tasks to be
covered must not have any significant accessibility issues (i.e., a screen reader
user should be able to successfully complete the tasks) when individuals with
disabilities will be recruited.
Following this initial meeting, those in UXRA tasked with recruiting begin
to solicit users for participation. Recruiting is accomplished using multiple
methods including emailing a list of individuals who have volunteered to
be included in Yahoo’s User Night database and making targeted calls to
individual users. Some User Nights require a higher percentage of users who
have used Yahoo products previously and others require those who have never
used a Yahoo product or participated in a previous User Night. The number of
participants recruited for a User Night averages from 50 to 100 depending on
the study.
To encourage participation, User Nights are held in the evening from
about 7:00 to 9:00 pm. Upon arrival, users register, sign NDAs (nondisclosure
agreements) and are feted with “heavy” hors d’oeuvres or a light dinner. This
serves several practical purposes: it ensures everyone is on time and ready when
the study begins, socializes, and relaxes nervous participants, and provides time
to prepare Yahoo product teams for their 1:1 interactions and coach them on
best practices for observation.
One hour before the start of a Yahoo User Night, the UXRA researcher and
PM conduct an orientation meeting with the Yahoos who will help facilitate
the user testing. They review the User Night agenda and schedule (welcome,
dinner, pairing with users, making observations, note-taking, debriefing, etc.),
review the script and answer any last-minute questions. Due to the large number
of product developers at Yahoo, User Night is often as new to the Yahoo product
developers as it is to the user, so this employee orientation time is crucial in
making the night successful and enjoyable.