Shuichiro Ogawa
日本語

Notes · updated 2026-11-03 · Markdown

A Checklist for User Interviews: What to Prepare, What to Ask, and How to Report to the Client

This note divides user interviewing with the people who use or buy something into nine stages, from planning through recruitment, consent, incentives, the interview guide and pilot, recording on the day, data storage, analysis, and reporting to the client, and turns each stage into a checklist of what to prepare and wh…

Contents (12)
  1. Why are the conventions of asking not enough?
  2. What should be agreed with the client in advance?
  3. Are the people who come really the people who meet the criteria?
  4. What should consent cover, and what should be kept?
  5. When, in what form, and how much should the incentive be?
  6. What should the interview guide contain, and how should it be tried?
  7. On the day, who records what?
  8. How long should data be kept, and how should it be deleted?
  9. When, and with whom, should the analysis be done?
  10. What should be given to the client, and in what form?
  11. What should be added when a screen operation check is included?
  12. How strong is the evidence for this list?

Why are the conventions of asking not enough?

User interviewing is the practice of meeting the people who use or buy something, before or while it is made, and listening to them to inform decisions about what to make. The conventions of asking (ask about past events instead of opinions, do not lead, listen in silence) are nearly the same in academia and industry (Conducting User Interviews: What Academia and Industry Share and Where They Diverge). Even an interviewer who has mastered these conventions can still fail. The person who came did not meet the criteria. The record button was never pressed. No evidence of consent remains. After reading the report, the client took one participant’s remark as a trend across all users. None of these failures arises from technique during the interview itself, and each of them can be prevented by preparation before and after it.

This note divides the work from receiving a commission to reporting to the client into nine stages, and lists for each stage what to prepare and what to confirm. The scope is one-to-one interviewing with the people who use or buy something, carried out to inform decisions about UIs, services, and businesses. The client may be an external client or an internal team that requested the research.

The evidence comes from four kinds of material. These are 28 academic works (peer-reviewed papers and methodological reviews), 27 guidance documents from public bodies, guidance on laws, and industry codes of research ethics, 26 materials from research firms and consultancies, and 18 first-hand accounts by practitioners. Numbers and rules were checked against the relevant passages of the full texts wherever these could be obtained. Academic works whose full texts could not be obtained are marked “abstract” after the year and were used only within the scope of the abstract. Each checklist item gives the source of its basis in parentheses. Where the source is an industry code, that code is self-regulation imposed on members, not a law.

What should be agreed with the client in advance?

The purpose of the interviews should be agreed with the client in writing. The GOV.UK guidance asks teams to set clear objectives for each round, to agree them as a team, and, before starting, to understand which assumptions they want to test and what they need to know now to make an informed decision about what to do next. A practitioner book on service design recommends starting with internal interviews in the organization before going into the field, and checking whether stakeholders share an understanding of the problem (Stickdorn et al., This Is Service Design Doing method pages). If the views on the client side are not aligned, the results of the interviews will not answer anyone’s question.

A Japanese industry code specifies the items that a proposal to the client must contain. These are the method of selecting participants (including the source of any list), the research method, and the fieldwork design (number of participants, conditions, date and time, venue, and duration) (JMRA, Group Interview Guidelines). NN/g recommends putting the purpose, the participants, the method and procedure, and related documents such as the consent form and screener into a single research plan (Rosala 2024). When stakeholders are invited to observe, NN/g recommends that they attend at least two sessions so that they do not draw conclusions from a single one (Farrell 2017, “How to Collaborate with Stakeholders”).

  • Wrote down the decisions the research should inform and the assumptions to test, and agreed them with the client (GOV.UK)
  • Asked inside the client organization, first, whether stakeholders share an understanding of the problem (This Is Service Design Doing)
  • Put the purpose, participant criteria, method, schedule, and related documents (consent form, screener, interview guide) into one research plan (NN/g)
  • Stated in the proposal the selection method and source of the list, the number of participants, conditions, date and time, venue, and duration (JMRA)
  • Decided who will observe and how many sessions each observer will attend (NN/g)
  • Agreed with the client on the form of the report (when, to whom, in what format)

Are the people who come really the people who meet the criteria?

A screener is a short questionnaire used to select applicants who meet the criteria for participation. Robinson (2014, abstract) divides the design of recruitment into four points. These are defining the sample universe by inclusion and exclusion criteria, deciding the sample size, selecting a sampling strategy, and sourcing the sample (advertising, incentives, avoidance of bias, and consent). Whatever the strategy, purposeful sampling, which deliberately selects information-rich cases, is the basis of qualitative research, and criterion sampling is the most common strategy in implementation research (Palinkas et al. 2015).

When recruitment is open and online, applicants who pretend to meet the criteria get in. Pozzar et al. (2020) recruited survey participants through social media and received 271 responses within 7 hours. Using indicators such as inconsistent responses to verifiable items, they classified 94.5% (256) as fraudulent, and the remaining 5.5% (15) as suspicious. Jean Louis and Thompson (2025) checked by telephone and email the 91 people recruited through social media who had completed the data, and only 8 (8.8%) were valid participants with usable data. The rest had given disconnected phone numbers or Google Voice numbers that went unanswered. Sharma et al. (2024) examined the 254 expressions of interest received for online recruitment that offered 50 Australian dollars, and judged that all potential participants were likely imposters. As red flags, they list large volumes of email responses, strange timings (in the middle of the night in the recruiting country), and unlikely demographic characteristics. These are cases from health research, and the rates may differ in design research. Even so, both kinds of study offer incentives and recruit online.

A single countermeasure is not enough. Jean Louis and Thompson (2025), following the Swiss Cheese Model of accident prevention, recommend layering ten prevention and verification measures. The layers include carefully designed recruitment media and compensation protocols, vetting of participant eligibility by people, and data verification. In their second wave of recruitment, they confirmed eligibility in a Zoom interview with the camera on before inviting people to enroll. The P-FROST recommendations of Mistry et al. (2026) also place countermeasures in four areas: study set-up, monetary incentives and recruitment, data collection (screening and interviews), and analysis and reporting.

For the screener itself, NN/g gives concrete techniques. To exclude certain occupations, ask an open question such as “What is your occupation?” instead of showing the list of excluded occupations (Whitenton 2019). Foil options, such as the name of a product that does not exist, mixed into the answer lists help identify people who answer without attention or misrepresent themselves, and they tend to work better than whole foil questions (Banawa 2025). A Japanese research firm checks whether candidates have taken part, within the last year, in research for the same client or the same brand, so as to avoid biased answers (Intage 2026). Whether people will talk can also be checked in advance. Portigal says that the recruiting process sometimes includes an “articulation screen” that screens out people who cannot answer a simple question from the recruiter, and that a quick phone call before the interview, in which participants can ask questions, can build rapport.

Recruitment should also prepare for people who do not show up. NN/g lists three ways: one backup participant for each slot, one for every two slots, or more slots than needed (Pernice 2020). In a report in which User Interviews aggregated nearly 20,000 projects recruited through the company, unmoderated studies with the most common incentive amount (60 dollars per hour) had a no-show rate of nearly 30%. The lowest rate among B2B unmoderated studies was 2.3%, at 110 dollars per hour. These are observed values from the company’s customers and do not establish that a higher incentive causes fewer no-shows. Recommending higher incentives is also in line with the company’s revenue.

GOV.UK asks teams not to use their own staff for research into the public-facing parts of a service. Staff knowledge and experience can mean that they use the service in a very different way. For disabled participants and participants who use assistive technology, GOV.UK advises allowing up to a month to find participants.

  • Wrote down the inclusion and exclusion criteria, the number of participants, the sampling strategy, and the recruitment channel (Robinson 2014 abstract)
  • Asked about occupations to be excluded with an open question in the screener (NN/g)
  • Mixed foil options into the screener’s answer lists (NN/g)
  • For online recruitment, decided on checks for fraudulent applicants (verifying contact details, checking timing and inconsistencies in answers, and if needed a pre-interview with the camera on) (Jean Louis and Thompson 2025, Sharma et al. 2024)
  • Wrote the conditions and method of the incentive into the terms at recruitment (Jean Louis and Thompson 2025, MRS rule 26)
  • Checked recent participation in research for the same client, the same brand, or competitors (Intage)
  • Did not use staff of the organization as participants for a public-facing service (GOV.UK)
  • Prepared backup participants or backup slots, and decided what to do in case of a no-show (NN/g)
  • When including disabled participants, allowed up to a month for recruitment and budgeted for companions or interpreters (GOV.UK)
  • When using an external recruiter, gave written instructions on date and time, place, eligibility and exclusion criteria, quotas, and numbers, and set out rights and responsibilities in a written contract (JMRA, MRS guidance for recruiters)
  • Where needed, asked a short question at recruitment to check whether people will talk (Portigal)

What the consent procedure needs is a record that later shows that this person agreed to this. GOV.UK requires teams to keep, for each participant, evidence of the consent obtained and what the participant consented to. For remote sessions, the consent form can be sent by email or post and the reply confirming consent kept, or verbal consent can be obtained on the recording after the participant has read the form, and that part of the recording kept. The record of consent should be matched to the data it covers, for example by the date collected, the round number, and the participant number.

There are also rules about when to inform participants. The MRS Code of Conduct, a UK industry code, requires that participants are informed about any recording, monitoring, or observation at recruitment and at the beginning of data collection (rule 29). The code of the Japan Marketing Research Association (JMRA) requires researchers to explain the research body, purpose, method, purposes of use of personal information, and that providing personal information is voluntary (article 12). The same article requires researchers to guarantee that participants can withdraw at any time, and to delete the data of those who withdraw on request. Separating consent to recording from consent to participation lets participants choose. NN/g recommends, instead of a blanket “I consent”, separate checkboxes for participation, audio recording, video recording, and publication of data (Fessenden 2022). The template of the U.S. government’s digital.gov also asks for approval of recording in a separate field, states that the full recordings will be deleted when the project is completed, and states that nothing shared will include the participant’s face or name.

The consent form should also be readable. The consent-form text provided by the review boards of 114 U.S. medical schools required, on average, a reading level of grade 10.6, and exceeded the boards’ own stated standards by 2.8 grade levels (Paasche-Orlow et al. 2003, abstract). This is a result for medical consent forms, but writers producing text harder than their own standard can also occur in information sheets for interviews. Kaiser (2009) argues that by considering the audience for one’s research and by reenvisioning the informed consent process, researchers can avoid confidentiality dilemmas that might otherwise lead them not to report rich, detailed data.

Children require parental consent. Japan’s Personal Information Protection Commission states that the specific age at which consent from a legal representative is required should be judged case by case, and that in general it is required for children aged 12 to 15 years or younger (FAQ Q1-62). The JMRA code also requires prior consent from a parent or a person with parental authority when the participant is a child (article 17).

  • Wrote in the information sheet the research body, purpose, method, duration, purposes of use of personal information, that participation is voluntary, and that participants can stop at any time (JMRA article 12, GOV.UK)
  • Arranged to announce recording, video, and observers both at recruitment and at the start (MRS rule 29)
  • Made a consent form in which participation, audio recording, video recording, and publication of quotes can be chosen separately (NN/g, digital.gov)
  • If observers may know participants, told participants before the start and gave them the chance to decline (MRS rule 41)
  • Wrote the retention period of the data (or how it is decided) and how to request deletion (MRS rule 47, GOV.UK)
  • Decided how to keep evidence of consent (signature, reply email, the relevant part of the recording) by participant number (GOV.UK)
  • Checked that the wording of the consent form is readable for the target participants (Paasche-Orlow et al. 2003 abstract)
  • Decided the scope promised in consent with the readers of the report and any publication in mind (Kaiser 2009)
  • For children, decided how to obtain consent from a parent or guardian (Personal Information Protection Commission Q1-62, JMRA article 17)
  • Had the wording checked by a legal or data protection specialist (GOV.UK, practitioner account from Sansan)
  • If recording or photographing, put the consent form or confidentiality agreement among the items to bring on the day (This Is Service Design Doing, “In-Depth Interview”)

When, in what form, and how much should the incentive be?

When the incentive is given changes the pressure that participants feel about how to answer. GOV.UK recommends giving the incentive at the start of the session. This avoids any suggestion that participants have to respond in a particular way to get it, and means that a problematic session can be ended without having to deal with the incentive. NN/g also recommends stating in the consent form that compensation does not depend on the participant’s abilities or on positive feedback (Fessenden 2022).

There is no public standard for the amount. The guidance of the UK Department for Education states that there is no standard monetary incentive value and asks researchers to offer the minimum amount necessary to encourage participation. GOV.UK also says that the amount depends on the type of participant and the length of the session, and gives no figure. The amount that a Japanese research firm shows to participants is around 8,000 yen for one to two hours (Intage’s information page for participants). This is one firm’s information and is not a measurement of the market rate. In HCI research, too, how compensation is decided and reported varies widely between papers (Pater et al. 2021, abstract).

There are also rules about what the incentive may be. The MRS code prohibits using client goods or services, or vouchers to purchase them, as incentives in projects conducted for research purposes (rule 25). Using the client’s own products as incentives makes the research hard to distinguish from sales promotion. In the 2023 survey by User Interviews, the most common type of incentive was gift cards (66%). The sample came from the company’s audience, and the answers on incentive types are not mutually exclusive.

When paying individuals in Japan, the tax treatment should also be checked. Japan’s National Tax Agency states that payments made under names such as an honorarium are subject to withholding tax if they are in substance the same as fees such as manuscript fees or lecture fees (Tax Answer No.2792). No official statement naming interview incentives was found for the income category of the recipient (occasional income or miscellaneous income). Occasional income is defined as one-time income that does not have the character of consideration for labor or services (No.1490). Whether talking in an interview falls under this depends on the substance, so for continuous payments or large amounts, check with a tax accountant or the tax office.

  • Agreed with the client on the amount of the incentive and how it is decided (time, burden, type of participant) (DfE, GOV.UK)
  • Decided to give the incentive at the start, or, if paying later for remote sessions, arranged to announce the timing and method of payment in advance and on the day (GOV.UK, Pop Insight)
  • Stated in the information sheet or consent form that the incentive does not depend on the content of the answers (NN/g)
  • Did not use the client’s products, or vouchers to buy them, as incentives (MRS rule 25)
  • Decided how to handle payment for people who stop partway (the digital.gov template states that participants still receive compensation if they end the session early)
  • Decided how to keep a record of the incentive (to whom, what, when)
  • Checked withholding tax and the recipient’s income treatment (National Tax Agency No.2792, No.1490)

What should the interview guide contain, and how should it be tried?

An interview guide (discussion guide) is the script that sets out the flow and questions of the session. GOV.UK asks that the guide include an introduction script (who you are, what the research is, recording), starter questions and possible follow-up questions for each topic, and a checklist for preparing on the day. During the interview, the interviewer should focus on stories and real examples, and avoid generalities and talk about how things “should” happen. Follow-up questions can take simple forms such as “You said… when, why, who was that?” and “Can you tell me more about…?” The reasons for asking about specific past events instead of opinions or predictions are summarized in Conducting User Interviews: What Academia and Industry Share and Where They Diverge (the limits of self-report and how strongly the interviewer shifts the answers).

Academic work also gives procedures for developing a guide. From a systematic review of ten methodological papers, Kallio et al. (2016, abstract) present five phases: identifying the prerequisites, retrieving and using previous knowledge, formulating a preliminary guide, pilot testing, and presenting the complete guide. The framework of Castillo-Montoya (2016, abstract) has four phases: aligning the interview questions with the research questions, constructing an inquiry-based conversation, receiving feedback on the protocol, and piloting it. Both include piloting within the procedure.

The pilot should not be placed just before the real sessions. NN/g recommends piloting with people who match the target profile, at least 1 day before the study, and assuming that there will be changes (Schade 2015). The pilot is a dress rehearsal for checking that materials are printed, consent forms copied, payment prepared, and the site ready, and it also gives an estimate of how long tasks take. For unmoderated studies, all communications, from recruiting information to confirmation emails, should be piloted. The practitioner Ronsen schedules pilots at least two days before the actual sessions, and, if suitable people are not available, over-recruits and treats the first one or two sessions as pilots. A practitioner at Raccoon Holdings planned one hour per person and marked the questions they most wanted to ask to set priorities. When testing the guide with colleagues, they found more problems than expected, such as questions whose intent did not get across the first time and later questions that did not fit in the time. GOV.UK also asks teams to test the guide by interviewing a colleague, to revise unclear questions, and to reorder topics if the interview does not flow well. Pilot studies are common in qualitative research, but few publications report what was learned from them (Malmqvist et al. 2019, abstract).

Whether to interview in person or remotely should also be decided before the guide is written. In a study comparing face-to-face and Zoom interviews, rapport and empathy were rated highly in both conditions, but significantly higher face-to-face for a subset of the sample, and there were no differences in conversational involvement or data quality (Anthony et al. 2025, abstract).

  • Wrote the introduction script word for word (self-introduction, purpose of the research, explanation of recording, that participants may stop) (GOV.UK)
  • Wrote starter and follow-up questions for each topic, with time and priority for each topic (GOV.UK, Raccoon)
  • Phrased questions to ask about specific past events (for example, “Tell me about the last time you…” instead of “What do you usually do?”) (GOV.UK, Conducting User Interviews: What Academia and Industry Share and Where They Diverge)
  • Prepared follow-up questions to return to specific events when the talk drifts into generalities or how things “should” be (GOV.UK)
  • Matched each question to a research question (Castillo-Montoya 2016 abstract)
  • Had someone else read the guide and give feedback (Castillo-Montoya 2016 abstract)
  • Ran a pilot with someone matching the profile, or with a colleague, at least one to two days before the real sessions (NN/g, Ronsen, GOV.UK)
  • In the pilot, ran through materials, consent forms, the incentive, screens, and timing (NN/g)
  • For emotionally sensitive topics, included phrasing that asks permission (“May I ask about…?”) and reminders that participants can stop at any time (GOV.UK, “Researching emotionally sensitive subjects”)
  • Gave the client an opportunity to review the guide (JMRA)

On the day, who records what?

Roles should be divided in advance. GOV.UK recommends inviting an observer who can also act as a note-taker, so that the interviewer can focus on the participant. A practitioner at Raccoon divided the roles into one interviewer, two note-takers, and one listener, and writes that two people taking notes reduced omissions. Goodpatch says that it is ideal to have two roles, one transcribing what users say and one noting insights that will help the analysis. There is also an opposing position. Indi Young asks that no notes be taken at all in her listening sessions, so that the listener can be fully present, and that the session be recorded and transcribed. When the purpose differs, the method of recording also changes.

Recording is the default, but there are exceptions. NN/g recommends recording the interview when possible and having it transcribed, which helps avoid misquotes and gives the team access to the same raw data (Fessenden 2021). Rutakumwa et al. (2020), on the other hand, compared audio-recorded transcripts with interview scripts written directly after the interview by experienced interviewers, and report that the data were comparable in the detail captured. The difference was in structure: the scripts grouped topics and ideas, whereas the transcripts followed the more scattered order of the conversation. The authors begin by describing circumstances in which a recorder is not, or may not be, possible, and suggest that in some circumstances not recording is the best approach, not the “second best”.

Automated transcripts should not be used as evidence as they are. Koenecke et al. (2024) showed that roughly 1% of transcriptions by OpenAI’s Whisper contained entire phrases or sentences that did not exist anywhere in the audio. Of the hallucinated content, 38% included explicit harms such as perpetuating violence, making up inaccurate associations, or implying false authority. Hallucinations occurred when no one was speaking, and disproportionately for speakers with aphasia, who have longer shares of non-vocal durations. Transcription is not a neutral task either. Whether to keep stutters and accents or to correct grammar changes how participants are understood (Oliver et al. 2005).

Remote sessions should be prepared for failed recordings and for personal information appearing on screen. The guidance of Pop Insight recommends recording with several tools at once, taking “a backup of the backup”, and checking in advance that bookmarks or file names on the participant’s screen do not show personal information. Ronsen recommends setting up video calls so that participant names are not recorded, which avoids having to remove them later when creating highlight reels.

Observers tend to draw conclusions quickly from a single observation. NN/g describes having 4 to 10 note-takers per session, including stakeholders, and holding a short team debrief after each session to gather key findings (Farrell 2017, “Group Notetaking”). A team working on a funding service at the UK Department for Education, using virtual observer rooms, ran daily debriefs and prefaced each one with a reminder that observations were individual pieces of evidence that could not by themselves inform recommendations.

  • Decided the roles and numbers of interviewer, note-takers, and observers, and told everyone (GOV.UK, Raccoon)
  • Decided whether to record, and if not how to record what was said, according to the topic and participants (NN/g, Rutakumwa et al. 2020)
  • If not recording, put time to write notes directly after each interview into the schedule (Rutakumwa et al. 2020)
  • Recorded audio and video with several tools, and arranged to check before starting that recording is running (Pop Insight)
  • For remote sessions, completed the connection test, screen-sharing practice, and checks for personal information on screen by the day before (Pop Insight)
  • Decided a file naming rule (participant number, date and time, attributes) and set up recordings so that real names are not kept (Ronsen, GOV.UK)
  • Told observers in writing not to interrupt, to note what they see and hear, and not to disclose participants’ personal information (GOV.UK, JMRA)
  • Agreed that any observer who knows a participant will stop observing (JMRA request on observation)
  • Left time between sessions for breaks and debriefs (NN/g)
  • For automated transcripts, set a procedure to check quoted passages against the recording (Koenecke et al. 2024)
  • Decided whether transcripts keep stutters and accents or tidy them (Oliver et al. 2005)

How long should data be kept, and how should it be deleted?

The retention period should be decided before starting and communicated to participants. The MRS code requires that the length of time, or criteria, for retaining personal data is clearly communicated to participants, subcontractors, and clients (rule 47). GOV.UK notes that broad findings can be valuable for a long time but that most raw data loses its value quite quickly, suggests 2 or 3 years as reasonable retention periods, and advises deleting the data sooner if it is no longer needed. Participants’ contact details should be deleted as soon as they are no longer needed to manage participation. The guidance of the UK Department for Education states that records of consent must be deleted no later than 2 years after they were created, and that they should be the last personal data deleted, because they are the evidence for holding other personal data.

Japanese law and industry codes also require deletion of data whose purpose has been fulfilled. The Act on the Protection of Personal Information requires business operators to endeavor to delete personal data without delay when it is no longer needed (Personal Information Protection Commission, General Rules Guidelines 3-4-1). The JMRA code likewise requires researchers to endeavor to delete or anonymize, without delay, personal information that there is no longer a reasonable reason to hold (article 4). Using external services for recruitment, transcription, or storage of recordings can count as entrusting the handling of personal data. The entrusting party is obliged to supervise the entrusted party: to select it after confirming its level of security measures, to include measures in the contract, and to understand how the data is being handled (General Rules Guidelines 3-4-4).

Anonymization is not achieved by removing names alone. Saunders et al. (2015), working with interviews with family members of people in vegetative and minimally conscious states, show how they developed context-sensitive strategies to preserve the richness of the interview material wherever possible while protecting participants. If data is preserved and reused, the consent agreement acts as the “contract” that governs how the data may be used (Corti et al. 2000).

  • Decided the retention period and the time of deletion, and wrote them into the consent form (MRS rule 47, GOV.UK)
  • Kept recordings, notes, and consent records only in storage approved by the organization, and deleted copies on personal devices (GOV.UK)
  • Included the date, round, and participant number in file names, so that one person’s data can be found and deleted (GOV.UK)
  • Planned to delete participants’ contact details once participation management is over (GOV.UK)
  • Concluded contracts covering handling and deletion with external recruiters, transcription, and recording storage services (Personal Information Protection Commission General Rules 3-4-4, MRS)
  • Set up a contact point for participants’ requests for deletion or correction (JMRA article 12, JMRA guideline 2.3)
  • Decided the anonymization policy (how to change names, employers, places, and unusual events) (Saunders et al. 2015)
  • Arranged to delete data when the period ends, with consent records deleted last (DfE)

When, and with whom, should the analysis be done?

Analysis should start soon after the interviews. GOV.UK recommends inviting everyone who observed to the analysis session, doing analysis while it is still fresh in people’s minds, and spending 1 hour analyzing every 2 hours of research. Each sticky note should carry one observation and record exactly what was seen or heard (verbatim quotes or observed behavior), not what people think it means. McMahon and Winch (2018) define the debriefing, a thorough, goal-oriented discussion of data immediately after it is collected, as a step of analysis. The service design book also recommends writing up individual key learnings right after the interview and comparing them within the team (This Is Service Design Doing, “In-Depth Interview”). A practitioner at Mercari shares impressions with the PM right after each session and posts a summary in the company chat for members who could not take part.

Faster analysis does not necessarily produce findings far from those of in-depth analysis. In an evaluation in the Veterans Health Administration, findings from a framework-guided rapid analysis and an in-depth thematic analysis were consistent, and the rapid analysis was less resource-intensive (Gale et al. 2019). In a comparison in maternity care research, the findings of rapid analysis overlapped with 79% of the findings of thematic analysis. Data management was faster (43 hours versus 116.5 hours), but interpretation and write-up took much longer (52 hours versus 8 hours), giving a modest overall saving (100 hours versus 126.5 hours) (Taylor et al. 2018). The authors write that the excessive time for interpretation in rapid analysis may be due to differences between the research teams.

A Japanese industry code requires that the analysis can go back to the original records. The JMRA guideline requires that analysis use the original records of the interview, whether tape or shorthand, and that there be documents showing that these records were used in interpreting the data. If AI is used in the analysis, its effects and the necessary technical information should be explained to the client as far as possible (JMRA code article 22).

  • Set time right after each session for the interviewer and observers to debrief (McMahon and Winch 2018, NN/g)
  • Put analysis time into the schedule, aiming for 1 hour per 2 hours of research (GOV.UK)
  • Invited those who observed to the analysis (GOV.UK)
  • Wrote what was seen and heard separately from interpretations on sticky notes and in notes (GOV.UK)
  • Added participant numbers and times so that findings can be traced back to the original remarks and records (JMRA)
  • Chose between rapid and in-depth analysis according to the deadline of the decision (Gale et al. 2019, Taylor et al. 2018)
  • If using AI for transcription or analysis, prepared to explain this, and the scope that people checked, to the client (JMRA article 22)

What should be given to the client, and in what form?

A report should separate facts from interpretations. The MRS code requires that outputs and presentations clearly distinguish between facts, opinion, and interpretation (rule 60). The JMRA code requires researchers to ensure that findings and their interpretation are clearly supported by the data, to distinguish insights and recommendations from findings as facts, and to explain how they were derived (article 7). This distinction prevents one participant’s strong remark from taking on a life of its own inside the client organization as a trend across all users.

Practitioner sources agree on what to include in a report. The service design book says to include the research process and what was done to ensure data quality, the key findings, and raw data supporting the findings (such as quotes), and writes that including raw data increases the credibility of the research (Stickdorn et al., This Is Service Design Doing method pages). The guidance of the New Jersey state UX team asks for the purpose, methods (who was spoken to and how), themes supported by anonymized quotes, recommendations, and limitations (sample size, scope, and constraints). The COREQ reporting criteria for qualitative research gather the reporting items for papers based on interviews and focus groups into 32 items (Tong et al. 2007, abstract). Braun and Clarke (2025, abstract), however, criticize the application of one-size-fits-all checklists for introducing criteria that do not fit the method. A report to a client need not follow the criteria for papers as they are, but stating the number of participants, how they were selected, how data was recorded, and the limitations is common to both.

A report has more than one reader. NN/g recommends structuring a report for three audiences (Krause 2022). These are headlines for decision makers who are not involved in day-to-day work, big themes for product managers, and in-depth details for data-driven team members such as researchers, designers, and developers. GOV.UK recommends sharing findings as soon as possible and gives a simple slide deck of 1 or 2 slides on the research, 5 to 10 slides on findings, and 1 or 2 slides on what comes next, which is easy to talk through at a show and tell. Besides handing over a report, teams can also work through the findings with stakeholders. NN/g recommends combining three types of workshop, research alignment, empathy, and research application, according to the state of the audience (Kaplan 2025).

There are limits to what data may be handed over. The ICC/ESOMAR International Code prohibits sharing or transferring personal data to a client unless the data subject has consented to the specific purpose and agreed to the transfer (article 6). The JMRA code also limits disclosure of personal information to third parties, including clients, to cases where the person has consented after the purpose has been shown in advance or where a law provides a basis (article 16). If the client intends to publish the results, the researcher should ask the client to consult in advance on the form and content (article 8).

  • Included in the report the purpose of the research, who was asked and how (number, selection, method, dates), findings, supporting quotes, recommendations, and limitations (This Is Service Design Doing, New Jersey state UX team)
  • Wrote facts (observations and remarks), opinions, interpretations, and recommendations separately, and gave the reasoning behind each recommendation (MRS rule 60, JMRA article 7)
  • Distinguished remarks by a single participant from findings shared across several participants (MRS rule 60)
  • Structured the report in three layers: headlines for decision makers, themes for those in charge, and details for practitioners (NN/g)
  • Shared a first summary as soon as possible (GOV.UK, Mercari)
  • Held a report session or a joint reading, and set aside time to move findings into the next decisions (NN/g)
  • Anonymized quotes, photos, and video excerpts, and did not hand them over beyond the scope of consent (ICC/ESOMAR article 6, JMRA article 16)
  • Decided, within the scope of consent, whether to give or show original records (recordings, verbatim transcripts) to the client (JMRA guideline 2.4)
  • Wrote down the steps in which AI was used and the scope that people checked (JMRA article 22)
  • Obtained the client’s promise to consult in advance before publishing results outside the organization (JMRA article 8)

What should be added when a screen operation check is included?

An interview may include a step in which participants operate a prototype or app and the team observes how they do it. In that case, preparation for listening must be supplemented with preparation of the thing to be operated, tasks, records, and evaluation. How to choose a format when a group operates the screen is covered in A Checklist for Group Interviews: What to Prepare, What to Ask, and How to Report to the Client.

The thing to be operated should be prepared so that the session does not stop when it breaks. GDS practitioners advise against Lorem Ipsum and funny or lazy names in prototypes that users will see, suggest starting to use real data as the prototype is refined, and suggest keeping alternative versions to go back to when a route does not work (Lanman et al. 2015). NN/g writes that an interactive prototype needs a response set for each possible user action before the test, while with a static prototype a person familiar with the design gives the responses in real time, for example in Wizard of Oz testing (Pernice 2016).

Data and accounts should be planned so that participants do not have to use their own information. NN/g notes that qualitative studies can benefit from participants’ own information, recommends fake but realistic information as an alternative, and asks that personal information be blurred in highlight videos and in screenshots in reports (Schade 2016). MeasuringU notes that it is difficult, and often prohibited, to ask users to use their own information, and lists alternatives such as stopping before the purchase and providing prepaid cards (Sauro 2014). The UK firm dxw, testing a service that handles sensitive personal data, risked showing private data to the wrong people when only real data were available, and generated fake but realistic data sets instead. GOV.UK recommends asking participants to use their own data and documents whenever possible.

Tasks should be written so that they do not give away the answer. NN/g recommends task scenarios that are realistic, encourage an action, and do not give away how the interface should be used (McCloskey 2014). A reanalysis of nine commercial teams found no significant correlation between the percentage of problems found and the number of test users, but a significant correlation with the number of tasks used (Lindgaard and Chattratichart 2007, abstract). A researcher at note inc. writes that she always runs a rehearsal, walking through the tasks herself, once the tasks are designed (Senda 2022).

Thinking aloud needs an explanation, practice, and a decision on how to intervene. NN/g notes that most users need prompting, that a short demo video can show what is expected, and that prompts and clarifying questions can bias behavior (Nielsen 2012). Boren and Ramey (2000, abstract) pointed out that thinking aloud as practiced in usability work does not conform to its usual theoretical basis (Ericsson and Simon). In a meta-analysis of 29 studies comparing thinking aloud during and after use, Hertzum (2024, abstract) found that concurrent thinking aloud makes task time longer but total time shorter, while retrospective thinking aloud produces more explanations, problem formulations, and design recommendations.

Screen recordings and personal information need more care than recordings of talk. GOV.UK writes that screenshots from usability tests used outside the research should either contain no personal data or have any personal data blurred. The U.S. GSA guide tells the facilitator to ask participants, before they share their screen, to close anything they do not want recorded. The guidelines of Japan’s Personal Information Protection Commission give video in which a person can be identified, and audio recordings that identify a person because they contain a name, as examples of personal information. For unreleased products, a non-disclosure agreement should also be prepared. The NIST template includes a non-disclosure agreement as a discretionary appendix (NISTIR 7742).

Remote operation checks should separate the technology check from the session. Researchers at the U.S. Census Bureau report that when they emailed instructions for downloading and installing software, a participant withdrew because it seemed too complicated, so they changed the email to announce a pre-meeting “tech check” that would step participants through the process (Olmsted-Hawala and Nichols 2021). The same report says that they embedded consent, satisfaction, and debriefing into one URL, so that participants did not have to open their email while being recorded. NN/g recommends, where possible, a 15-minute online practice meeting with each participant the day before the session (Moran and Pernice 2020). An adult social care team in Wales found that video was a barrier to inclusion for people less familiar with technology, and switched to telephone sessions. Participants who use assistive technology should be asked to bring their own, because it is hard to recreate someone’s personal setup (GOV.UK).

Observers affect participants particularly in operation checks. The presence of non-interactive observers led to a physiological stress response and lowered some performance measures (Sonderegger and Sauer 2009, abstract). A team at Tabelog shares its Zoom session with colleagues through a separate Teams stream, so that the participant does not face a large audience (Kusaka 2023).

Numbers and evaluation depend on the purpose. The rule that five users find about 85% of problems assumes problems that affect 31% of users, and applies only to discovering problems, not to comparing products or estimating completion rates (Sauro 2010, Nielsen 2000). Problem severity is recommended to be rated from frequency, impact, and persistence, with each evaluator rating independently (Nielsen 1994). Even with the same method and the same interface, the average agreement between two evaluators ranges only from 5% to 65% (Hertzum and Jacobsen 2001, abstract). The NIST template lists the number of tasks completed successfully within the allotted time without assistance as one of the data to collect (NISTIR 7742).

  • Made the wording and names in the prototype realistic, and kept alternative versions to fall back on (Lanman et al. 2015)
  • Decided the prototype’s responses (clickable areas, responses given by a person) and walked through it (Pernice 2016, Senda 2022)
  • Prepared test accounts and dummy data (realistic names and values), and decided how to avoid using participants’ own payment or personal details (Schade 2016, Sauro 2014, dxw)
  • Wrote tasks set in realistic scenarios that do not give away the answer, and set success criteria and time limits (McCloskey 2014, NISTIR 7742)
  • Put an explanation of thinking aloud and practice time into the script, and decided how to intervene (Nielsen 2012)
  • Made screen recording a separate item on the consent form, and arranged for participants to close notifications and personal files before sharing their screen (digital.gov, GOV.UK)
  • Decided how to blur personal information on screen in reports and highlight videos (GOV.UK, Schade 2016)
  • For unreleased products, prepared a non-disclosure agreement (NISTIR 7742)
  • For remote sessions, set up a separate short technology check and a fallback to the telephone (Olmsted-Hawala and Nichols 2021, Moran and Pernice 2020, the Welsh adult social care team)
  • Arranged for users of assistive technology to take part with their own equipment and settings (GOV.UK)
  • Kept observers out of the participant’s session and showed it to them through a separate channel (Kusaka 2023)
  • Set the number of participants according to whether the purpose is to find problems or to measure values (Sauro 2010)
  • Decided how to rate problem severity, with several people rating independently (Nielsen 1994, Hertzum and Jacobsen 2001 abstract)

How strong is the evidence for this list?

The evidence behind the items is uneven. The items checked with numbers are fraud in online recruitment (Pozzar et al., Jean Louis and Thompson, Sharma et al.), the comparison between recording and notes written directly after the interview (Rutakumwa et al.), and errors in automated transcription (Koenecke et al.). The comparison between rapid and in-depth analysis (Gale et al., Taylor et al.) and the readability of consent forms (Paasche-Orlow et al.) are also checked with numbers. All of these come from health or social science research, and whether the same magnitudes hold for design interviews has not been tested. Much of the guidance on the timing of incentives, the handling of observers, and the form of reports consists of norms based on guidance documents, industry codes, and practitioner experience, and their effects have not been measured. The figures on incentive amounts and no-shows are observed values from a research firm that aggregated its own customers, and recommending higher incentives is also in line with that firm’s revenue. Among the items for operation checks, thinking aloud, evaluator variation, the number of tasks, and the effect of observers rest on empirical results, while the preparation of prototypes and data, remote arrangements, and the handling of on-screen information rest on guidance and practitioner experience.

Some things were not found in this investigation. No study was found that tested how to write screener questions or how to deal with no-shows in design interviews. No survey with a stated sample and period was found on the market rate of incentives for interviews in Japan. No study was found that measured whether the form of a report to the client (length, layers, setting) changes how the findings are used.

Unverified items

No unverified statement remains in the main claims. The following academic works could not be obtained in full text by three or more routes (publishers refused machine access), and were used only within the scope of their abstracts: Anthony et al. (2025), Boren and Ramey (2000), Braun and Clarke (2025), Castillo-Montoya (2016), Hertzum (2024), Hertzum and Jacobsen (2001), Kallio et al. (2016), Lindgaard and Chattratichart (2007), Malmqvist et al. (2019), Paasche-Orlow et al. (2003), Pater et al. (2021), Robinson (2014), Sonderegger and Sauer (2009), and Tong et al. (2007). For Mistry et al. (2026), the full text was obtained, but only the division of the recommendations into four areas was used.

References

All accessed 2026-11-03. The ledger and verification records are in papers.md and industry.md under source/review/user-interview-checklist/.

Academic

  • Anthony, Kathryn E., Michelle Miller-Day, et al. 2025. “Is There Really a Difference? A Comparison of In-Person and Online Qualitative Interviews.” International Journal of Qualitative Methods 24. https://doi.org/10.1177/16094069251349580
  • Braun, Virginia, and Victoria Clarke. 2025. “Reporting Guidelines for Qualitative Research: A Values-Based Approach.” Qualitative Research in Psychology 22 (2): 399–438. https://doi.org/10.1080/14780887.2024.2382244
  • Castillo-Montoya, Milagros. 2016. “Preparing for Interview Research: The Interview Protocol Refinement Framework.” The Qualitative Report 21 (5): 811–831. https://doi.org/10.46743/2160-3715/2016.2337
  • Corti, Louise, Annette Day, and Gill Backhouse. 2000. “Confidentiality and Informed Consent: Issues for Consideration in the Preservation of and Provision of Access to Qualitative Data Archives.” Forum Qualitative Sozialforschung 1 (3). https://doi.org/10.17169/fqs-1.3.1024
  • Gale, Randall C., Justina Wu, Taryn Erhardt, Mark Bounthavong, et al. 2019. “Comparison of Rapid vs In-Depth Qualitative Analytic Methods from a Process Evaluation of Academic Detailing in the Veterans Health Administration.” Implementation Science 14: 11. https://doi.org/10.1186/s13012-019-0853-y
  • Jean Louis, Jose, and Tiffany Thompson. 2025. “Bots and Fake Participants: Ensuring Valid and Reliable Data Collection Using Online Participant Recruitment Methods.” International Journal of Social Research Methodology 28 (4): 463–473. https://doi.org/10.1080/13645579.2024.2410176
  • Kaiser, Karen. 2009. “Protecting Respondent Confidentiality in Qualitative Research.” Qualitative Health Research 19 (11): 1632–1641. https://doi.org/10.1177/1049732309350879
  • Kallio, Hanna, Anna-Maija Pietilä, Martin Johnson, and Mari Kangasniemi. 2016. “Systematic Methodological Review: Developing a Framework for a Qualitative Semi-Structured Interview Guide.” Journal of Advanced Nursing 72 (12): 2954–2965. https://doi.org/10.1111/jan.13031
  • Koenecke, Allison, Anna Seo Gyeong Choi, Katelyn X. Mei, Hilke Schellmann, and Mona Sloane. 2024. “Careless Whisper: Speech-to-Text Hallucination Harms.” Proceedings of FAccT 2024: 1672–1681. https://doi.org/10.1145/3630106.3658996
  • Malmqvist, Johan, Kristina Hellberg, Gunvie Möllås, Richard Rose, and Michael Shevlin. 2019. “Conducting the Pilot Study: A Neglected Part of the Research Process? Methodological Findings Supporting the Importance of Piloting in Qualitative Research Studies.” International Journal of Qualitative Methods 18. https://doi.org/10.1177/1609406919878341
  • McMahon, Shannon A., and Peter J. Winch. 2018. “Systematic Debriefing After Qualitative Encounters: An Essential Analysis Step in Applied Qualitative Research.” BMJ Global Health 3 (5): e000837. https://doi.org/10.1136/bmjgh-2018-000837
  • Mistry, Kamal, et al. 2026. “Fraudulent Participation in Online Qualitative Studies: Practical Recommendations on an Emerging Phenomenon.” Qualitative Health Research 36 (7): 647–657. https://doi.org/10.1177/10497323241288181
  • Oliver, Daniel G., Julianne M. Serovich, and Tina L. Mason. 2005. “Constraints and Opportunities with Interview Transcription: Towards Reflection in Qualitative Research.” Social Forces 84 (2): 1273–1289. https://doi.org/10.1353/sof.2006.0023
  • Paasche-Orlow, Michael K., Holly A. Taylor, and Frederick L. Brancati. 2003. “Readability Standards for Informed-Consent Forms as Compared with Actual Readability.” New England Journal of Medicine 348 (8): 721–726. https://doi.org/10.1056/NEJMsa021212
  • Palinkas, Lawrence A., Sarah M. Horwitz, Carla A. Green, Jennifer P. Wisdom, Naihua Duan, and Kimberly Hoagwood. 2015. “Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research.” Administration and Policy in Mental Health and Mental Health Services Research 42 (5): 533–544. https://doi.org/10.1007/s10488-013-0528-y
  • Pater, Jessica, Amanda Coupe, Rachel Pfafman, Chanda Phelan, Tammy Toscos, and Maia Jacobs. 2021. “Standardizing Reporting of Participant Compensation in HCI: A Systematic Literature Review and Recommendations for the Field.” Proceedings of CHI 2021: 1–16. https://doi.org/10.1145/3411764.3445734
  • Pozzar, Rachel, Marilyn J. Hammer, Meghan Underhill-Blazey, et al. 2020. “Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire.” Journal of Medical Internet Research 22 (10): e23021. https://doi.org/10.2196/23021
  • Robinson, Oliver C. 2014. “Sampling in Interview-Based Qualitative Research: A Theoretical and Practical Guide.” Qualitative Research in Psychology 11 (1): 25–41. https://doi.org/10.1080/14780887.2013.801543
  • Rutakumwa, Rwamahe, Joseph Okello Mugisha, Sarah Bernays, et al. 2020. “Conducting In-Depth Interviews with and without Voice Recorders: A Comparative Analysis.” Qualitative Research 20 (5): 565–581. https://doi.org/10.1177/1468794119884806
  • Saunders, Benjamin, Jenny Kitzinger, and Celia Kitzinger. 2015. “Anonymising Interview Data: Challenges and Compromise in Practice.” Qualitative Research 15 (5): 616–632. https://doi.org/10.1177/1468794114550439
  • Sharma, Ashna, et al. 2024. “Navigating the Challenges of Imposter Participants in Online Qualitative Research: Lessons Learned from a Paediatric Health Services Study.” BMC Health Services Research 24: 724. https://doi.org/10.1186/s12913-024-11166-x
  • Taylor, Beck, Catherine Henshall, Sara Kenyon, Ian Litchfield, and Sheila Greenfield. 2018. “Can Rapid Approaches to Qualitative Analysis Deliver Timely, Valid Findings to Clinical Leaders? A Mixed Methods Study Comparing Rapid and Thematic Analysis.” BMJ Open 8 (10): e019993. https://doi.org/10.1136/bmjopen-2017-019993
  • Tong, Allison, Peter Sainsbury, and Jonathan Craig. 2007. “Consolidated Criteria for Reporting Qualitative Research (COREQ): A 32-Item Checklist for Interviews and Focus Groups.” International Journal for Quality in Health Care 19 (6): 349–357. https://doi.org/10.1093/intqhc/mzm042
  • Boren, Ted, and Judith Ramey. 2000. “Thinking Aloud: Reconciling Theory and Practice.” IEEE Transactions on Professional Communication 43 (3): 261–278. https://doi.org/10.1109/47.867942
  • Hertzum, Morten. 2024. “Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic Review.” ACM Transactions on Computer-Human Interaction 31 (3): 1–29. https://doi.org/10.1145/3665327
  • Hertzum, Morten, and Niels Ebbe Jacobsen. 2001. “The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods.” International Journal of Human-Computer Interaction 13 (4): 421–443. https://doi.org/10.1207/s15327590ijhc1304_05
  • Lindgaard, Gitte, and Jarinee Chattratichart. 2007. “Usability Testing: What Have We Overlooked?” Proceedings of CHI 2007: 1415–1424. https://doi.org/10.1145/1240624.1240839
  • Sonderegger, Andreas, and Juergen Sauer. 2009. “The Influence of Laboratory Set-up in Usability Tests: Effects on User Performance, Subjective Ratings and Physiological Measures.” Ergonomics 52 (11): 1350–1361. https://doi.org/10.1080/00140130903067797

Public bodies, guidance on laws, and codes of research ethics

Research firms and consultancies

First-hand accounts by practitioners


Author: Shuichiro Ogawa (Design Researcher / Design Consultant) About me →