When designing a product, the most important goal is to match a user’s needs and wants with an elegant solution that satisfies those needs and wants. A good product satisfies the needs of a user, while a great product not only satisfies the needs of the user but goes beyond and offers such a satisfying experience that she’ll want to tell people about the product.

How does one build a great product that users will love? It starts with a deep understanding of the user – their desires, habits, values, and more. It is only with this understanding of what your user needs can you truly design a product that not only meets the users needs but also delights the user.

For Mason Park, our e-commerce marketplace startup, our users value quality made and artist driven goods. To understand what to prioritize and build we had to listen, observe, and gain a deep understanding of our users. To do this, we turned to user research.

We completed multiple rounds of user research sessions to build Mason Park, and to be honest our initial sessions weren’t that great (and thus our data and conclusions weren’t the most accurate) and we had to learn and improve our methodology each time. For each round of user research, we would source participants and conduct 1:1 sessions to observe and understand needs and reactions to Mason Park. Following the sessions we would summarize the takeaways, combine with our quantitative site metrics, develop new hypotheses and move forward to test.

The components of qualitative user research sessions can be broken down to:

1. Structuring the Research Session
2. Recruiting Users
3. Conducting the Session
4. Interpreting the Data

In this essay I won’t detail the methodology of each step, but rather recap the main takeaways learned throughout each of the components of user research.

Structuring the Research Session

When we first started conducting our research sessions, it became very easy to diverge into an open ended discussion of what users thought about what we were presenting them. While this offered some insight, we quickly learned that proper structure and consistency were key to obtaining the best data possible.

While there are many details that go into structuring an effective research session, here are a few of the major lessons we learned:

Hypothetical Questions or Thought Exercises Are Not Useful

The most obvious approach we took was to ask a participant directly what they thought about ideas we had. For example, we might show them wire frames and ask if they would use this product or what they thought. In reality, people are not that accurate at predicting their behavior or preferences, and when combined with a natural tendency to be nice, the data obtained from thought exercise questions aren’t accurate indicators of actual behavior. Seemingly simple questions such as “Would you use this product?” often gave us data that led us astray.

Seemingly simple questions such as “Would you use this product?” often gave us data that led us astray.

We found the best way to learn from users was to..

Give Users a Task to Perform

Rather than asking users what they thought of concepts, it was much better to have a user walk through a task and observe their behavior and reaction. By observing behavior rather than asking them to predict their behavior, we gained much more useful data.

In addition to observing behavior on our own product, we found it hugely insightful to have users also perform the same task with different and sometimes competing products. This allowed us to observe the reaction to our product, as well gain insight on what worked well or didn’t work well with other products.

Leave Questioning to the End

While observing users, we had a tendency to jump in and ask them “why” they performed a certain action. Asking questions while users were performing tasks snapped them out of their pre-existing mindset and interrupted their behavior flow. After answering our questions, users had to take a few seconds to re-orient themselves and get back to the task at hand. We found it better to hold all questions until after the user has performed the task requested of them.

Recruiting Users

We tried various methods of recruiting users – from reaching out to friends to approaching people in parks and cafes to placing ads on craigslist.

Overall, the most important in recruiting users is to be mindful of who we are recruiting and screening appropriately.

Select the Right Users Not Easy to Reach Users

With our research sessions consisting of a 45 min live session per participant it was easiest to reach out to our personal networks for participants. Personal networks are a completely valid source of users but it’s necessary to make sure that the users are the right audience. At first we focused on the quantity of sessions we could conduct, as it seemed right to get more data.

However, focus on getting the right users, while more difficult, is much more important.

Be Aware of Selection Bias

For one of our research rounds we sourced many of our users through Instagram. While the users we sourced were our target audience, we had to mindful of the fact that they all shared a certain aspect in common: they were heavy users of social media and Instagram, which may not have been representative of our entire audience. For this reason we also found it useful to source users from a variety of methods.

Be Meticulous With Screener Surveys

We had screener surveys to check for the right type of users, but we found that people often tried to provide the “right” answers to our surveys rather than providing accurate answers. We’re now a lot more careful to design screener surveys that not only obscure any “right” answers, but also provide us with the right insights to select relevant users.

Conducting and compensating for a 45min session with a participant who is clearly not the right audience is quite painful. Screen carefully.

Conducting the Session

Most participants want to be helpful and are enthusiastic, and we usually have an easy time chatting with them. This also means that we have to put extra effort into conducting the sessions as effectively as possible.

Here are some of the major takeaways we had for conducting the sessions:

Listen and Resist the Urge to Respond

Users would sometimes comment, ask questions, or get stuck on something, which provoked a response from us. We often felt that we wanted to jump in and explain why we did something a certain way or share our agreement with their enthusiasm. Just listening was even more difficult when the sessions sometimes felt more like dialogues with the users.

However, we found though that everything we said would influence the perspective and thoughts of the user. In order to get the best data, we had to limit our influence and focus on listening and observing.

Keep Users on Track

The energy some users brought to the sessions also meant that they could also get sidetracked. Users sometimes latched onto irrelevant topics and move away from the task at hand or spend excess amount of time giving their opinions on an unrelated area.

While we focused on listening to users, it became equally important to keep users on track and to refocus them as necessary.

Capture the Data While it’s Fresh

With back to back sessions and a focus on listening and observing, it was easy for us to move forward without capturing the data from the session.

In the worst case, this meant we would forget key information presented to us. This also meant that recording a session was invaluable as we invariably missed details as we recalled the sessions later on. We conduct most of our sessions via Google Hangout and record the sessions for review later on.

Making Sense of the Data

After each user research session was completed, we usually had a scratchpad of raw jot notes. Jot notes and first impressions weren’t terribly useful and we had to make a conscious effort to make sense of the data in order to act on it.

Here are some of the main takeaways in making sense of the data:

Decipher What Was Said

Often what people said and what they meant were different things. We found it useful to discuss amongst ourselves the meaning behind anything that was said. Body language, tone, and the user’s desire to be nice were all factors in deciphering what a user really meant vs what they were saying.

Beware of Confirmation Bias

The qualitative nature of user research sessions made it even easier for us to draw conclusions we wanted to see. It’s important to step back and look at what the data as a whole is indicating and not to jump on conclusions that may support an existing belief.

Organize the Responses Into Measurable Data

In order to aggregate and review the data appropriately, we found it useful to score the responses on various dimensions of interest. This allowed us to look at the data in aggregate and also made sure we analyzed the responses from the appropriate angles.  Without this, it was easy for us default to a few high level points from the few most recent sessions instead of a holistic view of the data.

Building a Better Product

User research has been arguably the most important component in shaping Mason Park. It is only through careful user observation and understanding are we able to develop features that serve our users best.

User research and building a better product are both iterative processes. We utilized user research to define our first iteration of Mason Park, and we continue to test and research with every feature and change we make. Similarly, with every round of user research we conduct, we tweak our methodology to help us get better and more useful data.

As we continue to refine our approach and continue to improve in our ability to conduct user research, we’re excited to see how Mason Park will evolve with our audience.

Fortunately, we’re having just as much fun listening to our users as we do in building Mason Park.