User Testing Questions: 3 I Still Ask and 7 I Stopped Asking

Quite frequently, people answering user testing questions are just super polite. Many make up details to hide that they're not very tech-savvy. Many lie about how they use tools.

And when they start something, for example downloading files, and they have a problem with the file format or anything else, they don't Google the solution. They try to use AI to answer those questions. Using AI is a bit problematic for pretty much every user interview, because people rely more on AI than on the product itself. They would love to delegate everything inside the product to their AI, not to the support team.

He was one of the power users of an application I've been designing, and it's very development-focused. It's been trained on development data, and the tool is about building websites on its internal CRM kind of system, but it's basically a CMS.

This person came up with a business idea that he could just develop a service that would print greeting cards, birthday cards, whatever, on demand. He ended up building the website design and everything else with the tool. But the problem was that he would love to have some visuals as well. So he started using it for those too, a tool designed for development only.

My client has a clause in the user agreement that it should be used for development only, not for generating any images and everything else. The client checked the history and saw that a huge number of tokens went through some heavy visual tasks. That's how they found out: just by checking what he had in his history, not from an interview.

I don't think it's wrong, but, I mean, it could have been anything else.

How do I pick ten people? It's quite random, actually, but I try to pick at least three power users, and the rest is just everyone else. One session each, plus follow-up user testing questions.

Why different paying plans? Because if you're paying $9.99 for something, and then there's a user who pays more than $200, they have quite different usage. A person on $9.99 could be just testing the waters, or could be out of budget, or it could be just a hobby. And for someone paying $200, for example, I always assume this is a heavy user, or they have something that requires a plan like this.

What goes wrong with the cheap plan? Nothing goes wrong with the cheap plan. The cheap plan is sometimes the best plan for almost everyone. It all depends on how you use the system.

Where do I find them? Lots of clients share their customer base, at least a chunk of it, and I always ask to see at least 10% of their customers, picked at random.

Many people book and don't show up. Many would say: I have nothing to say.

People who just don't show up are okay. They're fine. Some people would love to be polite and say, okay, yeah, I can do that, and then it's: sorry, I need to rebook, I'm very busy and I don't have time for it.

There's a small chunk of users who ask, can I get something in return? Can you pay me for the interview?

And there's a huge number of people who just don't care. They're using it, it works, it doesn't, they don't care, they're just paying, and after quite some time they could just drop off. That's very natural.

Most users just don't care about the product, about the founder, about the money, about all the rounds, about VCs and all this stuff. They need something for their routine, or they're testing something, or they're open to start using a new tool. It's quite random.

About 20 or 25 minutes. It's a free session. Some people demand some money for a session, or some gift cards, whatever.

At the start, before any user testing questions, I usually say that I will record only their answers, not their voice, not their identity. This is very important, because lots of people are nervous about all this stuff. Oh, do you use AI? Do you record everything, including my face? Yeah, they're nervous. So I record only text, only what they're saying, not their voice, to be compliant with GDPR and everything else.

And if someone gets stuck, I just try to say: okay, hey, if you don't have an answer to one of the user testing questions right now, you can just email me. And if they get stuck a couple of times, I remind them that I would love to have their answers.

The most important user testing questions are these two:

1. What happened right before you opened the app?
2. What did you see the last time it didn't work?

The second of these user testing questions is about errors and input validation. Some people, even when they're using AI tools, would love to understand what's going on, how it should work and why it doesn't. And they always blame the product, never themselves.

What happened right before you opened the app? The answer depends on the app, actually. Some people open something on their phone first, because they saw a notification or an email, and then continue on their desktop. Some people just say: you know what, I received a notification that I need to complete something, and that's how I got to my desktop to do whatever.

3. Why did you choose it over the competitors?

The answer you get to this user testing question is usually the price.

Lots and lots of people have been saying: you know what, I don't use my phone on an everyday basis, because I have some religious rules or anything. They would prefer a desktop version. Those people are looking for something quite simple that just does the job. And in most cases, it's quite surprising that lots of people would love to use their desktop instead of their phone, despite the phone being in their pocket every day.

About vibe coding: there's a percentage of people, and they're not developers. They're people who opened Claude or ChatGPT, and they say in the same line: so I can vibe code this, and I don't understand why I need to pay $25 for something I can build myself in a few days.

Which is not true, because, as we know, web development or web design has nothing to do with "I'll do it in a couple of days". You can just try to solve your problems by yourself, and it actually takes weeks, weeks, weeks, weeks. And there's a product that costs, I don't know, even 50 bucks. If it does the work, I mean, why should you just try to vibe code it? I don't understand it.

Many power users want to do less and less and leave even decision-making to AI. People are getting lazy, and afraid to take responsibility.

They outsource everything to AI. They say something like: okay, fine, I have this mistake, or I have this error, or I would love to do this, could you help me? Lots of people are using those browser plugins for Claude or anything else, and the tool just starts clicking for them. It has nothing to do with their decisions, because the AI will try to solve the problem for them.

That's what I see, and that's one of the reasons I try to persuade my clients to build an MCP server, a direct way in for AI assistants: you shouldn't click around if you're just an AI.

If you're outsourcing an error or a human mistake, that's one thing. But if you're trying to make the decision-making process easier for yourself, you just say: okay, AI, think for me. Make no mistakes.

Not everyone wants that, though. There's a very interesting category of people. They're in their late 40s or 50s, and they grew up using Linux or Windows 95 or 98. They got used to very, very simple interfaces.

There was one person in his early or mid-50s, and he turned the user testing questions around. He asked why the product has a very small AI agent that follows up about things that are incomplete, for example when he has completed 99% of a form and there's only 1% left to finish.

Why do you use AI, he said, when it could be so much simpler? You wouldn't be able to submit a form if it's not finished. Just save the state, and you save yourself time and tokens and everything else. So instead of sending notification emails like "Hey, you've finished 99% of the form, just please click submit", you can just save the state.

And there could be a decision: if you're 99% finished with the form, you can't just go back. You're stuck at the form, so just click the button.

I don't ask these user testing questions anymore:

1. Why they chose this plan instead of something else. Everyone's broke.
2. Whether they would love to change something very design-specific.
3. About colours.
4. What they think about the logotype.
5. What they think about the design system.
6. About button shapes, or anything like that. I don't ask that.
7. How they would rate the application from zero to ten. That's quite stupid.

It's very straightforward. We've all been using AI for quite a while, right? You can't imagine user interviews without AI, for at least two different reasons.

Reason number one: you need to transcribe everything people say into text files. For many reasons, I've been using very simple TXT files with the dialogue, as if it's a movie script.

Reason number two: during some conversations there's lots of emotion coming. Some people are not frustrated. They're just not sure why they've been asked all those complicated user testing questions. They're not complicated questions, but they come from someone with huge expertise. And sometimes it's very hard to make a note of when a person is frustrated about an answer, or frustrated about the product itself.

With the transcripts in those TXT files, I try to get to the bottom of it: what each of them does, what they have in common, how they use the product, how they use the screens, what they've been sharing during each second of the interview, plus some emotion recognition.

After some seven or eight sessions of user testing questions, I have a list of notes for the founders, for the design department if they outsource design, or for myself if I'm just a hired person, a consultant, whatever.

And then there are separate notes for the product manager, because lots and lots of product managers would love to vibe code everything, including design, including some development decisions. Quite often they're completely wrong. 100% wrong. Absolute mismatch. Many of those notes go directly to the product manager, just to give them a heads up: sorry guys, but you're failing. You should rethink why.

The user testing questions I ask:

• What happened right before you opened the app?
• What did you see the last time it didn't work?
• Why did you choose it over the competitors?

The user testing questions I don't ask anymore: why this plan, would you change something design-specific, colours, logotype, design system, button shapes, and "rate us from zero to ten".

These sessions are part of my UX design and SaaS product design work.