In short, we do rounds of testing at the end of each design sprint (or when we have a testable design hypotheses agreed, and packaged and prototyped in a way that we can gather meaningful results from a test).
These hypotheses are tested in person ,or via a Google HO with cameras on as a last resort, with no more than 5 participants per round. There are rapidly diminishing returns in doing more than 5, if the test is properly constructed, and testing is pretty time consuming, so we don’t want to over-egg it.
Testing is always done by members of the team designing and building the product, as it’s really really useful to actually watch people using your software. A joy when it a just clicks for the user, but also great to see your assumptions shot down, and a path appear. Humbling and invigorating!
Recruitment can be burdensome for sure, but we don’t outsource, and we always test on people brand new to the project.
Typically this involves approaching strangers, going through a screening question or two (scripted and determined by what we are aiming to test), testing with them whilst recording so it can be shared with the team, and then offering to buy a coffee as a thank you. Funnily enough, no one has ever taken up the offer of a coffee… most people really love helping out, and just want to know more after the test!
The demographic is actually pretty broad, as we are making core network features that should be usable by large swathes of the population. Typical screening questions are things like “have you heard of Bitcoin?” or “what web browser do you use?”, which allows us to exclude people who aren’t regular internet users, or are so far off the map that they are unlikely to provide helpful results.
Screening, combined with the locations we use to recruit from, such as coffeeshops near university campuses, or financial districts, has given us a pretty good source of participants.
I’ll perhaps go into more detail on what the tests showed, and how they influenced the designs in some of the screencasts after we get v1 of the Safe Network App rolling.