Crowdsourced Testing: Lessons Learned From a Customer's Point of View.

I am frequently asked about crowdsourced software testing companies and how to best work with them. It’s a tough question as every company is different and therefore the approach and strategy has to be customized accordingly based on what they want to get out of utilizing the crowdsourced software testing model.

I have freelanced with crowdsourced testing companies for two years and have learned a lot about the processes, platform tools, freelance tester communities, product management, challenges, and shortcomings from the tester side. In the last year I have expanded my knowledge by running a high volume of testing with crowdsourced testing vendors for Android, iOS and mobile web apps at my current company as a customer. Overall it has been very successful based on the feedback and data that we received from freelance testers. But it hasn't been a bed of roses as it has taken some trial and error as a customer to obtain the right knowledge and experience that in turn helped accomplish our testing goals. I highly recommend that your in-house professional testers handle the relationship with the crowdsourcing company(ies) because they understand your testing needs and your company's business.

Here are some tips on how to run a test cycle in case you decide to use a crowdsourced testing service.

1. Know what you want to get out of it – Sounds like common sense, right? However, there are so many folks who believe that crowdsourced testing will solve all of their testing needs. And therefore, they run their test cycles with high expectations while providing vague and conflicting instructions that only confuse the testing team.  Many customers don’t respond to inquiries and requests for additional information and then become upset with the end results which usually may then consist of useless bug reports and feedback. So knowing what kind of tests or types of investigation you want to run, whether it’s functional, feature complete or regression testing, is very important.


2. Test team selection – This is critical. If a team consists of people with the wrong set of testing skills you need (this goes back to knowing what you want) and they lack knowledge about your product and your industry, then you may fail to get the desired results.  Work with your crowdsourcing company project manager(s) on selecting the right people to test your product.  Request to review their platform tester and/or LinkedIn profiles to understand their backgrounds better. For example, testing news and media products requires a different set of skills from testing a game or financial application. Just because a tester has an overall high ranking does not mean that he is good at testing all types of products and apps that he works on. We have run into some duds who produced no value but were highly rated or recommended. There are also real rock stars, if you find them try hard to keep them on your projects as they are in high demand.


3.  Set up clear and concise testing goals – Talk to your team and form an idea of what kind of feedback you want to get from your crowdsourced testing team.  Are you looking for crashes, showstopper bugs, feature complete smoke test review, or usability feedback - be specific. This will take some effort to formulate but will save you time during and after a test cycle ends when it’s time to triage bugs.  When testers misunderstand the goals and expectations (there are always some who will) they will either bombard you with questions, log many useless bugs, or will not test your product at all. Consider including a video demo of your product under test where you can mention what to focus on. It doesn't have to be elaborate or long. A 5-10 minute video can be taken with either a Smartphone or iPad and attached to a test cycle. It saves SO much time and effort on explaining things!


4. Provide hardware/software/browser and other technical requirements – For example if you only intend to run testing in IE8 browser and above, or in a native browser on iPad or Android phone, then list all the other browsers as out of scope for testing.


5. Bug report requirements – When testers find issues or inconsistencies with your product they will submit bug reports. To make things easier for yourself and your team provide detailed instructions on what you want to see in a bug report. Some freelancers are good bug hunters but have not worked in Information Technology and will need guidance on what to provide with a bug report. For example, a bug title may include the build version, device and its OS or browser name, area where the issue occurred and a brief description of the problem (e.g.,’ v.2.1-iPhone 4S 5.1-Login - App crashes on login’ or ‘Win7 IE9 – Article page – Text rendering issues). Ask testers to provide clear steps and additional evidence such as snapshots, videos, console and error crash logs, Speedtest.net results, etc.


6.  Clearly list known and out of scope items - Not all of the features will be complete or in working order when you launch a test cycle unless you have the crowdsourced team regression test a production version. For example, your development team may not be interested in UX feedback, typos, security or non-native browser testing on the iPad. In your instructions outline what not to test to save yourself time and headache. Sometimes it is hard to list ALL of the known and unresolved issues but you may indicate areas in the app that are not fully completed. Consider providing a document with known bugs or utilizing Google docs if this information is not sensitive.


7. Be prepared to respond to inquiries and questions promptly – a test cycle is generally most active during the first 8 to 24 hours so it’s crucial that your team is prepared to respond to questions or assist with unforeseeable issues such as app installation, browser incompatibility, web app server and proxy setup problems, etc. If no one responds promptly to the testers, they will either lose interest and leave or submit bugs related to these very problems and not the ones you are looking for. The crowdsourcing Project Manager is there to help but their main job is to set you up, line up testers and execute testing. Most of those PMs have very little testing or technical background and only have a high level of understanding of your product under test.

There is no secret recipe managing crowdsourced testing. It’s not magic and will not solve all of your testing problems. It takes time and effort to figure out what works and what doesn't for your context. It can be very powerful, highly effective and even addictive if used correctly. On the flip-side it can be very frustrating, time-consuming and of no value. Crowdsourced testing will supplement and assist your in-house QA/Testing team but will not work as a substitute or replacement.