For the last year I have been running crowd-sourced test cycles for my company while testing Android and iOS native apps. I learned a lot, met many good testers, and had the privilege of working with and test leading a few unbelievably passionate and talented people. Leading testers in a remote model where you never meet your team in person is difficult, especially when you have to figure out their strengths and aspirations. Plus test management in this context is something I was never sure about because of the overall responsibility and worrying about the end result and gathered information on product quality. I keep thinking that what if the mission fails due to misunderstanding and misinterpretation of the goals and objectives? What if the team energy and dynamics will not form correctly?
To my advantage I was able to participate in hundreds of uTest cycles as a tester and could observe and learn from other participants as well as network and build relationships. Still, when it was time for me to launch my first test cycle I was a nervous wreck even though most of the testers had been hand-picked. Now, imagine working a test run for wider device and OS coverage with over 30-40 testers who are located in different countries and time zones, speak different native languages and think in their cultural values terms. Diversity is a great strength in software testing. However, how do you harness and make the best of it in a short 2-3 day turnaround while testing early beta builds? It felt like testing on steroids sometimes with the discussion forum exploding with questions and the number of bug reports growing like mushrooms. It was hard to set up the right expectations with so many unknowns and partially finished/functioning features. We did the best we could with what we had. We kept learning about the product as it evolved and as a team as we continued working together.
Eventually I came to the realization that having a bigger team does not guarantee better results and that common risk patterns could be identified on a smaller set of devices and ultimately with a smaller group of designated testers. And so I convinced my boss to allow me to form two designated teams for iOS and Android testing with 2 testers on each team. Mobile app tester selection process was not hard for me as I followed a few true gems in the community. The challenge was their willingness and availability as this testing involved working after normal business hours. Plus none of the folks that I wanted to recruit resided in the US. I must have really wanted this to happen as all 4 testers that we wanted were available – 2 in Europe, one in India and another in Brazil. The iOS Special Forces team sprinted first. Weekly test cycles with test goals and details frequently updated in the platform, open communication via Skype and email, quick feedback from the dev team and non-competition over the number of bugs submitted allowed us to work effectively. The first couple of weeks were a bit rough as we had to get used to testing internal alpha builds and lots of things were still incomplete and ‘broken’. It was hard for the team to hold back and not log obvious inconsistencies but we learned to restrain ourselves and shift focus to areas that required our input and attention. For example, the testers provided constructive and valuable usability and UX feedback and updated the regression test script while waiting for new builds.
When people talk about a match made in heaven that’s what this partnership felt like. It was truly amazing to see that people who never met, live in different cultural and time zones can come together to test software and have so much fun. And yes we debated and disagreed but we worked like a band of brothers empowered and driven by the same goals, beliefs and principles. We were doing what we love most, learning and delivering good value for our customers. We craved working together as we progressed further into testing more stable builds and new features. A native app life cycle is much shorter than a web or desktop one – just 3-5 months. And so it was amazing to watch the program evolve and be a part of the process from start to release. Granted we did not like many things and disagreed with design and dev folks, and advocated and pressed for change. Sometimes it worked and sometimes it didn't, and sometimes we were wrong as something we thought was a bug turned out to be a feature. The whole experience was very empowering and rewarding.
And so to sum it up I’d like to think we did a great job. I managed to select the right people, set up goals and expectations, empower and motivate, and create the atmosphere for learning and self development. It felt really good.