Crowdsourced Testing: Lessons Learned From a Customer's Point of View.

I am frequently asked about crowdsourced software testing companies and how to best work with them. It’s a tough question as every company is different and therefore the approach and strategy has to be customized accordingly based on what they want to get out of utilizing the crowdsourced software testing model.

I have freelanced with crowdsourced testing companies for two years and have learned a lot about the processes, platform tools, freelance tester communities, product management, challenges, and shortcomings from the tester side. In the last year I have expanded my knowledge by running a high volume of testing with crowdsourced testing vendors for Android, iOS and mobile web apps at my current company as a customer. Overall it has been very successful based on the feedback and data that we received from freelance testers. But it hasn't been a bed of roses as it has taken some trial and error as a customer to obtain the right knowledge and experience that in turn helped accomplish our testing goals. I highly recommend that your in-house professional testers handle the relationship with the crowdsourcing company(ies) because they understand your testing needs and your company's business.

How I Became a Tester

Last month I was asked to conduct an online webinar on how I became a mobile tester. It didn’t work out because of my crazy work schedule and miscommunication with the event organizers.  However I had an opportunity to work on my topic with James Bach. I got some helpful guidance, questions and tips from him and then decided to turn the content into a blog post.

I want to mention that James is probably the toughest mentor that I have trained with. But he is also the most approachable, sincere, committed, consistent teacher, in addition to being extremely smart and helpful. I came to him originally without any formal testing background or training. I feel I am a better tester since I started training with him and the fact that he is just a Skype away gives me a lot of confidence when I come across difficult situations in my professional endeavors. I have met many other role models and mentors in our global testing community but imho there is no one like him and I am grateful that I found him.

How did you get into testing? Why did testing attract you?

What initially attracted me to testing was the never-ending process of puzzle solving. I enjoy the creativity involved when you have to come up with a scheme of different ideas and approaches to dig into a program and find what’s broken or what may potentially cause a problem.

MoDevUx and the Latest On My Testing Journey

I attended another amazing event last week organized by MoDevDC, my local meetup group. The event was called MoDevUx – “A user experience, design and development conference discussing all things mobile.”

MoDevUx was packed with interesting content, amazing speakers and the exuberant energy of its participants from various mobile design and development shops in the Washington, DC area, and some from other parts of the country. The speakers shared their experiences on how to understand users (what users think and want), how to handle feedback, learn from it, and improve the user experience in mobile apps. I thought that the presenters talked from their heart, they were entertaining, witty and insightful. I walked away feeling enriched and that my time was well spent.

Kindle Fire Tips on Error Logging, Debugging and Snapshots

I have been testing on the Kindle Fire since December 2011 and have picked up a few good tricks. These include collecting error logs, taking snapshots, debugging with DDMS and installing applications that are currently not available in the Amazon Appstore such as Dropbox.  I am sure there is lots of information scattered around the web and posted on various forums but I want to share my tips and the following blog links in one location for your use.  I will try to add new ones to this post as I learn them.  Feel free to leave comments with your tips.

Questing in Boston

I received a 'QUEST 2011: Thank You Letter' from QAI Global Institute yesterday that was emailed to the participants list. I meant to post a blog about the conference when I came back on April 8 but got carried away with my busy work routine and getting ready for a release. This email brought back good memories and I thought I would share a few sentiments here.

This was my first testing conference and I did not know what to expect. Granted I have been to quite a few international conferences when I worked as an interpreter but those were on oil and gas, US-Russia business relations, auto shows and international law. My responsibility was to help people communicate, negotiate, resolve conflicts and build relationships. 

There were days when I would not have time to eat. Meals were provided of course, followed by scrumptious desert, but I was not able to eat as I was busy talking, or rather people talking with each other but through me. People tend to be more relaxed, personal and prone to heart to heart conversations when they share food. If you add alcohol into this equation then the conversions would get more colorful and run longer. I even had lunch with the Honorable Judges Scalia and Ginsburg and my clients. They were enjoying an amazing lunch and interesting conversation and I was busy passing information back and forth in Russian and English.

Needless to say I never really had time to explore any of those shows as I had to follow a rigid schedule and attend workshops and receptions of the clients' choice. This time it was different. Attendees were IT professionals and mostly testers and QA folks. Of course I know that there are so many of us scattered around the US and the rest of the world but seeing SO many test/qa-obsessed people in one place was something!

I got to my first Quest day at around lunch, registered, picked up my badge and joined the gang of hungry testers. I like people-watching and while sitting by myself and observing I was able to tell the ones who were noobs like me from the ones who have either attended Quest or similar events before. They were vibrantly discussing the agendas, upcoming workshops and instructors that they knew or heard of, liked, disliked or wanted to meet. I overheard people talking about Agile, mobile testing, Jon Bach, etc. I looked through my own 'Quest at a Glance' leaflet and circled additional tutorials based one what I had heard -- just in case!

I defined my objectives and mission for the event but also kept my options open. My plan was to learn as much as possible about Agile and mobile testing, attend Janet Gregory's and Jon Bach's workshops, spend time with the uTest team and network if I was brave enough.

My first tutorial on mobile apps testing was with uTest. Needless to say I was beyond myself excited to finally meet uTesters that I got to know over the last few months through either following the uTest blog or working as a freelance tester evenings and weekends. The tutorial covered planning, mobile testing approaches and challenges as well as crowdsourcing. One of the uTesters gave a hands-on walk-through  demo of an iPad application test. It was great. Participants seemed to share my feelings. They were curious and asked many interesting questions and shared their personal stories.

The tutorial was followed by the uTest reception. Right before it I finally met Peter Shih. Peter and I corresponded via email and the uTest forum. It was so cool to finally meet him and other uTesters face to face. It felt like I have known them for quite some time. I changed my career paths and jobs a few times after I moved to the United States. Working for UUNet Technologies, SRA International and with uTest have been my best gigs by far. 

The next two days were filled with good stuff.  I attended sessions on Agile estimation and dealing with defects, Performance Testing, and Test Planning and Design. Even though the material in some classes was a bit high level and over my head I was still learning about new ideas and how people do things in different organizations.  

Another highlight of my Quest agenda was an evening cruise aboard the Spirit of Boston, sponsored by Microsoft. Boy it was fun! I never thought that testers were such party animals! I tend to get seasick even from minor waves and it was quite windy in Boston that evening. I was nervous that I might get a wicked sea sickness attack at when I least expected it. A little bit of wine and dancing seemed to have taken care of that. A couple of people break danced. How cool is that! No one seemed to mind the waves. I had a light headache the next day but that was a small price to pay considering how much fun we had.

After Thursday’s workshop on Exploratory Testing taught by Jon Bach I was proud to say that I finally met both Bach brothers. I loved Jon’s workshop. It was dynamic, entertaining and full of fun examples and brain teaser exercises. Not only have I learned about the Japan Quake Map but ever since I have been obsessed with finding those special penny coins that don’t have a Lincoln memorial displayed upside down when you flip them over.

Quest 2011 is complete. It’s time to get ready for CAST2011 in August. Again I don’t know what to expect since it will be a whole new experience and a chance to explore. One thing I know is that I will meet other Association for Software Testing members and testers who are passionate about our craft and hopefully both Bach brothers under the same roof.