The increase in the pace of change caused by digitization is driving organizations to move faster than competitors and exceed customer expectations by offering frequent updates to their product and service enabled by technology. In some cases, as quickly as once every 12 seconds. The traditional testing approaches to ensuring reliability that is dependent on human intervention simply cannot keep up. Some companies are evaluating a move from test automation to autonomous testing that takes advantage of AI/ML to make testing less dependent on human intervention and self-learning. How does autonomous testing work? Is it ready for the real world? How can organizations transition from test automation to autonomous testing with confidence?

Contributors

• • Tony Mohanty, SVP and Global Head, Digital Assurance, Hexaware
  • Nagendra BS, Vice President, Digital Assurance – Practice & Solutions, Hexaware

Transcript

Sanjog Aul [00:00:00]:
Welcome listeners. This is Sanjog Aul, your host and the topic for conversation is From Automation to Autonomous Testing. With digitization, we are in a race. The race with our competitors to stay ahead in meeting and exceeding customer expectations. It requires that we keep innovating our products and services using technology and then take them to the customers as quickly as possible, in some cases in a matter of few hours. For such cases, the traditional approach to testing that is dependent on human intervention simply cannot keep up. Some companies are evaluating a move from test automation to autonomous testing that takes advantage of AI and ML to make testing less dependent on human intervention through self learning. So how does autonomous testing work? Is it ready for the real world? How can organizations transition from test automation to autonomous testing? With confidence to discuss this, I have with me Tony Mohanty and Nagendra BS.

Sanjog Aul [00:01:05]:
Tony is the Senior Vice President and Global Head of Digital Assurance and Nagendra BS is the Vice President and Head of Practice and Solutions at Hexaware, a consulting firm focused on transforming IT solutions and solving complex business problems using a combination of human creativity and and intellect. Their three pronged strategy of automate everything, cloudify everything and transform customer experiences enables enterprises fast track into the digital era. Hello Tony and Nagendra, thank you for joining us.

Tony Mohanty [00:01:38]:
Thanks Sanjog, thanks for having me.

Nagendra BS [00:01:41]:
My pleasure. Thank you for having me, Sanjog.

Sanjog Aul [00:01:44]:
And so Tony, my first question is for you. The test automation discipline is not new. Most organizations have already deployed it and are reaping benefits from related investments. So why should anyone drop what’s working well and consider autonomous testing?

Tony Mohanty [00:02:04]:
So Sanjog, automation per se has a very limited connotation and it’s usually associated with execution of tests from a QA perspective. Whereas autonomous testing, it has a far wider impact across all the phases of the testing lifecycle, primarily with the aim of eliminating human intervention in the related tasks. So organizations don’t really have to drop any of their existing automation assets since automation flows into the broader autonomous testing journey and I think this is becoming increasingly more relevant because of a few reasons. The first one being the pace of change that is driven today by all the digital transformation programs is rapidly increasing and we have a new generation of agile competitors emerging and customers have started expecting far more rapid updates to their products and testing just cannot be slowing the whole thing down. The second reason is beyond the current level of automation that exists today, what are the other levers that we have for increasing the speed of delivery and in parallel reducing human intervention? How do we increase the efficiency and actually the most important point is increase efficiency by reducing cost of quality. I believe autonomous testing is the answer to these questions. I will explain it with a simple example.

Tony Mohanty [00:03:24]:
So one of our customers in the airline business, they had pretty high level of test automation and they already operate in a DevOps model. They have daily builds, they are using continuous integration, continuous test. However, even there we find many test activities that are still done manually, which continues to have dependency on people and obviously there is an associated cost attached to it. For instance, impact analysis of all the requirement changes. That’s a complete manual process. Maintenance of automation scripts, it is done by the automation team, but there is manual effort that goes into maintaining the scripts. Similarly, data from various sources, like production logs, there is no proactive learning.

Tony Mohanty [00:04:06]:
It’s always a reactive process. When a defect or an incident occurs in production, that’s when the whole process kicks in manually. Similarly, the entire regression pack is usually run in every cycle, which is also not optimal. Ideally, we should be able to pinpoint which test scripts to execute for a given change and execute the same, thus saving both time as well as cost for the company concerned in every cycle. So I believe this is the same story with most of the enterprises in the industry today, where majority of the testing related activities and the decision making still happens manually. While these are some of the key challenges that we face today, this also presents a pretty big opportunity for us to support this pace of change. We believe that the next level of transformation in software testing lies in shifting our focus from test automation to autonomous testing. Using artificial intelligence machine learning to make testing the fastest cog in the entire DevOps chain.

Tony Mohanty [00:05:06]:
I think by making testing a process which is virtually independent of human intervention, this actually has the potential to result in incremental savings of anywhere between 30 to 70% depending on where the current level of maturity is with a particular customer and this is on their current QS spend. So these are precious dollars that organizations can save, especially in the times that we live in today. Also from a external view, from a market perspective, the analyst reports also corroborate this shift. According to Gartner’s study done a few months back, they state that by 2021, intelligent automation will generate an additional 20% savings over what is achievable today in application testing service. Similarly, Forrester survey a year back had predicted that using AI and ML will actually result in testing faster and an increase of quality. So 37% of organizations felt that this was true. Another 61% of the respondents said that they were already using AI ML algorithms to prevent incidents in production, and more than half of the respondents use AIML for augmenting the tester’s capabilities.

Tony Mohanty [00:06:19]:
So today I know of quite a few companies, or at least a dozen startups like Autonomic, FunctionEyes, AlgoShack, just to name a few, who are already building products to enable autonomous testing and they have been able to successfully raise millions of dollars in funding from various private equity and venture capital firms. Which is also a testimony to the fact that there’s a potential rise expected in autonomous testing. Hope this clarifies Sanjog as to why organizations need to shift their focus to autonomous testing and the big opportunity that lies in front of all of us.

Sanjog Aul [00:06:54]:
Absolutely. Thanks Tony. Now okay, granted, autonomous testing does sound like a great concept and perhaps can help organizations keep up with the pace of change but my question will be why hasn’t it taken off and what’s holding us back? So Nagendra, what are your views?

Nagendra BS [00:07:15]:
Sure Sanjog, truth as part of Intelligent Test Automation, different elements of autonomous testing are already in place. Not as an end to end solution that would make complete testing function independent of human intervention. There are many reasons for this. One of the biggest reason is testing not seen as strategic growth and efficiency enabler in the organizations and this discourages from any investments to transform the testing function. Other reason is skepticism around the solution being too futuristic organizations, for whatever reasons may not have been able to get the return on the investments they have made on test automation. Some of these are ground realities that we have to deal with. Till recently we also did not have the kind of access to AI enabled technology solutions and platforms like TensorFlow, Kera, Theano and similar platforms which helps in democratizing AI

Nagendra BS [00:08:24]:
and this was another limitation that was holding us back. Another technology constraint that is still holding us back is around unavailability of a unified platform to support autonomous testing for both functional and non functional testing across all the phases of the testing life cycle and all layers of an application, which is what Hexaware has embarked upon. If we also share some of the recent conversations with our customers on this topic. At the start of this year we had hosted many customers across varied industries like airline banking, insurance and retail where we presented our vision to move from test automation to autonomous testing. While we had some customers who bought into this and offered to run some pilots, there were also a few customers who were skeptical about the solution. In fact, some of them are customers whom we have been engaged with for more than eight to 10 years. While we acknowledge that their reservations are legitimate and are specific to their environments. We are also working with them to address their concerns since we believe this opportunity has greater upside to our customers.

Sanjog Aul [00:09:42]:
So Nagendra, what is the proof that autonomous testing would really work? Has it been tested in the field with real use cases? What are the results and the related learning based on all that field testing.

Nagendra BS [00:10:01]:
One thing is Sanjog as Tony mentioned earlier, one of the proof points from a business potential point of view is the fact that startups focusing on autonomous testing solutions have been able to raise millions of dollars in funding from various private equity and VC firms. This would give you some level of confidence that autonomous testing is real. We are also finding sponsors at the CIO level in our customer organizations who are encouraging us to deploy these point solutions. To give an example, we recently implemented one of the point solutions available in the market for a manufacturing services company. Through this solution we were able to do a seamless conversion of existing manual test cases into corresponding automation scripts without any human intervention. We also have seen test analytics solutions implemented in the industry that uses Python based ML libraries to predict defect patterns for the future releases based on the data from the previous releases. Another example that I can quote from our experience is to support testing in behavior driven development or BDD mode of SDLC, we implemented a solution using Python based natural language toolkit that can import a Gherkin language feature file and generate corresponding automation scripts without any human intervention. Similarly, there are many other examples where Python based AIML libraries and algorithms like logistic regression, reinforcement learning, clustering and many of these being used for solving challenges around script maintenance and impact analysis.

Nagendra BS [00:11:49]:
Through our experience of implementing different point solutions and elements of autonomous testing, one thing that we are absolutely convinced is that AI and ML is real and will make a difference to the way testing will be performed. Also, AI may not eliminate manual testers completely but will augment their skills. We also believe it is critical to have a C level sponsorship for the success of this transformation and there needs to be a top down approach to adopt the change as it requires collaboration between different IT teams outside of QA which involves development, infrastructure, release management and others to make this whole transformation happen. Finally, we see a need to have a unified platform that can orchestrate end to end testing activities without any human intervention. This platform should be able to integrate seamlessly with existing or third party automation solutions and if necessary have its features exposed as services or APIs for external consumption. Hope Sanjog this gives clarity on proof points around autonomous testing, its use cases and our associated learnings.

Sanjog Aul [00:13:10]:
Thanks Nagendra so let’s take a quick break listeners. We’ll be right back after these messages and Tony, when we come back would be great for you to share some of the tenets of an ideal solution to enable autonomous testing most effectively and also how well do currently available solutions, including Hexaware’s own offering, meeting that benchmark?

Sanjog Aul [00:13:33]:
Please stay tuned.

Sanjog Aul [00:13:34]:
We’ll be right back.

Sanjog Aul [00:14:36]:
Welcome back. So Tony, what are the tenets of an ideal solution to enable autonomous testing most effectively and how well do currently available solutions which include hexaware’s own offering, meet that benchmark?

Tony Mohanty [00:14:54]:
Sanjog I will answer this question in two parts. Coming to the key tenets of an ideal solution. I believe that there are three parts to the tenets. First one is a comprehensive maturity assessment framework that covers use cases, the different personas involved in the whole life cycle and the activities across the testing lifecycle to evaluate the customer’s current maturity on autonomous testing and then subsequently provide a roadmap for implementation. The second aspect is in terms of having an integrated test orchestration platform with a plug and play architecture that enables customers to either go with the vendor solution or integrate their existing or third party automation solutions or even have an additional option to consume the platform’s features as services through API calls and the third element which I think is the most important is data. Data is key for this platform to deliver results and we believe that there are four key processes that the data goes through before being operationalized and being consumed by the test orchestration platform. The first one being the acquisition of massive amounts of data that gets generated during various phases of the application lifecycle. This is called the acquire phase. We call it the acquire phase and it’s categorized into four different sub segments.

Tony Mohanty [00:16:19]:
Voice of customer, which is basically your end user’s feedback. Voice of machines, which is more like production logs that we get from systems. Voice of tests. These are typically defects that the QA team raises during a cycle of testing. Voice of business, which is more of functional requirements, and the voice of developer, which is basically the review comments from design and doing the source code reviews. So after the data acquisition happens, the next phase is analysis of this data, followed by the third phase which is developing inferences. We call it the Agree phase. Based on the analysis, what are the inferences that we draw from the data and finally act on the data that is processed and the inferences that we draw from it

Tony Mohanty [00:17:04]:
and this is where AI algorithms play a very key role in terms of decision making. To answer the second part of your question, Sanjog from Hexaware’s readiness point of view, we have a dedicated consulting team that performs a detailed due diligence over a period of up to three weeks and we use our ATMA framework. ATMA stands for Autonomous Test Maturity Assessment Framework and this has five levels of Autonomous Testing Maturity. The outcome of this assessment is a detailed report and the roadmap for implementing the recommendations. On a lighter note, the acronym ATMA is quite relevant in this context because ATMA means soul and this framework is the soul of the test maturity roadmap for all organizations. While there are existing point solutions in the market, Hexaware has an integrated platform in the form of a platform called ATOP Autonomous Test Orchestration Platform that uses machine learning, deep learning and natural language processing to enable this transition from automation to autonomous and it can be the one stop solution for all the testing needs of our customers going forward as well. As part of ATOP implementation for one of our customers, we actually brought to life the whole use case related to Voice of Machine. So we had the analysis done of all the production logs and it enabled us to automatically identify the issues from the production logs and generate corresponding automation scripts to strengthen the test pack without having to manually do any of the steps.

Tony Mohanty [00:18:41]:
Another use case is around Voice of Customer. This is to analyze the sentiments expressed by the end users. This could be on portals or on social media channels. The solution goes through the customer reviews from various sources classifies the positive and negative reviews clusters the predicted negative reviews using specific module names and this helps us to easily identify which modules were prone to issues and generate test cases for those specific modules. As part of our roadmap for ATOP, we’ve actually identified more than 50 such use cases across both functional and non functional testing to enable autonomous testing for all types of testing across all the application layers and we believe this is how organizations would have to plan this journey in order to embrace autonomous testing in the future Sanjog.

Sanjog Aul [00:19:36]:
Great so coming back to you Nagendra, how should organizations start on the test automation to autonomous testing journey all along ensuring the quality and accuracy of the results produced and minimizing risk during the initial implementation as well as when fully operationalized?

Nagendra BS [00:19:57]:
The first thing that we would recommend enterprise leaders is to recognize the fact that there is an opportunity to tap by thinking beyond automation and the journey from automation to autonomous would take anywhere between 12 to 24 months depending on their current level of QA and automation maturity. An assessment of the current maturity of autonomous testing and baselining of existing metrics must be done to arrive at a detailed roadmap for implementation. In the roadmap should cover details like the tasks that will be done in house versus the ones using partners, clarity on which are the testing activities that can be completely made autonomous versus activities that will still have to be done manually. What is the approach for building the unified orchestration platform? What is the approach for acquiring data across various tools in the ecosystem and many other relevant details to make both functional and non functional testing autonomous across all phases of the testing life cycle and the layers of the application? From an implementation point of view, we would recommend picking one or two pilot projects or programs which are mature enough to take up activities beyond test automation and show the initial proof points before we take up enterprise wide implementation and in terms of the ownership. Definitely the QA function would be the owner of this transformation, but we would recommend skin in the game approach both for stakeholders within the organization outside of the QA function and for the partners who will be supporting this transformation. It is also very important to have a very well defined engagement model that would enable organizations to measure their partners and the outcomes performance against the corresponding SLAs and KPIs and at the same time provide necessary ownership to partners so that they can drive all transformation and their set of activities independently. Finally, this transformation will not be successful without right people on the ground to deliver, so we need to enable workforce transformation through SDETs Software Development Engineer in Test who are multi skilled test engineers required for driving this kind of a transformation and also take a pragmatic approach for implementing autonomous testing by using existing assets that they already built and complement the same with the partner capabilities and finally accelerate this whole transformation through management function and the OCM coaches.

Sanjog Aul [00:23:11]:
Thanks Nagendra so finally Tony, transforming from test automation to autonomous testing does seem like a significant effort which may require help from a partner with specialized expertise and experience. Now since it is such a new discipline. How should an organization go about selecting the right partner for this effort and if an organization, say, selects Hexaware as a partner, besides just bringing your technology platform, how can your team help in ensuring success, reducing risk and maximizing business outcome?

Tony Mohanty [00:23:52]:
For selecting a partner for this kind of initiative, I believe organizations should look at potential partners for whom software testing is one of their strategic businesses. In the partners company, the partners should have solid experience and credentials in delivering quality engineering solutions across multiple technologies and industries. For a period of time, they should have demonstrable solutions that uses AI and ML in testing and corresponding case studies and client testimonials. From wherever they have implemented these solutions, they should be willing to put their skin in the game and Nagendra did allude to it earlier in his answer. We want to have partners who are willing to commit outcomes upfront, both from a delivery perspective and commercially as well. Now the second part of your question in terms of Hexaware’s readiness and if you look at Hexaware as an organization, we are a global IT services company. We have been delivering quality assurance and engineering services under the digital assurance umbrella for more than 20 years now.

Tony Mohanty [00:25:00]:
Nearly 20% of the organization’s revenue is from digital assurance services and this is a strategic business for us. There is a razor sharp focus to help customers succeed in their transformation journey that is built into our DNA. Compared to our peers in the industry, we have disproportionately invested more than 5% of each unit’s revenues towards maturing service offerings and driving innovation. This is done through a central innovation lab that supports rapid solution development. Now we have a well established partner ecosystem with both established players and niche startups that accelerate this whole journey towards autonomous testing. We have multiple referenceable case studies of both automation and autonomous testing solutions across clients, across geographies, across industries. For example, recently there was a autonomous testing led managed test service for a multibillion dollar global airline. Similarly for a large European insurance company.

Tony Mohanty [00:26:03]:
For the last four years we have been doing their end to end automation and maintaining the whole stack. For a secondary mortgage provider in North America we have what we call as extreme automation for them. Also, an important aspect which Nagendra did touch upon was to have carrier test analysts who are SDETs for us. 75% of our career test analysts are SDETs which is software development engineers in test and these are ideal consultants for implementing autonomous testing in DevOps and Agile programs. Another unique feature which I find in Xavier is the Brainbox platform where we have a crowdsourced platform for adding value to customers where every hexa variant can contribute ideas for eliminating manual work through automation in their respective engagements. Now with all of these, the icing on the cake is that we have an excellent leadership team that is very well aligned to our strategy for the future with focus on automate everything, cloudify everything and transform customer experience. We have built enormous trust over the years but both with existing clients and some of these clients have been with us for more than 20 years now and new customers as well whom we have onboarded recently. I believe that with such credentials and experience we are pretty well positioned to lead our customers all the way in this journey of autonomous testing Sanjog.

Sanjog Aul [00:27:26]:
Once again thank you both Tony and Nagendra for sharing your thoughts and insights about how an organization can transition from test automation to to autonomous testing to keep up with the pace of business change.

Tony Mohanty [00:27:43]:
Thank you Sanjog.

Tony Mohanty [00:27:45]:
Thank you Sanjog.

Sanjog Aul [00:27:46]:
And listeners, I invite you to find related conversations on our website at ciotalknetwork.com.

Download Podcast
Apple Podcast, Google Podcast, Spotify, Pandora, iHeartRadio, SoundCloud, TuneIn, and Stitcher. Find other syndication channels here or search CIO Talk Network podcast on any other app.
Explore More

• • Can Kofax deliver on its Intelligent Automation promise?
  • Evaluating Puppet’s Automation Capabilities
  • From Traditional IT Ops to Automated AI Ops
  • Automate Everything—The new cupid for BPOs
  • Charting the Path to Automating the Data Center
  • Taking advantage of digital in our companies
  • Effective Patch Management
  • Steps to Making Data Your Greatest Asset
  • Is Your IT Prepared for Fluid Customer Expectations?
  • Can IT Push Innovation Beyond All Customer Expectation?