No matter how big a customer you are, when a vendor wants to sell you a product at $239,000 a pop, you have to ask yourself serious questions whether it can save you money or give you some competitive edge. Reading Ian Murphy’s Spotlight Review of HP’s Service and Network Virtualization instantly set me off asking such questions with my old Test Manager hat on. It’s not a proposition any CIO I worked for would have expected from the testers alone. Right from the outset we see that this is not a project-only, dev-only or dev-and-test-only play. Groups across an organization would have to be convinced that the benefits were solid. And were I the CIO I’d look at the proposal and ask, ‘how on earth are we doing this testing now?’ or more pointedly, ‘what kind of costs and risks are we running without this’?
Complexity is not simple to test
Ian’s Spotlight lays out the background you’d expect: the move to shared resources in the cloud, multifaceted delivery across platforms and devices, the complexities of development and testing with shared internal and external services have all changed the application delivery landscape from the halcyon – well, relatively speaking – days of on-site development of internally hosted and network systems. I’d say this change is a transition nearly complete. The industry is there. We’ve had 10 solid years of SOA and web service-based applications and five good years of mobile and cloud. The avant-garde passed long ago, and there can’t now be many stragglers, not among organizations capable of a hefty investment in tools.
Functional and technical problems, potential points of failure and risks have proliferated with the increased complexity of application architecture. Beyond functionality, there’s performance. Beyond performance, there’s security, and further along there’s the complex DevOps implications of building apps with tentacles in all directions. All of this may be served up outside the data centre in the cloud, on an unpredictable and insecure internet to diverse mobile and desktop devices.
So how have organizations been testing these heterogeneous, multi-channel, multifaceted applications? The answers will sound familiar: manually, using one-off home-grown tools, incompletely and probably at great expense, or even not at all. It still astonishes me the relatively low market penetration of performance testing tools: 30% of projects using them is an optimistic number. How on earth do you test performance at volume (of transactions and data) without automated load generation and performance monitoring. Crowdsource? Push your untested app live to users and let them break it for you? That might work for a fledgling social media app but too much is at stake for a business that sells something more than hot air.
Taming test complexity: Lifecycle Virtualization
So what does HP have to help? HP has been selling into the testing tools space for a decade during which time it has branched out to cover the whole application lifecycle except for code cutting itself. In these last 10 years, vanilla virtualization of operating systems (VMs) was a first step towards solving the biggest time and money waster in software testing: creating a correct test environment. It was a boon. Nevertheless, it is still not uncommon to burn more than half of a test schedule just trying to establish the conditions in which meaningful tests can be performed.
HP’s current extended vision for virtualization in testing is one it calls Lifecycle Virtualization, which comprises four types of virtualization: Service, Network, User and Data. As you might expect, HP can offer you solutions in all four, albeit, one area is less well supported than the others, as we’ll see. What Ian points out, and we need to keep clear on, is that they’re talking here about virtualization on the dev side of the DevOps passthrough.
The first of the four pillars, Service Virtualization, is all about freeing us to test interfaces to external services without actually incurring all the difficulties of them actually being present in a test configuration. When we say external here, we mean anything outside the current application under test. Similarly, Network Virtualization is about creating realistic network conditions for testing without the real network present. User Virtualization is about testing the variety and loading of user transactions – all the various flavors of performance and load testing. And Data Virtualization, well, that seems to be about the data side of the transactions in load testing. Among the four pillars of Lifecycle Virtualization, Data seems the least well supported by HP at the moment. Perhaps they have plans.
Can lifecycle virtualizations really pay?
The focus in Ian’s Spotlight review is on the package of just two of the above: Service Virtualization and Network Virtualization. These are not entirely new products; we first heard from HP about Service Virtualization nearly three years ago and the Network Virtualization stems from HP’s acquisition of Shunra in 2014. The User and Data Virtualization are mainly about HP’s industry-leading LoadRunner software. (Pace Mercury Interactive.)
I’d hate to give the impression that this is just a rehash. It’s not. It’s a meaningful effort to produce some synergy for solving key challenges in testing, making it possible, in my view, to test what is, frankly, untestable without automation. I think it’s an excellent idea to position point tools within a coherent, overarching schema. It gives prospective users a clear, unified strategy for improving testing and reducing costs through automation. And if you can see the big picture you can also make a realistic case for a big spend on automation.
Now that’s the theory. Can you do it? Can you make it pay? HP has gone to some lengths to provide an ROI calculator and to compile case studies of those who’ve succeeded. Only by running your own numbers through the calculator can you tell whether the investment is worth it for your organization. Our take on whether it’s worth it? It will be if you’re sure you count what it’s really costing to test your complex, distributed apps manually, with homegrown tools, incompletely or not at all; and what it is you’re risking.
(For an in depth look, read Ian Murphy’s HP Virtualization for Development, Quality and Test Teams)