Welcome!

Java IoT Authors: Pat Romanski, Liz McMillan, Elizabeth White, Yeshim Deniz, Amit Gupta

Related Topics: @ThingsExpo, Java IoT, Agile Computing, Wearables, @CloudExpo

@ThingsExpo: Blog Feed Post

You Can’t Test All the 'Things' | @ThingsExpo #IoT #M2M #API #InternetOfThings

Young engineers building all sorts of IoT devices without a thought to protocol or integration standards

You Can’t Test All the Things: API, IoT, ROI TBD

I have three words for everyone in software testing: prioritize, prioritize, and prioritize.

You can't test every possible permutation of your software, especially so with APIs and IoT devices where you're placing much of the user experience in the hands of integrators to your core products and services. You just can't, so we should throw our hands up now and just give up, right?

What sort of tester are you anyway?

I'm not; at best I'm an automation engineer. I don't nearly have the mind or patience that is required of a full-time, dedicated testing professional. But I talk to them and lots of other people every day, and I'm still a developer by heart, so I feel very kin to everyone involved in the software delivery process.

You don't have to be a tester to understand the constraints of being human. Humans make mistakes and only have so much time per day. Developers who are starting to experience a shift-left in testing responsibilities might make sure that all functions and procedures are unit tested, but often don't go much further simply because there isn't enough time. Not enough people, not enough time...two things I hear an awful lot and part of the reason why I'm in the automated testing tools business with Ready! API.

Side note: I would also like to take this opportunity to say that "no budget" and "not a priority" are the worst reasons not to invest in a better future, one where automation gives you back time to build better software and where getting your delivery pipeline right is critical to moving forward with rapid iteration.

But shouldn't we always test more?

No. Well, yes. Actually, sort of and not always. There's an aspect of diminishing return to introducing more testing in that if you don't prioritize the most business-critical components and workflows to be tested first, you fail your job, your business, and yourself as a professional tester.

We simply can test everything, we just don't have the time, and we also need to stay nimble. If I were a tester, I'd probably get fired for wanting to confirm that what I was told to work on actually affects the bottom line of the business in the most impactful way at this moment in time. I would not sit there creating tests just so that we could report that we have 79% instead of 78% testing coverage today. I would want to see ROI, even though apparently "there ain't no ROI in software testing".

What about all the new stuff, shouldn't that be tested?

I got a question after my talk at APIdays IoT San Francisco 2015 in July: "If you don't know how people are going to combine your devices and APIs, how can you ensure that you've tested everything?" In no disrespectful way, this kind of question underlines the immaturity of the IoT space. We've got a very young startup culture (think 20-somethings with multiple PhDs) attacking big problems, most usually around how the digital world imbues itself back into the physical world, as quickly as possible for as cheap as possible.

There's going to be some broken things, incomplete thoughts, and definitely failures to learn from. I saw this first hand at O'Reilly Solid the week after APIdays too. The speakers were experienced industry leaders, and it was good that those people were on stage speaking to the young engineers building all sorts of IoT devices without a thought to protocol or integration standards. The median booth sponsor was grad school geniuses that are practically DevOps natives building our future...so busy implementing that they may not even know to ask the more important question, "should we"?

What do you have against startups?

Nothing personal, I've been involved in a few myself. The "startup kids" are working incredibly hard, many harder than I've ever worked, to make it or break it in the cutthroat technology market that we've built for ourselves. And when I say "kids", I mean figuratively; folks who bring a variety of backgrounds, skills, and experience to a startup still often have a lot to learn about running a lean operation before things lock in and start whirring.

Some things take time to learn (thanks to our lizard brains) and we (the technology industry) would do well to remember that people are still naturally slow to accept and properly utilize new technologies (like for instance the slow but increasing industry adoption of lightweight service virtualization tools).

If there was one thing I'd take issue with related to startup culture, it's that some important corners can get cut when you're so busy accelerating that you don't have time to ask "should we?". For instance, API security is a huge deal now because of a number of very public API security failures have occurred in the past few years. Moonpig, Uber, Tinder, and others could have avoided this just by applying free knowledge to their designs.

The stakes get bigger too the closer to money and safety your software is, like as in financial and medical institutions, or worse defense and air travel; proactively identifying vulnerabilities to critical services is a fundamental reason for regulatory audits, and keep omissions in safety introduced by startup culture thinking from having a massive negative affect on everyday life for the average digital citizen.

Then how do I know when I've tested enough?

That's up to you, it's your software, it's your customers and their expectations that should help you know when enough is enough. In the absence of formal UX and operational metrics, you may need to set some goals starting with your technical team. You should also be talking directly with your consumers, getting their feedback and input on what they expect from your software.

Go read some James Bach or Michael Bolton books if you want to learn about testing in general, but when it comes down to specifics like SLAs and reporting to management, there are very tangible questions to ask about quality. For instance, a slightly more matured set of questions around testing IoT and other multiples of design complexity (like Microservices, Hypermedia, etc.) might include:

  • Should my goal be to test all the things and permutations?
  • What do the user-experience metrics say about what my customers see as a priority?
  • What technical omissions contribute to our worst business deficits?
  • Do I need to build/test/ship this, right now, at this very moment?
  • What do we gain and/or lose by shipping what we have, such as it is?

Okay, so I don't have to test more?

Don't we all wish; sorry, no, this isn't a free lunch. Yes, you should be testing more, almost universally. Most software vendors and engineers know that automated testing is the only way to achieve high quality with less manual intervention, but haven't seen the pain that a lack of proper testing causes.

Good engineers and technicians have consciences and know when something needs more attention, which is why I leave it up to you to answer the question "how much testing do I need?" You don't need your inner voice telling you that you need to test and monitor something that only you know would cripple your business if it went down for any amount of time. So in those moments, I humbly submit to you that if you see or feel the future pain, it is your responsibility to write the test, or at least put it in the priority backlog, or something that your team is sure to follow up on.

Often backlogged is "test coverage", a useful assessment tool, but also just a broad-spectrum approach to getting visibility over technical gaps in software delivery. It helps you see when new things are implemented without the proper team communication, but should not be a goal on its own right. When was the last time your customer said "if you don't have at least 82% testing coverage, I don't want to use your software"? Right. Most consumers care about the downstream effects of bad software like app crashes, not the things you do to prevent bugs from getting to production. Coverage is not a golden idol, it's just a method of assessing where your weaknesses lie.

Like security testing, test coverage is a measurement of only one aspect of readiness that should be part of a larger perspective on customer-focused software quality. The connection between testing and customer value shows up in other areas like beta acceptance testing, performance monitoring, and consumer feedback. It's there for the taking when we want it.

That's my experience with testing "all the things", but what's yours? I'd love to hear your thoughts.

Read the original blog entry...

More Stories By SmartBear Blog

As the leader in software quality tools for the connected world, SmartBear supports more than two million software professionals and over 25,000 organizations in 90 countries that use its products to build and deliver the world’s greatest applications. With today’s applications deploying on mobile, Web, desktop, Internet of Things (IoT) or even embedded computing platforms, the connected nature of these applications through public and private APIs presents a unique set of challenges for developers, testers and operations teams. SmartBear's software quality tools assist with code review, functional and load testing, API readiness as well as performance monitoring of these modern applications.

@ThingsExpo Stories
SYS-CON Events announced today that Daiya Industry will exhibit at the Japanese Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ruby Development Inc. builds new services in short period of time and provides a continuous support of those services based on Ruby on Rails. For more information, please visit https://github.com/RubyDevInc.
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
As businesses evolve, they need technology that is simple to help them succeed today and flexible enough to help them build for tomorrow. Chrome is fit for the workplace of the future — providing a secure, consistent user experience across a range of devices that can be used anywhere. In her session at 21st Cloud Expo, Vidya Nagarajan, a Senior Product Manager at Google, will take a look at various options as to how ChromeOS can be leveraged to interact with people on the devices, and formats th...
SYS-CON Events announced today that Taica will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Taica manufacturers Alpha-GEL brand silicone components and materials, which maintain outstanding performance over a wide temperature range -40C to +200C. For more information, visit http://www.taica.co.jp/english/.
SYS-CON Events announced today that SourceForge has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SourceForge is the largest, most trusted destination for Open Source Software development, collaboration, discovery and download on the web serving over 32 million viewers, 150 million downloads and over 460,000 active development projects each and every month.
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities – ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups. As a result, many firms employ new business models that place enormous impor...
SYS-CON Events announced today that TidalScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale is the leading provider of Software-Defined Servers that bring flexibility to modern data centers by right-sizing servers on the fly to fit any data set or workload. TidalScale’s award-winning inverse hypervisor technology combines multiple commodity servers (including their ass...
As popularity of the smart home is growing and continues to go mainstream, technological factors play a greater role. The IoT protocol houses the interoperability battery consumption, security, and configuration of a smart home device, and it can be difficult for companies to choose the right kind for their product. For both DIY and professionally installed smart homes, developers need to consider each of these elements for their product to be successful in the market and current smart homes.
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, will lead you through the exciting evolution of the cloud. He'll look at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering ...
As hybrid cloud becomes the de-facto standard mode of operation for most enterprises, new challenges arise on how to efficiently and economically share data across environments. In his session at 21st Cloud Expo, Dr. Allon Cohen, VP of Product at Elastifile, will explore new techniques and best practices that help enterprise IT benefit from the advantages of hybrid cloud environments by enabling data availability for both legacy enterprise and cloud-native mission critical applications. By rev...
SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp emp...
SYS-CON Events announced today that TidalScale, a leading provider of systems and services, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale has been involved in shaping the computing landscape. They've designed, developed and deployed some of the most important and successful systems and services in the history of the computing industry - internet, Ethernet, operating s...
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
Infoblox delivers Actionable Network Intelligence to enterprise, government, and service provider customers around the world. They are the industry leader in DNS, DHCP, and IP address management, the category known as DDI. We empower thousands of organizations to control and secure their networks from the core-enabling them to increase efficiency and visibility, improve customer service, and meet compliance requirements.
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
Amazon is pursuing new markets and disrupting industries at an incredible pace. Almost every industry seems to be in its crosshairs. Companies and industries that once thought they were safe are now worried about being “Amazoned.”. The new watch word should be “Be afraid. Be very afraid.” In his session 21st Cloud Expo, Chris Kocher, a co-founder of Grey Heron, will address questions such as: What new areas is Amazon disrupting? How are they doing this? Where are they likely to go? What are th...