Welcome!

Java IoT Authors: Stackify Blog, APM Blog, Liz McMillan, Elizabeth White, Yeshim Deniz

Related Topics: FinTech Journal, Java IoT, @CloudExpo, @DevOpsSummit

FinTech Journal: Blog Feed Post

Which API Metrics Even Matter? By @JustinRohrman | @CloudExpo #API #Cloud

Metrics can be dangerous, but we can also use them to build a more complete story about our software

Which API Metrics Even Matter?
by Justin Rohrman

Software Measurement is a tricky business. When we all agree on what the thing we are trying to count is, like pieces of candy in a jar, it is simple. Sadly, software isn't candy in a jar.

We try to count things like bugs, or lines of code, of number of stories completed in a sprint but never agree on what any of those things are in the first place. I have been in trouble more than once in my role as a tester from having my performance judged based on number of bugs I reported in a sprint, or the number of test cases written for a release.

Metrics can be dangerous, but we can also use them to build a more complete story about our software.

Let's take a look at how metrics and APIs can go together.

User Data
Learning about our customers - how they work, how software will help them be better, and what they value - is a good start for building a product. I like knowing that we are creating new features for real people instead of personas or descriptions on a story card. There are usually important things missing though, even when I have been able to go straight to the source. Getting feedback once that code is in production can be a whole other challenge.

Gathering data from API monitoring tools about what our customers are doing can help me collect information I might have never found.

There are a few couple of data points I have found to be useful - traffic source, and user type. Most companies I have worked with in the past several years have had some sort of mobile app. Maybe it wasn't the main line of business, but it was there and it was important. Traffic source data was a way for us to see how many people were using what parts of the product, and at what points they went from mobile to desktop. When we combined that with user type, we focused in a little more on who was spending the most time where.

Software Performance
This is one place where the magic of software monitoring tools comes in to play. An API, or at least one that gets good usage, throws back a constant stream of data in the form of a HTTP response. And sometimes more than that if the developers design for deeper feedback.

There are a lot of different types of data we can be collect related to performance - server data like available memory and CPU cycles, load time data and latency in the browser, and database response time. All of these are part of the larger puzzle that is how fast or slow your product is responding. Each time I run a performance scenario and notice a difference, the big questions are how big is the change and does it matter. Sometimes the answer is the change doesn't matter at all. Collecting that data gives us the choice to make those decisions instead of waiting for the customer to tell us when something is wrong.

Imagine you are running a Black Friday ad a few days before Thanksgiving day try to drive people into your stores instead of your competitors'. You want that page to be fast enough that your potential customers don't give up but checking the site every few minutes isn't realistic.

API Metrics

Over the course of that few days, we can check metrics like median throughput, API call volume, and throughput distribution by endpoint can help shape our understanding. This information might point to one Amazon ec2 instance running slow that needs load balancing, or maybe a virtual machine that needs better memory allocation. Either way, the data collection may have saved the ad from being inaccessible at a bad time.

Software performance is a tricky thing. There are a lot of different aspects to think about like the database, server resources, code optimization, and technologies used in the user interface. With performance testing efforts, the feedback loop is usually at the release level at best. So every two or more weeks, new information about how performance has changed comes into view. We get that information in the middle of a probably already stacked sprint, so the earliest something gets done is the next spring. That is a long lag between discovery and action.

Error Data

I test APIs so that hopefully when we push to production everyone is happy and nothing unexpected happens. Software is tricky though, we can never imagine all the ways our customers will use our products and the ways things might go wrong. And so occasionally things do go wrong, pages don't submit when someone clicks the save button and lists of data fail to load. One of the nice things about APIs is that stream of data we get back. Each time a page fails to submit, or a list of data doesn't load correctly we get some data back.

Seeing an error once might be a fluke, seeing an error several times or at the same time every day might be something interesting to look into.

Collecting data on what has gone wrong can make people feel defensive. I have had more than one development manager that was interested in collecting information on problems found in production. The idea was that they could get a better understanding of how well the testers were working out with this data. If there weren't many bugs n production then testers were doing alright, if there were a lot then someone was probably getting an unpleasant talk.

Collecting error data can be dangerous in the type of environment where it is wielded like a weapon. My personal preference is to use it as a tool for inquiry and investigation. An upward trend in error reporting can trigger studying our process, and our product to figure out how to make things better next time around.

Probably the easiest way to take metrics from complicated and dangerous to helpful is to use them as a trigger to start a deeper investigation. Seeing trends in user interface load times for a page getting slower? Set up a project to look at the page and see where it can be optimized. If error reporting gets really active at a certain time every day, maybe we should be making a call to the customer to learn about how they are using the product and see how we can better accommodate that.

Everything is information, we just need to make it useful.

Read the original blog entry...

More Stories By SmartBear Blog

As the leader in software quality tools for the connected world, SmartBear supports more than two million software professionals and over 25,000 organizations in 90 countries that use its products to build and deliver the world’s greatest applications. With today’s applications deploying on mobile, Web, desktop, Internet of Things (IoT) or even embedded computing platforms, the connected nature of these applications through public and private APIs presents a unique set of challenges for developers, testers and operations teams. SmartBear's software quality tools assist with code review, functional and load testing, API readiness as well as performance monitoring of these modern applications.

@ThingsExpo Stories
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...