Click here to close now.

Welcome!

Java Authors: AppDynamics Blog, Kelly Murphy, Pat Romanski, Liz McMillan, Elizabeth White

Related Topics: @ThingsExpo, Java, Linux, Web 2.0, Cloud Expo, Big Data Journal

@ThingsExpo: Article

Adventures in Digital Real-Time Wonderland

The only way we’ll truly achieve real-time behavior is by understanding the connections the wonderland of real-time requires

In last week’s Forbes article I discussed various senses of the term real-time: low latency user interfaces, up-to-date information, live human interactions, and high-performance data processing – to name but a few. Today, for the Cortex audience (as well as the Wired Innovations and SYS-CON audiences), it’s time to channel Lewis Carroll and have a wondrous adventure to shed light on the true significance and challenges of real-time.

drinkmeAs we venture down the rabbit hole of our technology-infused world, it’s easy to see that everything is getting faster and bigger and, well, just more. Moore’s Law has just taken a big swig from a bottle that says drink me, as we have more memory, more storage, faster networks, more network-connected doodads, more and faster processors than ever before. And of course, we’re also getting better at everything: better apps. Better operating systems. Better ways of abstracting every element of our environment to provide even greater performance and flexibility. We are truly living in a time of plenty, if not excess. It’s no wonder that we want real-time in everything we do.

Today’s challenge isn’t only making stuff go faster. It’s figuring out how all the acceleration of all the bits and pieces fit together. Furthermore, this push to achieve a holistic perspective on all this gear drives our quest for real-time, as it only takes one bottleneck to slow everything else down. The only way we’ll truly achieve real-time behavior is by understanding the connections this wonderland of real-time requires. So let’s get started, or my ears and whiskers, we’ll be late!

rabbitReal-Time Starting Point: Reactive Programming

Google “real-time.” Right after Bill Maher’s HBO show and a general Wikipedia entry comes Wikipedia’s “real-time computing” page – presumably the real-time we’re talking about here. Load that page and you’ll notice two curiouser and curiouser facts right off the bat: first, the Wikipedia article itself has serious issues – as though no one who cares about real-time computing actually wants anybody else to understand it. Second, “real-time computing” is apparently synonymous with “reactive computing,” a much less familiar term. The rest of the article focuses on the sort of real-time we want from our antilock brakes – useful to be sure, but not the enterprise context we were looking for.

caterpillarMaybe reactive computing is closer to the mark? Well, there’s inexplicably no Wikipedia page for that. The closest we can come is the reactive computing mock turtle: reactive programming. The basic idea with reactive programming is that the behavior of pieces of software can be declaratively defined, and thus evaluated in real-time – just as spreadsheet cells update automatically when a value they refer to changes.

There’s more to the reactive story than software that updates automatically, however, as a visit to the reactive manifesto illustrates. This wise caterpillar of a manifesto calls out four key reactive traits: event-driven, scalable, responsive, and resilient – essentially calling for Cloud-friendly, event-driven architectures that have the declarative behavior definition we know and love from the spreadsheet – only now across a hybrid enterprise context. Mushroom, anyone?

catIt’s no coincidence that Bloomberg Agile Architecture™ (BAA) also calls out responsiveness and resilience, although the BAA contexts for these terms are aspects of business agility rather than software – but suffice it to say, if your software doesn’t have these traits, it’s unlikely your organization is agile. Alas, we thought we saw the Cheshire Cat of agility, but all we saw was its smile. The people behind the reactive manifesto, however, have a far more technical context for these terms – Play Framework, an open source web application framework for Java and Scala that bills itself as lightweight, stateless, and Web-friendly.

walrusAt this point this Cortex might have gone down the Scala rabbit hole – but I’ll save that for a future issue (Through the Looking Glass, perhaps?). Just for fun, however, let’s follow the stateless thread of this adventure to the beach where the Walrus and the Carpenter entertain their oysters. I’ve discussed statelessness over the years in many contexts, from the challenge of maintaining business process instance state with stateless Services, to the role hypermedia play in transferring state to the client if you actually follow REST properly (which almost no one does), to the challenges state management presents to Cloud-based applications. Understanding the relationship between statelessness and real-time behavior, however, ties all these concepts together in a nice package. The oysters, however, aren’t nearly so satisfied.

tartsState, in fact, is the Queen’s tarts of real-time computing. Sure, sometimes your software behavior can be completely reactive: event happens, do some stuff, give some kind of result, and never keep track of anything or wait around for somebody else to finish something. Such processing can be blisteringly fast, of course. It’s when you have to keep track of something that problems arise: where do you do the tracking? Do you have to keep track of multiple things at once when they might interact somehow? How permanent does the tracking have to be? And most importantly: won’t all this tracking slow everything down?

Time to hide the tarts: we could simply keep track of everything in the database. We get unlimited persistence, but databases are relatively slow and scaling them can be difficult. So let’s call upon the knave of hearts to spirit away those tarts to some piece of infrastructure in our middle tier, like an application server or an ESB. The database breathes a sigh of relief, but now we have a Cloud-unfriendly centralized state management approach. So instead, let’s pass the tarts to the client – after all, that’s where REST got its name (Representational State Transfer, natch). We now have scalability and Cloud friendliness, but this approach doesn’t deal well with shared state (as we would need for any type of collaborative application), and nobody likes HATEOAS, even when they understand it.

croquetEnter caching. The idea of a cache is to temporarily store those pesky state tarts, thus lightening the load on the persistence tier. And calloo callay! We can now cache in memory, making it wicked fast. But we still have the Cloud-friendliness problem, so enter from the Queen’s croquet pitch the distributed in-memory cache. Cloud-friendly, check. Wicked fast, check. Responsive? Well, it depends on the color of the roses. The problem here, of course, is the problem caches always have: if all your data are always changing or every interaction always requires different data, caches are worse than useless, since caching something only makes sense if somebody is going to use it a few times before you need to refresh it.

At this point there’s only one more place for the tarts to go: back to the database. We need faster databases that are both Cloud-friendly and deal well with the continually exploding nature of Big Data. It’s no wonder, therefore, that the database marketplace is undergoing a dramatic period of innovation. Yes, another rabbit hole for yet another day – but let’s tease out a single architectural tea party that relates directly to real-time: immutability.

hatterYour mad hatter of a database is immutable if it only supports writes that append data but no updates or deletes. Instead, to handle these pesky changes, additional records are added that indicate a previous record has changed. Immutability is essential for solving some knotty problems with concurrency – a mischievous dormouse for distributed computing since the client/server days and still a hassle in today’s Cloud-enabled world. As anyone who has used GitHub or a similar immutable data store can attest, immutability is the key to scaling a database that supports a large number of users who can add information, since all changes are handled as new data, and furthermore, the data store maintains a complete audit trail of everything that has ever happened, regardless of whether we all move down one seat at the table in search of clean dishes.

dormouseGitHub additionally works well with caching because it must assemble the current version of each stored file by adding together all the changes, or diffs, to that file. Temporary storage of each current version thus lightens the load on the underlying data store. But in other situations where the underlying data are always in flux, immutability still helps to address the real-time need. Reads can be extraordinarily fast compared to traditional databases, because the database can look to the index to identify the latest version of a record.

And so our adventure through real-time computing brings us to indexing in all its glory – not just for finding the desired record, but also for all the metadata necessary to assemble the various diffs in order to deliver the current version of a record in real-time. The metadata story for real-time, however, doesn’t stop at indices, as the army of metadata playing cards are central to the notion of declarative programming.

cardsWe have thus come full circle to the notion of reactive programming, which includes declaratively defining the behavior of pieces of software as simply as entering formulas into cells in a spreadsheet. And while a single worksheet may have tens of thousands of cells, extending the role metadata play to a distributed enterprise context ups the ante on the relationship between reactive programming and metadata: being able to resolve the desired behavior of any software given the combination of all metadata in the relevant environment – for every interaction, in real-time.

We call such resolution dynamic constraint satisfaction, where the metadata describe the relevant constraints, even though they may be fully dynamic. Calculating the result, therefore, must take place in real-time. Envision one massive spreadsheet, only instead of formulas in the cells, you have any reactive software you might find anywhere in your IT environment. The cell with your final answer is always correct, and always up to date – in real-time. Off with their heads!

The Intellyx Take

Our adventure down the real-time rabbit hole in this enterprise IT wonderland took us many different places. And while each of the critters we met had its own real-time story, our adventure tied all the individual stories together. Such is the nature of real-time: we have many moving parts and they must all be working at top form together in order to deliver a true real-time experience to each user.

Real-time behavior, therefore, is an important challenge for any digital professional, as there is more to digital transformation than meets the eye. Your customers, partners, and broader audience expect such behavior from your digital efforts, and to keep them happy you need the right technology and most importantly, the right architecture to glue everything together in real-time.

More Stories By Jason Bloomberg

Jason Bloomberg is the leading expert on architecting agility for the enterprise. As president of Intellyx, Mr. Bloomberg brings his years of thought leadership in the areas of Cloud Computing, Enterprise Architecture, and Service-Oriented Architecture to a global clientele of business executives, architects, software vendors, and Cloud service providers looking to achieve technology-enabled business agility across their organizations and for their customers. His latest book, The Agile Architecture Revolution (John Wiley & Sons, 2013), sets the stage for Mr. Bloomberg’s groundbreaking Agile Architecture vision.

Mr. Bloomberg is perhaps best known for his twelve years at ZapThink, where he created and delivered the Licensed ZapThink Architect (LZA) SOA course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, the leading SOA advisory and analysis firm, which was acquired by Dovel Technologies in 2011. He now runs the successor to the LZA program, the Bloomberg Agile Architecture Course, around the world.

Mr. Bloomberg is a frequent conference speaker and prolific writer. He has published over 500 articles, spoken at over 300 conferences, Webinars, and other events, and has been quoted in the press over 1,400 times as the leading expert on agile approaches to architecture in the enterprise.

Mr. Bloomberg’s previous book, Service Orient or Be Doomed! How Service Orientation Will Change Your Business (John Wiley & Sons, 2006, coauthored with Ron Schmelzer), is recognized as the leading business book on Service Orientation. He also co-authored the books XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996).

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting).

@ThingsExpo Stories
The true value of the Internet of Things (IoT) lies not just in the data, but through the services that protect the data, perform the analysis and present findings in a usable way. With many IoT elements rooted in traditional IT components, Big Data and IoT isn’t just a play for enterprise. In fact, the IoT presents SMBs with the prospect of launching entirely new activities and exploring innovative areas. CompTIA research identifies several areas where IoT is expected to have the greatest impact.
Every day we read jaw-dropping stats on the explosion of data. We allocate significant resources to harness and better understand it. We build businesses around it. But we’ve only just begun. For big payoffs in Big Data, CIOs are turning to cognitive computing. Cognitive computing’s ability to securely extract insights, understand natural language, and get smarter each time it’s used is the next, logical step for Big Data.
The 4th International Internet of @ThingsExpo, co-located with the 17th International Cloud Expo - to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA - announces that its Call for Papers is open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
There's no doubt that the Internet of Things is driving the next wave of innovation. Google has spent billions over the past few months vacuuming up companies that specialize in smart appliances and machine learning. Already, Philips light bulbs, Audi automobiles, and Samsung washers and dryers can communicate with and be controlled from mobile devices. To take advantage of the opportunities the Internet of Things brings to your business, you'll want to start preparing now.
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo in Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal an...
P2P RTC will impact the landscape of communications, shifting from traditional telephony style communications models to OTT (Over-The-Top) cloud assisted & PaaS (Platform as a Service) communication services. The P2P shift will impact many areas of our lives, from mobile communication, human interactive web services, RTC and telephony infrastructure, user federation, security and privacy implications, business costs, and scalability. In his session at @ThingsExpo, Robin Raymond, Chief Architect at Hookflash, will walk through the shifting landscape of traditional telephone and voice services ...
The 17th International Cloud Expo has announced that its Call for Papers is open. 17th International Cloud Expo, to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, APM, APIs, Microservices, Security, Big Data, Internet of Things, DevOps and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal today!
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at Internet of @ThingsExpo, James Kirkland, Chief Architect for the Internet of Things and Intelligent Systems at Red Hat, described how to revolutioniz...
All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades. With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo, June 9-11, 2015, at the Javits Center in New York City. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be
The security devil is always in the details of the attack: the ones you've endured, the ones you prepare yourself to fend off, and the ones that, you fear, will catch you completely unaware and defenseless. The Internet of Things (IoT) is nothing if not an endless proliferation of details. It's the vision of a world in which continuous Internet connectivity and addressability is embedded into a growing range of human artifacts, into the natural world, and even into our smartphones, appliances, and physical persons. In the IoT vision, every new "thing" - sensor, actuator, data source, data con...
SYS-CON Events announced today that DragonGlass, an enterprise search platform, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. After eleven years of designing and building custom applications, OpenCrowd has launched DragonGlass, a cloud-based platform that enables the development of search-based applications. These are a new breed of applications that utilize a search index as their backbone for data retrieval. They can easily adapt to new data sets and provide access to both structured and unstruc...
Container frameworks, such as Docker, provide a variety of benefits, including density of deployment across infrastructure, convenience for application developers to push updates with low operational hand-holding, and a fairly well-defined deployment workflow that can be orchestrated. Container frameworks also enable a DevOps approach to application development by cleanly separating concerns between operations and development teams. But running multi-container, multi-server apps with containers is very hard. You have to learn five new and different technologies and best practices (libswarm, sy...
Buzzword alert: Microservices and IoT at a DevOps conference? What could possibly go wrong? In this Power Panel at DevOps Summit, moderated by Jason Bloomberg, the leading expert on architecting agility for the enterprise and president of Intellyx, panelists will peel away the buzz and discuss the important architectural principles behind implementing IoT solutions for the enterprise. As remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud environment, and we must architect and code accordingly. At the very least, you'll have no problem fil...
There's Big Data, then there's really Big Data from the Internet of Things. IoT is evolving to include many data possibilities like new types of event, log and network data. The volumes are enormous, generating tens of billions of logs per day, which raise data challenges. Early IoT deployments are relying heavily on both the cloud and managed service providers to navigate these challenges. In her session at Big Data Expo®, Hannah Smalltree, Director at Treasure Data, discussed how IoT, Big Data and deployments are processing massive data volumes from wearables, utilities and other machines...
SYS-CON Events announced today that MetraTech, now part of Ericsson, has been named “Silver Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. Ericsson is the driving force behind the Networked Society- a world leader in communications infrastructure, software and services. Some 40% of the world’s mobile traffic runs through networks Ericsson has supplied, serving more than 2.5 billion subscribers.
The worldwide cellular network will be the backbone of the future IoT, and the telecom industry is clamoring to get on board as more than just a data pipe. In his session at @ThingsExpo, Evan McGee, CTO of Ring Plus, Inc., discussed what service operators can offer that would benefit IoT entrepreneurs, inventors, and consumers. Evan McGee is the CTO of RingPlus, a leading innovative U.S. MVNO and wireless enabler. His focus is on combining web technologies with traditional telecom to create a new breed of unified communication that is easily accessible to the general consumer. With over a de...
Disruptive macro trends in technology are impacting and dramatically changing the "art of the possible" relative to supply chain management practices through the innovative use of IoT, cloud, machine learning and Big Data to enable connected ecosystems of engagement. Enterprise informatics can now move beyond point solutions that merely monitor the past and implement integrated enterprise fabrics that enable end-to-end supply chain visibility to improve customer service delivery and optimize supplier management. Learn about enterprise architecture strategies for designing connected systems tha...
Cloud is not a commodity. And no matter what you call it, computing doesn’t come out of the sky. It comes from physical hardware inside brick and mortar facilities connected by hundreds of miles of networking cable. And no two clouds are built the same way. SoftLayer gives you the highest performing cloud infrastructure available. One platform that takes data centers around the world that are full of the widest range of cloud computing options, and then integrates and automates everything. Join SoftLayer on June 9 at 16th Cloud Expo to learn about IBM Cloud's SoftLayer platform, explore se...
SYS-CON Media announced today that 9 out of 10 " most read" DevOps articles are published by @DevOpsSummit Blog. Launched in October 2014, @DevOpsSummit Blog offers top articles, news stories, and blog posts from the world's well-known experts and guarantees better exposure for its authors than any other publication. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce softw...
15th Cloud Expo, which took place Nov. 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA, expanded the conference content of @ThingsExpo, Big Data Expo, and DevOps Summit to include two developer events. IBM held a Bluemix Developer Playground on November 5 and ElasticBox held a Hackathon on November 6. Both events took place on the expo floor. The Bluemix Developer Playground, for developers of all levels, highlighted the ease of use of Bluemix, its services and functionality and provide short-term introductory projects that developers can complete between sessions.