Welcome!

Java IoT Authors: Automic Blog, Liz McMillan, Elizabeth White, Pat Romanski, Roger Strukhoff

Related Topics: Java IoT

Java IoT: Article

Java Feature — Building Real-Time Applications with Continuous Query Technology

The promise of a robust new development model

The client/server development model prevalent in the mid-1990's resulted in extremely easy-to-build rich GUI applications that interacted directly with a relational database. 4GL tools such as Visual Basic and PowerBuilder let even junior developers visually compose both the presentation and most of the backend data binding. While this made for impressive Rapid Application Development (RAD) productivity, the client/server architecture was severely challenged when dealing with real-time environments where the data changes rapidly and applications require visibility to the correct data at all times. As a result, client applications were forced to poll the database continuously to check for changes.

The same is true in today's browser-based or Java Swing-based multi-tier applications, where the user is forced to issue a screen refresh to view the latest state. Real-time applications such as a trader desktop where the screens are continuously refreshed are still sophisticated proprietary applications that require specialized application design to push events from backend servers to the GUI clients. Such applications result in hundreds or even thousands of views like this: Maintain a continuous view of all Intel and Dell orders placed today and notify me when AMD moves up or down by 5%.

However, today a new "push-based" architecture enables data changes to be monitored continuously in a backend data management system and changes continuously pushed to client applications, maintaining a real-time view at all times.

Traditional Databases Are Passive
Most complex GUI screens use complex SQL - multi-table joins, column aggregations, and multiple predicates for filtering, grouping, etc. to construct the dataset being viewed. Consider a real-time application like a financial stock monitoring program or a traffic management system with hundreds of concurrent clients with equally large numbers of complex queries that are continuously being executed once every second. The traditional relational database that's built for storing data efficiently and guaranteeing consistency won't be able to cope with this demand. Relational databases are passive, executing queries on sitting data only. Today's complex applications, however, require a system that can very efficiently execute queries as data streams in.

A SQL Continuous Query Engine - An Active Data Management System
By building a database engine designed specifically so queries can remain standing and active - or continuous - the scalability, reactivity, and organization of multi-tiered data-centric applications can be radically altered. Continuous queries (CQ) let users get new results from a database without having to issue the same query repeatedly.

Queries no longer have to be reissued to refresh result sets, logic that has to execute in response to complex changes in a data model can actively register interest directly from the source, and business logic can be safely co-located with application data in a relational model without scalability limitations.

Continuous querying technology works through an engine that efficiently groups and filters predicates from large numbers of queries, enabling several key things to happen:

  • When the server first gets a continuous query, it not only replies with an initial result set, but it analyzes the query predicates (selection criteria) to group it logically with other similar queries.
  • The engine can then quickly identify what continuous queries are affected by any given data modification (an insert, update, or delete against the relationally structured operational data).
  • The engine can send only the deltas to each CQ client needed to update its existing result set, in effect exactly the data necessary for the client to hold a materialized view of data from the server.
The inherent power of this technology lies in both its simplicity and natural scalability. With an in-memory database and some very simple extensions to existing query languages, we're suddenly capable of building a middle-tier that combines all of a database's operational benefits and none of its limitations. At the same time, we can build application server clients that express interest in data through conventional queries without having to trade performance for data currency or functional sophistication for development effort.

Where Does It Apply?
While the focus of this article is to illustrate the power of CQ to provide real-time view maintenance in graphical user interfaces, the power of this paradigm is well beyond this. Continuous querying is part of a new data management paradigm called stream data processing (see the References section below for further information) and can be used to monitor multiple streaming sources of data, analyze these streams for patterns of interest, and respond instantaneously. The sources of the data could be disparate - RFID sensor events, events from business applications across an enterprise, external sources, etc.

If the applicability of the technology were to be characterized in two points they would be:

  1. Data is changing very rapidly and decisions have to be made instantaneously.
  2. The system can analyze hundreds to thousands of patterns (rules or query predicates) with thousands of events pouring in every second.
Leveraging CQ in a Financial Order Tracking System
A great example of a system with stringent real-time data requirements is a securities trading order-tracking system. Diverse event streams such as customer orders, order executions, and market data quotes must be combined into a continuously updated view provided to multiple end users. Early implementations - built at a time of much smaller order volumes and slower workflows - used client/server architectures that required GUIs to poll the database for updates.

As trading volumes grew and firms realized that they could gain a competitive advantage with faster trading systems, we started seeing trading application servers that could publish real-time updates to trader GUIs. Implementing this functionality efficiently - with predictable low-latency and high throughput - required a much more sophisticated development model than client/server could provide and so the effort to build, maintain, and operate these systems grew quickly. The example in Figure 1 strips functionality to its bare essentials to illustrate the relative merits of conventional and CQ architectures.

Data Model
For simplicity's sake, our example uses only two tables - Orders and Quotes, as shown in Table 1.

The Orders table simply tracks the orders clients have sent you and the number of shares filled in each one. The Quotes table keeps track of market activity for any symbol that exists in the Orders table, so the GUI displays market activity associated with the order to the responsible broker. This means that we have three event streams coming into the system: orders from clients, order fills from a stock exchange, and market data quotes from a quote feed. These three event streams must be coalesced into a coherent view within the Order Tracking System. Naturally, in the real world these systems are much more sophisticated - requiring transactions, many more data entities and streams, and complex user interactions, but we'll keep things simple and explain later how the example may be extrapolated to additional complexity.

We'll describe the components and options for this use case in three sections. The first shows how to build a simple database publisher application for quotes and orders. The second shows how you might build a client application using plain JDBC, or with the addition of database and JMS queues, and third how you'd build a client application with Continuous Querying. Note that the full source code and pre-configured runtime configurations for the DataPublisher CQExampleClient applications are available for download at www.gemstone.com/download/.

A Simple Data Publisher for an Order Tracking System
The Data Publisher application is quite straightforward. It instantiates Order and Quote generation simulators and handles their events by "publishing" them in a JDBC database with SQL Insert and Update statements.

Let's take a closer look at the Publisher application by inspecting some of its code. The classes we need to understand are SimpleDataPublisher, SimpleQuoteGenerator, and SimpleOrderGenerator.

  • SimpleOrderGenerator has a pre-defined list of Orders and Customers. Its constructor has only one argument - a listener that implements the methods onNewOrder(SimpleOrder o) and onOrderUpdate(SimpleOrder o). In the constructor it creates a basket of 63 orders (which results in onNewOrder() callbacks), and then spawns a thread to gradually fill the orders by incrementing the fillQty property of each one.
  • SimpleQuoteGenerator generates simple market data quote streams in response to calls to its addSymbol (String symbol) method. It's constructed with a listener that implements onQuote ( SimpleQuote q ), and it spawns a thread that randomly updates each Symbol's quote at a steady rate.
SimpleDataPublisher connects to a JDBC database with a JDBC driver, either creates or clears out the Quote and Order tables with DDL/DML statements, and then uses the two generator classes as event sources. Each generator callback event (onQuote(), onNewOrder() and onOrderUpdate()) fires logic to perform a SQL INSERT or UPDATE to the CQ server, thus creating or modifying records in the Quote and Order tables. It contains only the three following members:

Connection c = null; // The JDBC Connection to the CQ Server
SimpleQuoteGenerator quoteGenerator = new SimpleQuoteGenerator ( this );
SimpleOrderGenerator orderGenerator = new SimpleOrderGenerator ( this );

The constructor of SimpleDataPublisher looks like Listing 2.

Using a handy helper class to hide the boring details, the code gets a handle on a valid JDBC Connection object, creates or clears the necessary tables, initializes the OrderGenerator (which causes an onNewOrder() callback for each new order), and then starts the QuoteGenerator and OrderGenerator threads to initiate the event streams.

Note that for each new order created, SimpleDataPublisher calls the SimpeQuoteGenerator.addSymbol() method to register a new Symbol for quote generation - thus making sure that we get quotes for each symbol handled by system (duplicate symbols are ignored).

Each time SimpleDataPublisher gets a callback invocation, it creates a SQL statement and submits it to the CQ Engine. For example, when onUpdatedOrder() fires, the logic in Listing 1 executes:

You can see that this is a very simple O/R mapping exercise plus JDBC database interaction logic. The logic that executes in response to onQuote() and onNewOrder() is fundamentally the same - and uses SQL/JDBC to update or insert a table record. Once started, SimpleExamplePublisher continues to update the order and quote records in response to the callbacks it's getting from the generators. This is a good time to reflect on the how this fits with your existing experience - the type of code above already exists in most systems, but how often does it serve as BOTH a database update and a notification to systems that have previously queried the updated record?


More Stories By Gideon Low

Gideon Low is senior technical architect for GemStone Systems. He has over 10 years of experience in the development, management, and sales of high-performance large-scale distributed systems. His focus over the last seven years has been in high-speed electronic trading systems, with roles including CTO at Silicon Summit Technologies (A FIX/OMS vendor) and VP, Client Connectivity Technology at Lehman Brothers.

More Stories By Jags Ramnarayan

Jags Ramnarayan is the chief architect for the high-performance, distributed data management product line at GemStone Systems. He puts on multiple hats - evangelizing the technology, exploring requirements with customers, and managing the overall direction of the architecture decisions. In the past Jags participated in several Java and W3C standards for GemStone and BEA. On the side, Jags is also pursuing a MBA degree, but hopes to remain technically focussed.

Comments (5) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
BR 09/18/06 11:46:11 AM EDT

The connection API discussed here is JDBC (a well known standard), the query language is SQL (an uber standard that can support joins and other complicated functions). Yes, the callnack CQ fucntionality requires a proprietary API.

With Tangosol, connection, query language and CQ call-backs are all handled via proprietary APIs, and the query langauge is more of a filtering language.

Tangosol 09/17/06 08:44:01 PM EDT

Yes, Continuous Query in Tangosol Coherence requires the use of an API designed for Continuous Query, since there is no standard API in Java to support Continuous Query. The product covered in the article requires the use of a proprietary API to use its own continuous query features, and that seems perfectly acceptable in the absence of a standard.

BR 08/29/06 04:40:52 PM EDT

But, Tangosol's continuous query requires the use of a proprietary API (not as intuitive as SQL) and by no means can accomodate the complex conditions/predicates that can be modeled with a SQL based continuous query paradigm. Seems like a classic marketing/"me too" strategy from Tangosol..:-)

AJAXWorld News Desk 08/28/06 05:56:14 PM EDT

The client/server development model prevalent in the mid-1990's resulted in extremely easy-to-build rich GUI applications that interacted directly with a relational database. 4GL tools such as Visual Basic and PowerBuilder let even junior developers visually compose both the presentation and most of the backend data binding. While this made for impressive Rapid Application Development (RAD) productivity, the client/server architecture was severely challenged when dealing with real-time environments where the data changes rapidly and applications require visibility to the correct data at all times. As a result, client applications were forced to poll the database continuously to check for changes.

Tangosol 08/17/06 03:00:39 PM EDT

Tangosol Coherence also provided Continuous Query, including Continuous Query Caching.

@ThingsExpo Stories
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, discussed how they built...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, examined the regulations and provided insight on how it affects technology, challenges the established rules and will usher in new levels of diligence arou...
The 22nd International Cloud Expo | 1st DXWorld Expo has announced that its Call for Papers is open. Cloud Expo | DXWorld Expo, to be held June 5-7, 2018, at the Javits Center in New York, NY, brings together Cloud Computing, Digital Transformation, Big Data, Internet of Things, DevOps, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
No hype cycles or predictions of a gazillion things here. IoT is here. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, an Associate Partner of Analytics, IoT & Cybersecurity at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He also discussed the evaluation of communication standards and IoT messaging protocols, data...
22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud ...
22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud ...
DevOps at Cloud Expo – being held June 5-7, 2018, at the Javits Center in New York, NY – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Among the proven benefits,...
@DevOpsSummit at Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, is co-located with 22nd Cloud Expo | 1st DXWorld Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait...
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
SYS-CON Events announced today that T-Mobile exhibited at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on qua...
SYS-CON Events announced today that Cedexis will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Cedexis is the leader in data-driven enterprise global traffic management. Whether optimizing traffic through datacenters, clouds, CDNs, or any combination, Cedexis solutions drive quality and cost-effectiveness. For more information, please visit https://www.cedexis.com.
SYS-CON Events announced today that Google Cloud has been named “Keynote Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Companies come to Google Cloud to transform their businesses. Google Cloud’s comprehensive portfolio – from infrastructure to apps to devices – helps enterprises innovate faster, scale smarter, stay secure, and do more with data than ever before.
SYS-CON Events announced today that Vivint to exhibit at SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California. As a leading smart home technology provider, Vivint offers home security, energy management, home automation, local cloud storage, and high-speed Internet solutions to more than one million customers throughout the United States and Canada. The end result is a smart home solution that sav...
SYS-CON Events announced today that Opsani will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Opsani is the leading provider of deployment automation systems for running and scaling traditional enterprise applications on container infrastructure.
SYS-CON Events announced today that Nirmata will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Nirmata provides a comprehensive platform, for deploying, operating, and optimizing containerized applications across clouds, powered by Kubernetes. Nirmata empowers enterprise DevOps teams by fully automating the complex operations and management of application containers and its underlying ...
SYS-CON Events announced today that Opsani to exhibit at SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California. Opsani is creating the next generation of automated continuous deployment tools designed specifically for containers. How is continuous deployment different from continuous integration and continuous delivery? CI/CD tools provide build and test. Continuous Deployment is the means by which...