Welcome!

Java IoT Authors: Liz McMillan, Pat Romanski, Yeshim Deniz, Elizabeth White, Kong Yang

Related Topics: Java IoT

Java IoT: Article

Java Feature — Building Real-Time Applications with Continuous Query Technology

The promise of a robust new development model

The client/server development model prevalent in the mid-1990's resulted in extremely easy-to-build rich GUI applications that interacted directly with a relational database. 4GL tools such as Visual Basic and PowerBuilder let even junior developers visually compose both the presentation and most of the backend data binding. While this made for impressive Rapid Application Development (RAD) productivity, the client/server architecture was severely challenged when dealing with real-time environments where the data changes rapidly and applications require visibility to the correct data at all times. As a result, client applications were forced to poll the database continuously to check for changes.

The same is true in today's browser-based or Java Swing-based multi-tier applications, where the user is forced to issue a screen refresh to view the latest state. Real-time applications such as a trader desktop where the screens are continuously refreshed are still sophisticated proprietary applications that require specialized application design to push events from backend servers to the GUI clients. Such applications result in hundreds or even thousands of views like this: Maintain a continuous view of all Intel and Dell orders placed today and notify me when AMD moves up or down by 5%.

However, today a new "push-based" architecture enables data changes to be monitored continuously in a backend data management system and changes continuously pushed to client applications, maintaining a real-time view at all times.

Traditional Databases Are Passive
Most complex GUI screens use complex SQL - multi-table joins, column aggregations, and multiple predicates for filtering, grouping, etc. to construct the dataset being viewed. Consider a real-time application like a financial stock monitoring program or a traffic management system with hundreds of concurrent clients with equally large numbers of complex queries that are continuously being executed once every second. The traditional relational database that's built for storing data efficiently and guaranteeing consistency won't be able to cope with this demand. Relational databases are passive, executing queries on sitting data only. Today's complex applications, however, require a system that can very efficiently execute queries as data streams in.

A SQL Continuous Query Engine - An Active Data Management System
By building a database engine designed specifically so queries can remain standing and active - or continuous - the scalability, reactivity, and organization of multi-tiered data-centric applications can be radically altered. Continuous queries (CQ) let users get new results from a database without having to issue the same query repeatedly.

Queries no longer have to be reissued to refresh result sets, logic that has to execute in response to complex changes in a data model can actively register interest directly from the source, and business logic can be safely co-located with application data in a relational model without scalability limitations.

Continuous querying technology works through an engine that efficiently groups and filters predicates from large numbers of queries, enabling several key things to happen:

  • When the server first gets a continuous query, it not only replies with an initial result set, but it analyzes the query predicates (selection criteria) to group it logically with other similar queries.
  • The engine can then quickly identify what continuous queries are affected by any given data modification (an insert, update, or delete against the relationally structured operational data).
  • The engine can send only the deltas to each CQ client needed to update its existing result set, in effect exactly the data necessary for the client to hold a materialized view of data from the server.
The inherent power of this technology lies in both its simplicity and natural scalability. With an in-memory database and some very simple extensions to existing query languages, we're suddenly capable of building a middle-tier that combines all of a database's operational benefits and none of its limitations. At the same time, we can build application server clients that express interest in data through conventional queries without having to trade performance for data currency or functional sophistication for development effort.

Where Does It Apply?
While the focus of this article is to illustrate the power of CQ to provide real-time view maintenance in graphical user interfaces, the power of this paradigm is well beyond this. Continuous querying is part of a new data management paradigm called stream data processing (see the References section below for further information) and can be used to monitor multiple streaming sources of data, analyze these streams for patterns of interest, and respond instantaneously. The sources of the data could be disparate - RFID sensor events, events from business applications across an enterprise, external sources, etc.

If the applicability of the technology were to be characterized in two points they would be:

  1. Data is changing very rapidly and decisions have to be made instantaneously.
  2. The system can analyze hundreds to thousands of patterns (rules or query predicates) with thousands of events pouring in every second.
Leveraging CQ in a Financial Order Tracking System
A great example of a system with stringent real-time data requirements is a securities trading order-tracking system. Diverse event streams such as customer orders, order executions, and market data quotes must be combined into a continuously updated view provided to multiple end users. Early implementations - built at a time of much smaller order volumes and slower workflows - used client/server architectures that required GUIs to poll the database for updates.

As trading volumes grew and firms realized that they could gain a competitive advantage with faster trading systems, we started seeing trading application servers that could publish real-time updates to trader GUIs. Implementing this functionality efficiently - with predictable low-latency and high throughput - required a much more sophisticated development model than client/server could provide and so the effort to build, maintain, and operate these systems grew quickly. The example in Figure 1 strips functionality to its bare essentials to illustrate the relative merits of conventional and CQ architectures.

Data Model
For simplicity's sake, our example uses only two tables - Orders and Quotes, as shown in Table 1.

The Orders table simply tracks the orders clients have sent you and the number of shares filled in each one. The Quotes table keeps track of market activity for any symbol that exists in the Orders table, so the GUI displays market activity associated with the order to the responsible broker. This means that we have three event streams coming into the system: orders from clients, order fills from a stock exchange, and market data quotes from a quote feed. These three event streams must be coalesced into a coherent view within the Order Tracking System. Naturally, in the real world these systems are much more sophisticated - requiring transactions, many more data entities and streams, and complex user interactions, but we'll keep things simple and explain later how the example may be extrapolated to additional complexity.

We'll describe the components and options for this use case in three sections. The first shows how to build a simple database publisher application for quotes and orders. The second shows how you might build a client application using plain JDBC, or with the addition of database and JMS queues, and third how you'd build a client application with Continuous Querying. Note that the full source code and pre-configured runtime configurations for the DataPublisher CQExampleClient applications are available for download at www.gemstone.com/download/.

A Simple Data Publisher for an Order Tracking System
The Data Publisher application is quite straightforward. It instantiates Order and Quote generation simulators and handles their events by "publishing" them in a JDBC database with SQL Insert and Update statements.

Let's take a closer look at the Publisher application by inspecting some of its code. The classes we need to understand are SimpleDataPublisher, SimpleQuoteGenerator, and SimpleOrderGenerator.

  • SimpleOrderGenerator has a pre-defined list of Orders and Customers. Its constructor has only one argument - a listener that implements the methods onNewOrder(SimpleOrder o) and onOrderUpdate(SimpleOrder o). In the constructor it creates a basket of 63 orders (which results in onNewOrder() callbacks), and then spawns a thread to gradually fill the orders by incrementing the fillQty property of each one.
  • SimpleQuoteGenerator generates simple market data quote streams in response to calls to its addSymbol (String symbol) method. It's constructed with a listener that implements onQuote ( SimpleQuote q ), and it spawns a thread that randomly updates each Symbol's quote at a steady rate.
SimpleDataPublisher connects to a JDBC database with a JDBC driver, either creates or clears out the Quote and Order tables with DDL/DML statements, and then uses the two generator classes as event sources. Each generator callback event (onQuote(), onNewOrder() and onOrderUpdate()) fires logic to perform a SQL INSERT or UPDATE to the CQ server, thus creating or modifying records in the Quote and Order tables. It contains only the three following members:

Connection c = null; // The JDBC Connection to the CQ Server
SimpleQuoteGenerator quoteGenerator = new SimpleQuoteGenerator ( this );
SimpleOrderGenerator orderGenerator = new SimpleOrderGenerator ( this );

The constructor of SimpleDataPublisher looks like Listing 2.

Using a handy helper class to hide the boring details, the code gets a handle on a valid JDBC Connection object, creates or clears the necessary tables, initializes the OrderGenerator (which causes an onNewOrder() callback for each new order), and then starts the QuoteGenerator and OrderGenerator threads to initiate the event streams.

Note that for each new order created, SimpleDataPublisher calls the SimpeQuoteGenerator.addSymbol() method to register a new Symbol for quote generation - thus making sure that we get quotes for each symbol handled by system (duplicate symbols are ignored).

Each time SimpleDataPublisher gets a callback invocation, it creates a SQL statement and submits it to the CQ Engine. For example, when onUpdatedOrder() fires, the logic in Listing 1 executes:

You can see that this is a very simple O/R mapping exercise plus JDBC database interaction logic. The logic that executes in response to onQuote() and onNewOrder() is fundamentally the same - and uses SQL/JDBC to update or insert a table record. Once started, SimpleExamplePublisher continues to update the order and quote records in response to the callbacks it's getting from the generators. This is a good time to reflect on the how this fits with your existing experience - the type of code above already exists in most systems, but how often does it serve as BOTH a database update and a notification to systems that have previously queried the updated record?


More Stories By Gideon Low

Gideon Low is senior technical architect for GemStone Systems. He has over 10 years of experience in the development, management, and sales of high-performance large-scale distributed systems. His focus over the last seven years has been in high-speed electronic trading systems, with roles including CTO at Silicon Summit Technologies (A FIX/OMS vendor) and VP, Client Connectivity Technology at Lehman Brothers.

More Stories By Jags Ramnarayan

Jags Ramnarayan is the chief architect for the high-performance, distributed data management product line at GemStone Systems. He puts on multiple hats - evangelizing the technology, exploring requirements with customers, and managing the overall direction of the architecture decisions. In the past Jags participated in several Java and W3C standards for GemStone and BEA. On the side, Jags is also pursuing a MBA degree, but hopes to remain technically focussed.

Comments (5) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
BR 09/18/06 11:46:11 AM EDT

The connection API discussed here is JDBC (a well known standard), the query language is SQL (an uber standard that can support joins and other complicated functions). Yes, the callnack CQ fucntionality requires a proprietary API.

With Tangosol, connection, query language and CQ call-backs are all handled via proprietary APIs, and the query langauge is more of a filtering language.

Tangosol 09/17/06 08:44:01 PM EDT

Yes, Continuous Query in Tangosol Coherence requires the use of an API designed for Continuous Query, since there is no standard API in Java to support Continuous Query. The product covered in the article requires the use of a proprietary API to use its own continuous query features, and that seems perfectly acceptable in the absence of a standard.

BR 08/29/06 04:40:52 PM EDT

But, Tangosol's continuous query requires the use of a proprietary API (not as intuitive as SQL) and by no means can accomodate the complex conditions/predicates that can be modeled with a SQL based continuous query paradigm. Seems like a classic marketing/"me too" strategy from Tangosol..:-)

AJAXWorld News Desk 08/28/06 05:56:14 PM EDT

The client/server development model prevalent in the mid-1990's resulted in extremely easy-to-build rich GUI applications that interacted directly with a relational database. 4GL tools such as Visual Basic and PowerBuilder let even junior developers visually compose both the presentation and most of the backend data binding. While this made for impressive Rapid Application Development (RAD) productivity, the client/server architecture was severely challenged when dealing with real-time environments where the data changes rapidly and applications require visibility to the correct data at all times. As a result, client applications were forced to poll the database continuously to check for changes.

Tangosol 08/17/06 03:00:39 PM EDT

Tangosol Coherence also provided Continuous Query, including Continuous Query Caching.

@ThingsExpo Stories
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...
Artificial intelligence, machine learning, neural networks. We’re in the midst of a wave of excitement around AI such as hasn’t been seen for a few decades. But those previous periods of inflated expectations led to troughs of disappointment. Will this time be different? Most likely. Applications of AI such as predictive analytics are already decreasing costs and improving reliability of industrial machinery. Furthermore, the funding and research going into AI now comes from a wide range of com...
In this presentation, Striim CTO and founder Steve Wilkes will discuss practical strategies for counteracting fraud and cyberattacks by leveraging real-time streaming analytics. In his session at @ThingsExpo, Steve Wilkes, Founder and Chief Technology Officer at Striim, will provide a detailed look into leveraging streaming data management to correlate events in real time, and identify potential breaches across IoT and non-IoT systems throughout the enterprise. Strategies for processing massive ...
SYS-CON Events announced today that GrapeUp, the leading provider of rapid product development at the speed of business, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Grape Up is a software company, specialized in cloud native application development and professional services related to Cloud Foundry PaaS. With five expert teams that operate in various sectors of the market acr...
SYS-CON Events announced today that Ayehu will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara California. Ayehu provides IT Process Automation & Orchestration solutions for IT and Security professionals to identify and resolve critical incidents and enable rapid containment, eradication, and recovery from cyber security breaches. Ayehu provides customers greater control over IT infras...
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex software systems for startups and enterprises. Since 2009 it has grown from a small group of passionate engineers and business...
SYS-CON Events announced today that Cloud Academy named "Bronze Sponsor" of 21st International Cloud Expo which will take place October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara, CA. Cloud Academy is the industry’s most innovative, vendor-neutral cloud technology training platform. Cloud Academy provides continuous learning solutions for individuals and enterprise teams for Amazon Web Services, Microsoft Azure, Google Cloud Platform, and the most popular cloud com...
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...
SYS-CON Events announced today that CA Technologies has been named "Platinum Sponsor" of SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business - from apparel to energy - is being rewritten by software. From planning to development to management to security, CA creates software that fuels transformation for companies in the applic...
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA
Amazon started as an online bookseller 20 years ago. Since then, it has evolved into a technology juggernaut that has disrupted multiple markets and industries and touches many aspects of our lives. It is a relentless technology and business model innovator driving disruption throughout numerous ecosystems. Amazon’s AWS revenues alone are approaching $16B a year making it one of the largest IT companies in the world. With dominant offerings in Cloud, IoT, eCommerce, Big Data, AI, Digital Assista...
SYS-CON Events announced today that Enzu will exhibit at SYS-CON's 21st Int\ernational Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive advantage. By offering a suite of proven hosting and management services, Enzu wants companies to focus on the core of their ...
Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists looked at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deliver...
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), provided an overview of various initiatives to certify the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldwide re...
IoT solutions exploit operational data generated by Internet-connected smart “things” for the purpose of gaining operational insight and producing “better outcomes” (for example, create new business models, eliminate unscheduled maintenance, etc.). The explosive proliferation of IoT solutions will result in an exponential growth in the volume of IoT data, precipitating significant Information Governance issues: who owns the IoT data, what are the rights/duties of IoT solutions adopters towards t...
With the introduction of IoT and Smart Living in every aspect of our lives, one question has become relevant: What are the security implications? To answer this, first we have to look and explore the security models of the technologies that IoT is founded upon. In his session at @ThingsExpo, Nevi Kaja, a Research Engineer at Ford Motor Company, discussed some of the security challenges of the IoT infrastructure and related how these aspects impact Smart Living. The material was delivered interac...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
No hype cycles or predictions of zillions of things here. IoT is big. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, Associate Partner at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He discussed the evaluation of communication standards and IoT messaging protocols, data analytics considerations, edge-to-cloud tec...