Welcome!

Java IoT Authors: Pat Romanski, Roger Strukhoff, Liz McMillan, Yeshim Deniz, Elizabeth White

Related Topics: Java IoT

Java IoT: Article

End-to-End Transactionality

End-to-End Transactionality

In addition, it was suggested that traditional Online Transaction Processing systems (OLTP) don’t suffer from such limitations, rendering them more suitable for the emerging e-commerce applications that may require such guarantees.

This article discusses this question and shows that there’s nothing inherently wrong with these new models that prevents applications from using them to obtain end-to-end transactionality. However, before addressing the question of whether or not any specific transaction system can be used to provide end-to-end transactional guarantees, it’s important to realize the following: end-to-end transactionality is not some holy grail that people have been searching for in myths and legends; it’s a solution to one specific problem area, not a global panacea to all transaction issues. The ability or lack thereof to guarantee end-to-end transaction integrity does not in and of itself prevent a specific transaction system provider from tackling many other equally important issues in today’s evolving world of e-commerce and mobile applications.

What Is End-to-End Transactionality?
Let’s consider what such end-to-end guarantees are. Atomic transactions (transactions) are used in application programs to control the manipulation of persistent (long-lived) objects. Transactions have the following ACID properties:

Atomic: If interrupted by failure, all effects are undone (rolled back).

Consistent: The effects of a transaction preserve invariant properties.

Isolated: A transaction’s intermediate states are not visible to other transactions. Transactions appear to execute serially, even if they’re performed concurrently.

Durable: The effects of a completed transaction are persistent; they’re never lost (except in a catastrophic failure).

A transaction can be terminated in two ways: committed or aborted (rolled back). When a transaction is committed, all changes made within it are made durable (forced on to stable storage, e.g., disk). When a transaction is aborted, all the changes are undone. Atomic transactions can also be nested; the effects of a nested transaction are provisional upon the commit/abort of the outermost (top-level) atomic transaction.

Commit Protocol
A two-phase commit protocol is required to guarantee that all the transaction participants either commit or abort any changes made. Figure 1 illustrates the main aspects of the commit protocol: during phase one, the transaction coordinator, C, attempts to communicate with all the transaction participants, A and B, to determine whether they’ll commit or abort. An abort reply from any participant acts as a veto, causing the entire transaction to abort. Based upon these (lack of) responses, the coordinator decides whether to commit or abort the transaction. If the transaction will commit, the coordinator records this decision on stable storage and the protocol enters phase two, where the coordinator forces the participants to carry out the decision. The coordinator also informs the participants if the transaction aborts.

When each participant receives the coordinator’s phase-one message, they record sufficient information on stable storage to either commit or abort changes made during the transaction. After returning the phase-one response, each participant that returned a commit response must remain blocked until it has received the coordinator’s phase-two message. Until they receive this message, these resources are unavailable for use by other transactions. If the coordinator fails before this message is delivered, these resources remain blocked. However, if crashed machines eventually recover, crash recovery mechanisms can be employed to unblock the protocol and terminate the transaction.

A participant’s role depends on the application in which it occurs. For example, a J2EE JTA XAResource is a participant that typically controls the fate of work performed on a database (e.g., Oracle) within the scope of a specific transaction.

Note: The two-phase commit protocol that most transaction systems use is not client/server-based. It simply talks about a coordinator and participants, and makes no assumption about where they’re located. Different implementations of the protocol may impose certain restrictions about locality, but these are purely implementation choices.

What Is an End-to-End Transaction?
In this section I use the Web as an example of where end-to-end transactionality integrity is required. However, there are other areas (e.g., mobile) that are equally valid. Transposing the issues mentioned here to these other areas should be relatively straightforward.

Atomic transactions, with their “all-or-nothing” property, are a well-known technique for guaranteeing application consistency in the presence of failures. Although Web applications exist that offer transactional guarantees to users, these guarantees extend only to resources used at or between Web servers; clients (browsers) are not included, even though they play a significant role in certain applications, as mentioned earlier.

Therefore, providing end-to-end transactional integrity between the browser and the application (server) is important, as it allows work that involves both the browser and the server to be atomic. However, current techniques based on CGI scripts can’t provide end-to-end guarantees. As shown in Figure 2, the user selects a URL that references a CGI script on a Web server (message one), which then performs the transaction and returns a response to the browser (message two) after the transaction is complete. Returning the message during the transaction is incorrect since it may not be able to commit the changes.

In a failure-free environment, this mechanism works well. However, in the presence of failure it’s possible for message two to be lost between the server and the browser, resulting in work at the server not being atomic with respect to any browser-related work. Thus, there’s no end-to-end transactional guarantee.

Is End-to-End Transactionality Possible?
Yes. There’s nothing inherent in any transaction protocol (two-phase, three-phase, presumed abort, presumed nothing, etc.) that prevents end-to-end transactionality. The transaction engine (essentially the coordinator) has very little effect on this, whether or not it’s embedded in a proprietary service or within an industry standard application server. What does make end-to-end transactionality difficult is that it requires a transactional participant to reside at each “end,” but it does not require a transaction coordinator and its associated baggage to reside at each “end.”

A contract exists between transaction coordinator and participants, which states (in brief and with many simplifications):

• Once the transaction coordinator has decided to commit or roll back the transaction, it guarantees delivery of this information to every participant, regardless of failures. Note: There are various optimizations to this, such as a presumed abort protocol, but in essence the contract remains the same.

• Once told to prepare, a participant will make sufficient durable information for it to either commit or cancel the work that it controls. Until it determines the final outcome, it should neither commit nor cancel the work itself. If a failure occurs, or the final outcome of the transaction is slow in arriving, the resource can typically communicate with the coordinator to determine the current progress of the transaction.

In most transaction systems the majority of the effort goes into designing and developing the transaction-coordinator engine, making it as performant and reliable as possible. However, this by itself is insufficient to provide a usable system: participants are obviously required. Although any contract-conformant participant implementation can be plugged into the two-phase protocol, typically the only ones that most people use are those that control work performed on (distributed) databases, e.g., the aforementioned XAResource. This tends to result in the fact that many people equate transactions with databases only, and hence the significant amount of resources required for these participant implementations. However, this is not the case: a participant can be as resource hungry as necessary in order to fulfill the contract. Thus, a participant could use a local file system to make state information durable, for example, or it could use nonvolatile RAM. It depends on what the programmer deems necessary.

With the advent of Java it’s possible to empower thin (resource scarce) clients (e.g., browsers) so they can fully participate within transactional applications. Transaction participants tailored to the application and environment where they’ll work can be downloaded (on demand) to the client to ensure that they can fully participate within the two-phase commit protocol. There’s nothing special about specific transaction system implementations that makes them more easily adapted to this kind of environment. The specialization comes from the end-point resource – the client-side participant.

OLTP vs OO-TP
Are “traditional” online transaction processing (OLTP) engines more suited to end-to-end transactionality guarantees than “newer” object-oriented transaction systems? The quick answer is no. Why should they be? It doesn’t matter whether a transaction system is supported by a proprietary remote procedure call (RPC) and stub generation techniques or by an open-standard remote object invocation mechanism such as a CORBA ORB; once the distributed layers are removed, they all share the same core – a two-phase commit protocol engine that supports durable storage and failure recovery. How that engine is invoked, and how it invokes its participants, is immaterial to its overall workings.

The real benefit of OO-TP over OLTP is openness. Over the past eight years there’s been a significant move away from proprietary transaction processing systems and their support infrastructure for open standards. This move has been driven by users who traditionally found it extremely difficult to move applications from one vendor’s product to another, or even between different versions of a product from the same vendor. The OMG pioneered this approach with the Object Transaction Service (OTS) in 1995, when IBM, HP, Digital, and others got together to provide a means whereby their existing products could essentially be wrapped in a standard veneer; this approach allowed applications developed with this veneer to be ported from one implementation to another, and for different implementations to interact (something else that was extremely difficult to do reliably).

Therefore, it’s inaccurate to conclude that OLTP systems are superior in any way to OO-TP equivalents because of their architecture, support environment, or distribution paradigm. OLTP systems are typically monolithic closed systems, tying users to vendor-specific choices for implementations, languages, etc. If the experiences gained by the developers of efficient and reliable implementations of OLTP are transposed to OO-TP, then there’s nothing to prevent such an OO-TP system from competing well. The advantages should be obvious: open systems allow customers to pick and choose the components they require to develop their applications without worrying about vendor lock-in. Such systems are also more readily ported to new hardware and operating systems, allowing customers even more choice for deployment.

The OTS
The OTS architecture provides standard interfaces to components that are possessed by all transaction engines. It  doesn’t modify the model underlying all the existing different transaction monitor implementations; it mandates a two-phase commit protocol with presumed abort and all implementations of the OTS must comply with this. It was intended as an adjunct to these systems, not as a replacement.

No company that has spent many years building up reliability in such a critical piece of software as transactions would be prepared to start from scratch and implement again. In addition, no user of such reliable transaction software would be prepared to take the risk of transitioning to this new software, even if it were “open.” In the area of transactions, which are critical fault-tolerance components, it takes time to convince customers that new technology is stable and performant enough to replace what they’ve been using for many years.

Although the CORBA model is typically discussed in terms of client/server architecture, from the outset its designers did not want to impose any restrictions on the type of environment in which it could run. There’s no assumption about how “thin” a client is, or how “fat” a server must be, in order to execute a CORBA application. Many programmers these days simply use the client/server model as a convenient way in which to reason about distributed applications. But at their core these applications never have what would traditionally be considered a thin client; services that a user requires may well be colocated with that user within the same process. CORBA was the first open architecture to support the configurable deployment of services in this way, correctly seeing this separation of client and service as just that: a deployment issue. Nothing in the CORBA model requires a client to be thin and functionally deficient.

The OTS is comparable to CICS, Tuxedo, and DEC ACMS. It differs only in that it’s a standard and allows interoperation. There’s nothing fundamentally wrong with the OTS architecture that prevents it from being used in an end-to-end manner. The OTS supports end-to-end transactionality in exactly the same way CICS or any other “traditional” OLTP would – through its resources.

To ORB or Not to ORB?
There also appears to be some confusion as to whether CORBA implementations (ORBs) are sufficient for mission-critical applications. In the mid-’90s when implementations first appeared on the market, their performance was not as good as handcrafted solutions. However, that has certainly changed over the past few years. The footprint of some ORBs is certainly large, but there are other ORBs that have been tailored specifically for real-time or embedded use. Companies such as IBM, IONA, and BEA have seen ORBs develop over the years to become a critical part of the infrastructure that they have to control and therefore they have their own implementations. Other companies have licensed ORB implementations from elsewhere.

The crash failure of an ORB doesn’t typically mean that it can recover automatically and continue applications from where it left off. This is because the ORB doesn’t have necessary semantic and syntactic information to automatically check point state for recovery purposes. However, by using suitably defined services such as the OTS and the persistence service, it’s possible for applications to do this themselves or for vendors to use these services to do this for applications.

It’s been said that OLTP systems provide this kind of feature out-of-the-box, and it may be true. However, it’s an unfair comparison: an OLTP system does just one thing and does it well – it manages transactions. A CORBA ORB is meant to provide support for arbitrary distributed applications, the majority of which probably won’t even need fault tolerance, let alone transactions. However, for those applications that do need these capabilities, it’s entirely possible to provide exactly the same recovery functionality using OMG open standards.

J2EE
Although the J2EE model is client /server based, as with CORBA there’s nothing to prevent a client from being rich in functionality. It’s a deployment choice that’s made at build-time and runtime (obviously, the capability for being so rich is required to be built into the client, and even if it were present, it would be up to the user to determine whether or not such functionality was required or possible). Note: Although J2EE didn’t start out as an infrastructure that used CORBA, it quickly became evident that the OMG companies’ experiences in developing CORBA were extremely important to any distributed system. As a result, over the past few years J2EE has gotten closer and closer to the CORBA world, and now requires some critical ORB components in order to run.

The typical way in which J2EE programmers use transactions is through the Java Transaction API (JTA), which is a mandated part of the specification. The JTA is intended as a higher-level API for programmers to try to isolate them from some of the more complex (and sometimes esoteric) aspects of constructing transactional applications. The JTA does not imply a specific underlying implementation for a transaction system, so it could be layered on CICS, Tuxedo, etc. However, because the OTS is now the transaction standard for most companies and it allows interoperation between different implementations, it was decided that the preferred implementation would be based on this (called the JTS, just to place it firmly in the Java domain). The JTS is currently optional, but it may eventually become a mandated part of the J2EE specification.

Application Server Means Thin Client?
No. As shown above, this is essentially a deployment issue. It’s certainly correct to say that most J2EE programmers currently use a thin(-ish) client, with most of the business logic residing within the server; however, this is simply because this solution matches 90% of the problems. Closer examination of all application server applications would certainly reveal that although thin clients are the norm, they are by no means the complete picture.

The application server has nothing fundamentally wrong with its model either. Not to say that the client-side of an application server application has to be wafer-thin. If the client wants to embed functionality such as a two-phase aware transactional resource within itself, that’s entirely possible. In fact, a client could just as easily be embedded within an application server, if the footprint allowed. The reasons for not doing this have more to do with the footprint size than any architectural issue.

Conclusion
Can end-to-end transactional guarantees be provided by modern transaction systems such as JTA? Yes, as we’ve shown there’s nothing inherent in these models that prevents them from providing such guarantees. The two-phase commit protocol doesn’t know anything about clients or servers, doesn’t make assumptions about the locality of the coordinator or participants, and doesn’t require any semantic knowledge of the applications. End-to-end transactional guarantees are simply a deployment view on the relative locality of different participants.

Are OLTP systems more suited to end-to-end guarantees than their modern OO-TP cousins? As we’ve shown, since they’re both based on two-phase commit protocols, there’s nothing in either model that would mean they are any more or less ideal for any specific problem domain. However, there are obvious design and implementation decisions that can be made when building a transaction system using either model, which may mean that specific instances are not best suited for end-to-end transactional solutions. It’s important to realize that this is an implementation choice only.

References
• Little, M.C., Shrivastava, S.K., Caughey,S.J. and Ingham, D.B. (1997).“Constructing Reliable Web Applications Using Atomic Actions.” Proceedings of the Sixth Web Conference. April.

• Little, M.C. (1997). “Providing End-to-End Transactional Web Applications Using the Object Transaction Service” OMG Success Story.

• “CORBAservices: Common Object Services Specification.” OMG Document Number 95-3-31. March 1995.

• “Java Transaction API 1.0.1 (JTA).” Sun Microsystems. April 1999.

• “Java Transaction Service 1.0 (JTS).” Sun Microsystems. December 1999.

Recently, the question was asked whether or not the models on which current transaction systems are based (e.g., JTA and JTS) are powerful enough to support end-to-end transactional guarantees for applications.

More Stories By Mark Little

Mark Little was Chief Architect, Transactions for Arjuna Technologies Ltd, a UK-based company specialising in the development of reliable middleware that was recently acquired by JBoss, Inc. Before Arjuna, Mark was a Distinguished Engineer/Architect within HP Arjuna Labs in Newcastle upon Tyne, England, where he led the HP-TS and HP-WST teams, developing J2EE and Web services transactions products respectively. He is one of the primary authors of the OMG Activity Service specification and is on the expert group for the same work in J2EE (JSR 95). He is also the specification lead for JSR 156: Java API for XML Transactions. He's on the OTS Revision Task Force and the OASIS Business Transactions Protocol specification. Before joining HP he was for over 10 years a member of the Arjuna team within the University of Newcastle upon Tyne (where he continues to have a Visiting Fellowship). His research within the Arjuna team included replication and transactions support, which include the construction of an OTS/JTS compliant transaction processing system. Mark has published extensively in the Web Services Journal, Java Developer's Journal and other journals and magazines. He is also the co-author of several books including “Java and Transactions for Systems Professionals” and “The J2EE 1.4 Bible.”

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone inn...
Digital Transformation (DX) is not a "one-size-fits all" strategy. Each organization needs to develop its own unique, long-term DX plan. It must do so by realizing that we now live in a data-driven age, and that technologies such as Cloud Computing, Big Data, the IoT, Cognitive Computing, and Blockchain are only tools. In her general session at 21st Cloud Expo, Rebecca Wanta explained how the strategy must focus on DX and include a commitment from top management to create great IT jobs, monitor ...
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, discussed how they built...
Recently, WebRTC has a lot of eyes from market. The use cases of WebRTC are expanding - video chat, online education, online health care etc. Not only for human-to-human communication, but also IoT use cases such as machine to human use cases can be seen recently. One of the typical use-case is remote camera monitoring. With WebRTC, people can have interoperability and flexibility for deploying monitoring service. However, the benefit of WebRTC for IoT is not only its convenience and interopera...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, examined the regulations and provided insight on how it affects technology, challenges the established rules and will usher in new levels of diligence arou...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
The 22nd International Cloud Expo | 1st DXWorld Expo has announced that its Call for Papers is open. Cloud Expo | DXWorld Expo, to be held June 5-7, 2018, at the Javits Center in New York, NY, brings together Cloud Computing, Digital Transformation, Big Data, Internet of Things, DevOps, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
No hype cycles or predictions of a gazillion things here. IoT is here. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, an Associate Partner of Analytics, IoT & Cybersecurity at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He also discussed the evaluation of communication standards and IoT messaging protocols, data...
Product connectivity goes hand and hand these days with increased use of personal data. New IoT devices are becoming more personalized than ever before. In his session at 22nd Cloud Expo | DXWorld Expo, Nicolas Fierro, CEO of MIMIR Blockchain Solutions, will discuss how in order to protect your data and privacy, IoT applications need to embrace Blockchain technology for a new level of product security never before seen - or needed.
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
"Evatronix provides design services to companies that need to integrate the IoT technology in their products but they don't necessarily have the expertise, knowledge and design team to do so," explained Adam Morawiec, VP of Business Development at Evatronix, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Digital transformation - what we knew about it in the past has been redefined. Automation is going to play such a huge role in that because the culture, the technology, and the business operations are being shifted now," stated Brian Boeggeman, VP of Alliances & Partnerships at Ayehu, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud ...
22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud ...