Java IoT Authors: Elizabeth White, Yeshim Deniz, Liz McMillan, Pat Romanski, ManageEngine IT Matters

Related Topics: Java IoT

Java IoT: Article

End-to-End Transactionality

End-to-End Transactionality

In addition, it was suggested that traditional Online Transaction Processing systems (OLTP) don’t suffer from such limitations, rendering them more suitable for the emerging e-commerce applications that may require such guarantees.

This article discusses this question and shows that there’s nothing inherently wrong with these new models that prevents applications from using them to obtain end-to-end transactionality. However, before addressing the question of whether or not any specific transaction system can be used to provide end-to-end transactional guarantees, it’s important to realize the following: end-to-end transactionality is not some holy grail that people have been searching for in myths and legends; it’s a solution to one specific problem area, not a global panacea to all transaction issues. The ability or lack thereof to guarantee end-to-end transaction integrity does not in and of itself prevent a specific transaction system provider from tackling many other equally important issues in today’s evolving world of e-commerce and mobile applications.

What Is End-to-End Transactionality?
Let’s consider what such end-to-end guarantees are. Atomic transactions (transactions) are used in application programs to control the manipulation of persistent (long-lived) objects. Transactions have the following ACID properties:

Atomic: If interrupted by failure, all effects are undone (rolled back).

Consistent: The effects of a transaction preserve invariant properties.

Isolated: A transaction’s intermediate states are not visible to other transactions. Transactions appear to execute serially, even if they’re performed concurrently.

Durable: The effects of a completed transaction are persistent; they’re never lost (except in a catastrophic failure).

A transaction can be terminated in two ways: committed or aborted (rolled back). When a transaction is committed, all changes made within it are made durable (forced on to stable storage, e.g., disk). When a transaction is aborted, all the changes are undone. Atomic transactions can also be nested; the effects of a nested transaction are provisional upon the commit/abort of the outermost (top-level) atomic transaction.

Commit Protocol
A two-phase commit protocol is required to guarantee that all the transaction participants either commit or abort any changes made. Figure 1 illustrates the main aspects of the commit protocol: during phase one, the transaction coordinator, C, attempts to communicate with all the transaction participants, A and B, to determine whether they’ll commit or abort. An abort reply from any participant acts as a veto, causing the entire transaction to abort. Based upon these (lack of) responses, the coordinator decides whether to commit or abort the transaction. If the transaction will commit, the coordinator records this decision on stable storage and the protocol enters phase two, where the coordinator forces the participants to carry out the decision. The coordinator also informs the participants if the transaction aborts.

When each participant receives the coordinator’s phase-one message, they record sufficient information on stable storage to either commit or abort changes made during the transaction. After returning the phase-one response, each participant that returned a commit response must remain blocked until it has received the coordinator’s phase-two message. Until they receive this message, these resources are unavailable for use by other transactions. If the coordinator fails before this message is delivered, these resources remain blocked. However, if crashed machines eventually recover, crash recovery mechanisms can be employed to unblock the protocol and terminate the transaction.

A participant’s role depends on the application in which it occurs. For example, a J2EE JTA XAResource is a participant that typically controls the fate of work performed on a database (e.g., Oracle) within the scope of a specific transaction.

Note: The two-phase commit protocol that most transaction systems use is not client/server-based. It simply talks about a coordinator and participants, and makes no assumption about where they’re located. Different implementations of the protocol may impose certain restrictions about locality, but these are purely implementation choices.

What Is an End-to-End Transaction?
In this section I use the Web as an example of where end-to-end transactionality integrity is required. However, there are other areas (e.g., mobile) that are equally valid. Transposing the issues mentioned here to these other areas should be relatively straightforward.

Atomic transactions, with their “all-or-nothing” property, are a well-known technique for guaranteeing application consistency in the presence of failures. Although Web applications exist that offer transactional guarantees to users, these guarantees extend only to resources used at or between Web servers; clients (browsers) are not included, even though they play a significant role in certain applications, as mentioned earlier.

Therefore, providing end-to-end transactional integrity between the browser and the application (server) is important, as it allows work that involves both the browser and the server to be atomic. However, current techniques based on CGI scripts can’t provide end-to-end guarantees. As shown in Figure 2, the user selects a URL that references a CGI script on a Web server (message one), which then performs the transaction and returns a response to the browser (message two) after the transaction is complete. Returning the message during the transaction is incorrect since it may not be able to commit the changes.

In a failure-free environment, this mechanism works well. However, in the presence of failure it’s possible for message two to be lost between the server and the browser, resulting in work at the server not being atomic with respect to any browser-related work. Thus, there’s no end-to-end transactional guarantee.

Is End-to-End Transactionality Possible?
Yes. There’s nothing inherent in any transaction protocol (two-phase, three-phase, presumed abort, presumed nothing, etc.) that prevents end-to-end transactionality. The transaction engine (essentially the coordinator) has very little effect on this, whether or not it’s embedded in a proprietary service or within an industry standard application server. What does make end-to-end transactionality difficult is that it requires a transactional participant to reside at each “end,” but it does not require a transaction coordinator and its associated baggage to reside at each “end.”

A contract exists between transaction coordinator and participants, which states (in brief and with many simplifications):

• Once the transaction coordinator has decided to commit or roll back the transaction, it guarantees delivery of this information to every participant, regardless of failures. Note: There are various optimizations to this, such as a presumed abort protocol, but in essence the contract remains the same.

• Once told to prepare, a participant will make sufficient durable information for it to either commit or cancel the work that it controls. Until it determines the final outcome, it should neither commit nor cancel the work itself. If a failure occurs, or the final outcome of the transaction is slow in arriving, the resource can typically communicate with the coordinator to determine the current progress of the transaction.

In most transaction systems the majority of the effort goes into designing and developing the transaction-coordinator engine, making it as performant and reliable as possible. However, this by itself is insufficient to provide a usable system: participants are obviously required. Although any contract-conformant participant implementation can be plugged into the two-phase protocol, typically the only ones that most people use are those that control work performed on (distributed) databases, e.g., the aforementioned XAResource. This tends to result in the fact that many people equate transactions with databases only, and hence the significant amount of resources required for these participant implementations. However, this is not the case: a participant can be as resource hungry as necessary in order to fulfill the contract. Thus, a participant could use a local file system to make state information durable, for example, or it could use nonvolatile RAM. It depends on what the programmer deems necessary.

With the advent of Java it’s possible to empower thin (resource scarce) clients (e.g., browsers) so they can fully participate within transactional applications. Transaction participants tailored to the application and environment where they’ll work can be downloaded (on demand) to the client to ensure that they can fully participate within the two-phase commit protocol. There’s nothing special about specific transaction system implementations that makes them more easily adapted to this kind of environment. The specialization comes from the end-point resource – the client-side participant.

Are “traditional” online transaction processing (OLTP) engines more suited to end-to-end transactionality guarantees than “newer” object-oriented transaction systems? The quick answer is no. Why should they be? It doesn’t matter whether a transaction system is supported by a proprietary remote procedure call (RPC) and stub generation techniques or by an open-standard remote object invocation mechanism such as a CORBA ORB; once the distributed layers are removed, they all share the same core – a two-phase commit protocol engine that supports durable storage and failure recovery. How that engine is invoked, and how it invokes its participants, is immaterial to its overall workings.

The real benefit of OO-TP over OLTP is openness. Over the past eight years there’s been a significant move away from proprietary transaction processing systems and their support infrastructure for open standards. This move has been driven by users who traditionally found it extremely difficult to move applications from one vendor’s product to another, or even between different versions of a product from the same vendor. The OMG pioneered this approach with the Object Transaction Service (OTS) in 1995, when IBM, HP, Digital, and others got together to provide a means whereby their existing products could essentially be wrapped in a standard veneer; this approach allowed applications developed with this veneer to be ported from one implementation to another, and for different implementations to interact (something else that was extremely difficult to do reliably).

Therefore, it’s inaccurate to conclude that OLTP systems are superior in any way to OO-TP equivalents because of their architecture, support environment, or distribution paradigm. OLTP systems are typically monolithic closed systems, tying users to vendor-specific choices for implementations, languages, etc. If the experiences gained by the developers of efficient and reliable implementations of OLTP are transposed to OO-TP, then there’s nothing to prevent such an OO-TP system from competing well. The advantages should be obvious: open systems allow customers to pick and choose the components they require to develop their applications without worrying about vendor lock-in. Such systems are also more readily ported to new hardware and operating systems, allowing customers even more choice for deployment.

The OTS architecture provides standard interfaces to components that are possessed by all transaction engines. It  doesn’t modify the model underlying all the existing different transaction monitor implementations; it mandates a two-phase commit protocol with presumed abort and all implementations of the OTS must comply with this. It was intended as an adjunct to these systems, not as a replacement.

No company that has spent many years building up reliability in such a critical piece of software as transactions would be prepared to start from scratch and implement again. In addition, no user of such reliable transaction software would be prepared to take the risk of transitioning to this new software, even if it were “open.” In the area of transactions, which are critical fault-tolerance components, it takes time to convince customers that new technology is stable and performant enough to replace what they’ve been using for many years.

Although the CORBA model is typically discussed in terms of client/server architecture, from the outset its designers did not want to impose any restrictions on the type of environment in which it could run. There’s no assumption about how “thin” a client is, or how “fat” a server must be, in order to execute a CORBA application. Many programmers these days simply use the client/server model as a convenient way in which to reason about distributed applications. But at their core these applications never have what would traditionally be considered a thin client; services that a user requires may well be colocated with that user within the same process. CORBA was the first open architecture to support the configurable deployment of services in this way, correctly seeing this separation of client and service as just that: a deployment issue. Nothing in the CORBA model requires a client to be thin and functionally deficient.

The OTS is comparable to CICS, Tuxedo, and DEC ACMS. It differs only in that it’s a standard and allows interoperation. There’s nothing fundamentally wrong with the OTS architecture that prevents it from being used in an end-to-end manner. The OTS supports end-to-end transactionality in exactly the same way CICS or any other “traditional” OLTP would – through its resources.

To ORB or Not to ORB?
There also appears to be some confusion as to whether CORBA implementations (ORBs) are sufficient for mission-critical applications. In the mid-’90s when implementations first appeared on the market, their performance was not as good as handcrafted solutions. However, that has certainly changed over the past few years. The footprint of some ORBs is certainly large, but there are other ORBs that have been tailored specifically for real-time or embedded use. Companies such as IBM, IONA, and BEA have seen ORBs develop over the years to become a critical part of the infrastructure that they have to control and therefore they have their own implementations. Other companies have licensed ORB implementations from elsewhere.

The crash failure of an ORB doesn’t typically mean that it can recover automatically and continue applications from where it left off. This is because the ORB doesn’t have necessary semantic and syntactic information to automatically check point state for recovery purposes. However, by using suitably defined services such as the OTS and the persistence service, it’s possible for applications to do this themselves or for vendors to use these services to do this for applications.

It’s been said that OLTP systems provide this kind of feature out-of-the-box, and it may be true. However, it’s an unfair comparison: an OLTP system does just one thing and does it well – it manages transactions. A CORBA ORB is meant to provide support for arbitrary distributed applications, the majority of which probably won’t even need fault tolerance, let alone transactions. However, for those applications that do need these capabilities, it’s entirely possible to provide exactly the same recovery functionality using OMG open standards.

Although the J2EE model is client /server based, as with CORBA there’s nothing to prevent a client from being rich in functionality. It’s a deployment choice that’s made at build-time and runtime (obviously, the capability for being so rich is required to be built into the client, and even if it were present, it would be up to the user to determine whether or not such functionality was required or possible). Note: Although J2EE didn’t start out as an infrastructure that used CORBA, it quickly became evident that the OMG companies’ experiences in developing CORBA were extremely important to any distributed system. As a result, over the past few years J2EE has gotten closer and closer to the CORBA world, and now requires some critical ORB components in order to run.

The typical way in which J2EE programmers use transactions is through the Java Transaction API (JTA), which is a mandated part of the specification. The JTA is intended as a higher-level API for programmers to try to isolate them from some of the more complex (and sometimes esoteric) aspects of constructing transactional applications. The JTA does not imply a specific underlying implementation for a transaction system, so it could be layered on CICS, Tuxedo, etc. However, because the OTS is now the transaction standard for most companies and it allows interoperation between different implementations, it was decided that the preferred implementation would be based on this (called the JTS, just to place it firmly in the Java domain). The JTS is currently optional, but it may eventually become a mandated part of the J2EE specification.

Application Server Means Thin Client?
No. As shown above, this is essentially a deployment issue. It’s certainly correct to say that most J2EE programmers currently use a thin(-ish) client, with most of the business logic residing within the server; however, this is simply because this solution matches 90% of the problems. Closer examination of all application server applications would certainly reveal that although thin clients are the norm, they are by no means the complete picture.

The application server has nothing fundamentally wrong with its model either. Not to say that the client-side of an application server application has to be wafer-thin. If the client wants to embed functionality such as a two-phase aware transactional resource within itself, that’s entirely possible. In fact, a client could just as easily be embedded within an application server, if the footprint allowed. The reasons for not doing this have more to do with the footprint size than any architectural issue.

Can end-to-end transactional guarantees be provided by modern transaction systems such as JTA? Yes, as we’ve shown there’s nothing inherent in these models that prevents them from providing such guarantees. The two-phase commit protocol doesn’t know anything about clients or servers, doesn’t make assumptions about the locality of the coordinator or participants, and doesn’t require any semantic knowledge of the applications. End-to-end transactional guarantees are simply a deployment view on the relative locality of different participants.

Are OLTP systems more suited to end-to-end guarantees than their modern OO-TP cousins? As we’ve shown, since they’re both based on two-phase commit protocols, there’s nothing in either model that would mean they are any more or less ideal for any specific problem domain. However, there are obvious design and implementation decisions that can be made when building a transaction system using either model, which may mean that specific instances are not best suited for end-to-end transactional solutions. It’s important to realize that this is an implementation choice only.

• Little, M.C., Shrivastava, S.K., Caughey,S.J. and Ingham, D.B. (1997).“Constructing Reliable Web Applications Using Atomic Actions.” Proceedings of the Sixth Web Conference. April.

• Little, M.C. (1997). “Providing End-to-End Transactional Web Applications Using the Object Transaction Service” OMG Success Story.

• “CORBAservices: Common Object Services Specification.” OMG Document Number 95-3-31. March 1995.

• “Java Transaction API 1.0.1 (JTA).” Sun Microsystems. April 1999.

• “Java Transaction Service 1.0 (JTS).” Sun Microsystems. December 1999.

Recently, the question was asked whether or not the models on which current transaction systems are based (e.g., JTA and JTS) are powerful enough to support end-to-end transactional guarantees for applications.

More Stories By Mark Little

Mark Little was Chief Architect, Transactions for Arjuna Technologies Ltd, a UK-based company specialising in the development of reliable middleware that was recently acquired by JBoss, Inc. Before Arjuna, Mark was a Distinguished Engineer/Architect within HP Arjuna Labs in Newcastle upon Tyne, England, where he led the HP-TS and HP-WST teams, developing J2EE and Web services transactions products respectively. He is one of the primary authors of the OMG Activity Service specification and is on the expert group for the same work in J2EE (JSR 95). He is also the specification lead for JSR 156: Java API for XML Transactions. He's on the OTS Revision Task Force and the OASIS Business Transactions Protocol specification. Before joining HP he was for over 10 years a member of the Arjuna team within the University of Newcastle upon Tyne (where he continues to have a Visiting Fellowship). His research within the Arjuna team included replication and transactions support, which include the construction of an OTS/JTS compliant transaction processing system. Mark has published extensively in the Web Services Journal, Java Developer's Journal and other journals and magazines. He is also the co-author of several books including “Java and Transactions for Systems Professionals” and “The J2EE 1.4 Bible.”

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

@ThingsExpo Stories
SYS-CON Events announced today that Datanami has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datanami is a communication channel dedicated to providing insight, analysis and up-to-the-minute information about emerging trends and solutions in Big Data. The publication sheds light on all cutting-edge technologies including networking, storage and applications, and the...
SYS-CON Events announced today that EnterpriseTech has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. EnterpriseTech is a professional resource for news and intelligence covering the migration of high-end technologies into the enterprise and business-IT industry, with a special focus on high-tech solutions in new product development, workload management, increased effi...
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex software systems for startups and enterprises. Since 2009 it has grown from a small group of passionate engineers and business...
SYS-CON Events announced today that GrapeUp, the leading provider of rapid product development at the speed of business, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Grape Up is a software company, specialized in cloud native application development and professional services related to Cloud Foundry PaaS. With five expert teams that operate in various sectors of the market acr...
SYS-CON Events announced today that Ayehu will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara California. Ayehu provides IT Process Automation & Orchestration solutions for IT and Security professionals to identify and resolve critical incidents and enable rapid containment, eradication, and recovery from cyber security breaches. Ayehu provides customers greater control over IT infras...
Artificial intelligence, machine learning, neural networks. We’re in the midst of a wave of excitement around AI such as hasn’t been seen for a few decades. But those previous periods of inflated expectations led to troughs of disappointment. Will this time be different? Most likely. Applications of AI such as predictive analytics are already decreasing costs and improving reliability of industrial machinery. Furthermore, the funding and research going into AI now comes from a wide range of com...
In this presentation, Striim CTO and founder Steve Wilkes will discuss practical strategies for counteracting fraud and cyberattacks by leveraging real-time streaming analytics. In his session at @ThingsExpo, Steve Wilkes, Founder and Chief Technology Officer at Striim, will provide a detailed look into leveraging streaming data management to correlate events in real time, and identify potential breaches across IoT and non-IoT systems throughout the enterprise. Strategies for processing massive ...
SYS-CON Events announced today that Conference Guru has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. A valuable conference experience generates new contacts, sales leads, potential strategic partners and potential investors; helps gather competitive intelligence and even provides inspiration for new products and services. Conference Guru works with conference organi...
SYS-CON Events announced today that Cloud Academy named "Bronze Sponsor" of 21st International Cloud Expo which will take place October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara, CA. Cloud Academy is the industry’s most innovative, vendor-neutral cloud technology training platform. Cloud Academy provides continuous learning solutions for individuals and enterprise teams for Amazon Web Services, Microsoft Azure, Google Cloud Platform, and the most popular cloud com...
SYS-CON Events announced today that CA Technologies has been named "Platinum Sponsor" of SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business - from apparel to energy - is being rewritten by software. From planning to development to management to security, CA creates software that fuels transformation for companies in the applic...
SYS-CON Events announced today that TMC has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo and Big Data at Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Global buyers rely on TMC’s content-driven marketplaces to make purchase decisions and navigate markets. Learn how we can help you reach your marketing goals.
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
"MobiDev is a Ukraine-based software development company. We do mobile development, and we're specialists in that. But we do full stack software development for entrepreneurs, for emerging companies, and for enterprise ventures," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA
Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists looked at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deliver...
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...
SYS-CON Events announced today that Enzu will exhibit at SYS-CON's 21st Int\ernational Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive advantage. By offering a suite of proven hosting and management services, Enzu wants companies to focus on the core of their ...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.