Welcome!

Java IoT Authors: Liz McMillan, Elizabeth White, Carmen Gonzalez, Yeshim Deniz, Yakov Fain

Related Topics: Java IoT

Java IoT: Article

Enabling A File System As A Transactional Resource

Building An Adapter

Transactional support is fundamental to application development. While most data sources are transactional in nature, some data sources like the file system are not.

In typical J2EE deployments, applications often need to interface with other applications through file-based messages. Such deployments would benefit from an adapter that would buffer all file operations and provide a transactional view to the file system.

This article, targeted at developers, demonstrates how J2EE Connector Architecture and JTA specifications can be implemented to build such an adapter, XAFileConnector. We'll also suggest enhancements to extend the adapter functionality in light of the new proposed draft connector specifications.

Most applications are bound to the local file system. They use the file system to store and share data. Often the success of interactions with the file system needs to be tied to the successful completion of other actions involving other Enterprise Information Systems (EIS). For instance, we often encounter situations where a record is to be deleted from the database only after it has been successfully written to a file. While the same can be easily built into the application logic, it would be infinitely more neater if we could develop an extension on the file system that would help us include the file operations such as "create", "read", "update", and "delete" as part of a transactional unit of work that could be committed or rolled back as a group. This extension is called a Resource Adapter. The ubiquitous presence of the file system and the role it plays in Enterprise Application Integration strategy justify the need to build such a resource adapter.

For a J2EE-managed environment, i.e., an environment characterized by the presence of a container for runtime support, J2EE recommends Connector Architecture as a standard way of integrating heterogeneous EISs with the application server. The Connector Architecture specifies application- and system-level contracts to be implemented by the EIS. The resource adapter implements the EIS side of the contracts. The adapter uses a native interface specific to the underlying EIS, runs in the containers address space, and manages access to the EIS resources. Adherence to the Connector contracts on one hand ensures that the same adapter can be deployed on different vendors' containers, while ensuring that application and tool developers have a standard interface to program to for accessing different EIS systems. Among the set of contracts that need to be implemented by the resource adapter to seamlessly plug-in to any J2EE platform is the XAResource transaction contract. This contract defines the communication interfaces between the parties involved in a distributed transaction. Implementation of this interface by the adapter enables seamless propagation of transaction context and allows the associated resource manager (RM) to participate in a two-phase commit protocol along with other resources.

With this introduction we proceed to discuss what these contracts are and what it takes to implement these contracts to build an adapter.

Understanding and Implementing the Contracts
The application contract manifests itself as a Common Client Interface (CCI). CCI is the client side of the adapter and performs the following roles:

  • Specifies how application components connect to the EIS through a resource adapter
  • Provides an API for coding EIS function calls and retrieving results

    Connecting to the EIS
    ConnectionFactory is the application's gateway to the EIS. It's instantiated at the time of deployment by the container and a reference to it, or its serialized representation, is bound to the JNDI namespace. Later the application components do a JNDI lookup to get hold of the factory to create connection objects to the EIS. The ConnectionFactory does not represent the actual connection repository, nor is the connection the actual physical connection to the EIS. Both are client-side proxies that nevertheless maintain handles to the actual factory and connection objects, respectively. The container plays the role of an intermediary between the real objects and their proxies, and the mechanism allows the container to inject value-added services by intercepting calls made on the proxies before they're actually delegated to the real objects.

    When a ConnectionFactory encounters a getConnection request, it delegates it to the ConnectionManager instance.

    public Connection getConnection()
    throws ResourceException {

    return (Connection)connMgr.allocateConnection
    (mngdConnFactory, new ConnectionRequestInfoImpl());
    }

    In a managed environment the ConnectionManager implementation is provided by the container. The ConnectionManager maintains a pool of connections corresponding to each factory. The ConnectionManager checks whether it can service the request from the pool. If not, it requests the real factory (ManagedConnectionFactory) to create a new connection.

    How does the container know about the availability status of a connection?

    The Connector Architecture prescribes an elaborate mechanism of listeners to implement callbacks. Whenever a new connection is created, the container registers a listener with it. The connection raises event notifications to intimate the container about the happenings on the connection. When a close is called on the connection proxy it terminates its association with the physical connection. The physical connection in turn generates an event. All the listeners registered on the connection can react to the event. The container on its part uses the event to change the availability status of the connection from in-use to available, i.e., if the connection is not participating in a transaction. If the connection is participating in a transaction, the container waits for the transaction to commit before it can make the connection available (see Listing 1).

    Invoking EIS Functions
    An InteractionSpec implementation holds properties for driving an interaction with an EIS instance. In our case each property maps to a File Operation that may be performed.

    public static final int CREATE = 10;
    public static final int UPDATE = 20;
    public static final int DELETE = 30;
    public static final int MOVE = 40;
    public static final int SKIP = 50;
    public static final int READ = 60;

    Since CCI is EIS independent it cannot use any of the EIS data structures. To get over this, CCI introduces the concept of a record. A record is a generic representation of data that's exchanged with the EIS. More specific implementations to represent hierarchical, tabular data collections can also be implemented. XAFileConnector extends the MappedRecord, which is a key value representation of record elements for both input and output records (see Listing 2).

    CCI defines an interaction interface that allows a client to interact with EIS. An interaction represents a single communication with the EIS, an EIS function call. An interaction instance is obtained from a connection. The interaction instance is required to maintain its association with the connection instance. The interaction delegates its execution to the connection, which in turn delegates its execution to the associated ManagedConnection (see Listing 3).

    Transaction Contract
    A transaction as we know it is a set of operations performed in such a way that the state of the system after the transaction is either the cumulative effect of state changes due to successful individual operations or is the initial state before the commencement of the transaction in case one or more operations happens to fail. Transactions can be classified into two types based on the number of resources participating.
    1.  A local transaction performs operations on a single RM. The local transactions can be further classified based on the manner of demarcating transaction boundaries.

  • javax.resource.cci.LocalTransaction: These run under the control of the RM. The transaction object can be obtained from the connection object and used to demarcate transaction boundaries somewhat analogous to the way JDBC setAutoCommit(false) and commit APIs are used to demarcate transaction boundaries.
  • javax.resource.spi.LocalTransaction: Work in this type of transaction can accrue over multiple client components. A client component may do some work on a resource and pass control to another component, which may again do some work while still being part of the same transaction. The transaction context is passed along with the control, and it contains the following information:
    - A reference to the transaction manager (TM), the external entity that is used to demarcate transaction boundaries
    - A globally unique transaction identifier, XID, generated by the TM
    - A transaction timeout time

    2.  Global/distributed transactions - work in this type of transaction can span multiple RMs. The transaction manager demarcates the transaction boundaries. A TM coordinates the activities on the participating RMs using the XAResource interface. An RM knows only about the work it does for its transaction branch. The XAResource interface implements the two-phase commit protocol between the RMs and the TM. The XAResource interface allows for a one-phase optimization in case only one resource is participating.

    With this brief introduction to transactions we'll start with how we can incorporate transactional ability into our file system adapter.

    For a file system adapter to be able to roll back a transaction, i.e., revert to state prior to the start of a transaction, an adapter must either memorize the original state or it must be able to get back to the original state by performing certain anti-operations that reverse the effect of the operations. To ensure feasibility and consistency at all times, the anti-operations have to be performed in the order that is exactly the reverse order in which the operations were performed. Similarly, committing a transaction should flush all transactional logs to free all memory.

    For each file operation on the connection that modifies the state of the file system (nonread only call), XAFileConnector adds a WorkItem to the associated transaction's work list. If there's no active transaction associated with the connection, it autocommits the work and no WorkItem is created. The WorkItem stores enough information about the associated operation to be able to reverse its effect in case of a rollback or to do a cleanup of the backup in case of a commit. A work area is designated to store the information needed for restoration in the event of a rollback (see Listing 4).

    Note how MOVE and CREATE file operation do not have any associated cleanup during operation commit while UPDATE and DELETE operations delete the backups.

    Depending on the kind of transaction running, javax.resource.cci.LocalTransaction, javax.resource.spi. LocalTransaction, or XAResource keeps track of the work being done in the transaction impending completion.

    CCI Local Transaction implementations are obtained directly from the connection, without the container having any role to play. Since the container is not directly involved, it is not intimated of the state of the transaction, so CCI implementations of local transactions differ from SPI implementations in the sense that they need to raise event notifications to apprise the container of transaction life-cycle events. The events help the container in connection pool management. Since there's only a marginal difference in functionality, the same class is used to implement both forms of local transactions with a flag identifying the type of transaction the current instance represents.

    Our implementation model exhibits the following relationships shown in Figure 1.

     
    Figure 1

    Each ManagedConnection can provide an XAResource implementation for distributed transaction management. The beginTransaction call associates the application component's thread of control with a global transaction. The TM generates a globally unique XID and calls start on all open RMs that are linked with the thread. The RMs are enlisted with the transaction. To guarantee scalability of the Connector Architecture, it recommends that a physical connection and hence the associated XAResource can participate in multiple transactions, but at any point only one of these transactions can be active. To ensure this, the start call first checks that the associated connection isn't running a local transaction nor is the XAResource instance associated with an active transaction. Once assured, it associates the passed XID with the XAResource instance and activates the transaction. All work done henceforth on this physical connection, until the transaction is suspended or completed, accumulates against the active transaction.

    The start call is accompanied by the following flags:

  • TMNOFLAGS: Indicates that it is a new transaction. A call to getTransactionState with flag true creates new entries for the new transaction and initializes the instance.
  • TMRESUME: Indicates a suspended transaction is being resumed. A call to getTransactionState with flag false means no new entries need to be made. The state of the transaction is retrieved and the current instance is initialized with that state. The state of the transaction is also changed from SUSPENDED to NOT SUSPENDED.
  • TMJOIN: A two-phase optimization, the TM, when it detects that a particular RM instance is already participating in a transaction, instead of creating a new branch decides to merge all work done on a single RM against a single XID. By doing this it saves on running the two-phase protocol on all branches separately. No new entries have to be made and the state of the transaction remains throughout NOT SUSPENDED (see Listing 5).

    The application component calls commit to make the effects of the transaction permanent. The TM first calls end for each involved RM, from the AP's thread of control, to dissociate the thread from the global transaction. The TM then executes the two-phase commit protocol.

    STAGE 1 is the prepare stage or the voting stage. TM calls prepare for each RM that was associated with the global transaction. RMs express their readiness to commit the transaction. A single negative vote ensures rollback. A prepare attempt on a suspended transaction would throw back a protocol exception. Similarly it throws back an exception, albeit a different one, when the transaction has been marked earlier for rollback due to internal errors. Prepare stage also offers a two-phase optimization. A transaction that has not changed the state of the RM doesn't need to go through the second phase if prepare returns XA_RDONLY (see Listing 6).

    Stage 2 is the commit stage. If all RMs return success from prepare, the TM records a decision to commit the transaction and calls commit for each RM. The XA specification allows for an optimization in this procedure. If the TM has dealt with only one subordinate RM in the global transaction, it can omit Phase 1 by directly calling commit with the onePhase flag set to true (see Listing 7).

    Deployment Process
    Before we can start using the adapter, we need to deploy it. All adapter interfaces, classes, native libraries, etc., along with the Deployment Descriptor (DD), are packaged in a .rar archive. In situations where multiple J2EE applications need to share the XAFileConnector, it can be deployed directly into an application server as a standalone unit. However, the resource adapter module may also be bundled with a J2EE application, if it is needed only by a single application.

    The Deployment Descriptor provides a lot of information to the container, including:

  • General information such as name and version of adapter, vendor name, licensing requirements, etc.
  • Names of concrete classes for certain core interfaces; for example, we have to provide the name of the class that implements the ManagedConnectionFactory Interface.
  • Information about the kind of support built into the adapter. This allows the container to perform certain optimizations. XAFileConnector offers full support for XATransaction. No authentication support is needed as all native calls to connect to the resource, i.e., the local file system, are made in the current user's context.
  • Configurable properties that are dependent on the deployment environment. The property naming conventions follow the JavaBean standard. XAFileConnector uses a property by the name of workingFolder to store the name of the folder where transaction logs of the running transactions are kept. The deployer sets this property at the time of deployment by invoking the corresponding setter method. The method checks for the existence of the folder and sufficient access permissions before returning (see Listings 8 and 9).

    To be able to successfully deploy an adapter, a better understanding of the deployment process is essential. We will briefly walk through the responsibilities of the deployment code and how it interacts with the adapter code.

  • The deployment code looks at the DD to find out which class implements the ManagedConnectionFactory Interface. It dynamically creates an instance of the same and configures properties on the instance.XAFile The Connector uses only one property workingFolder as discussed earlier.
  • In a managed environment the deployment code instantiates a ConnectionManager specific to the container and obtains a ConnectionFactory instance from ManagedConnectionFactory, as shown in the following implementation:

    public Object createConnectionFactory(ConnectionManager cxManager)
    throws ResourceException
    {
    return new ConnectionFactoryImpl(this, cxManager);
    }

    Note how the ConnectionManager instance is passed around while creating the ConnectionFactory. The mechanism helps associate the two factories and the ConnectionManager. The ConnectionManager is the ConnectionFactory's interface to the container.

  • The deployment code then binds the ConnectionFactory instance in the JNDI namespace from where it's later looked up by the application component.

    Writing a Client
    Clients needing to interact with the file system will have to use the CCI interface. They cannot use the java.io package to access the file system if they want the transactional support. While the approach may look nonintuitive and tiresome, which in fact it is, it is not as big a pain as you would imagine it to be.

    By this time we would have understood how to acquire a connection to the file system.

  • Once connected we can obtain an interaction from the connection.

    Interaction interaction = conn.createInteraction();

    Interaction maintains an association with the connection from which it was created and executes on the same connection.

  • A connection object also serves as the source for RecordFactory. RecordFactory hands out all types of records. Records are the means of exchanging data, both incoming and outgoing, with the files system.

    RecordFactory rf = cf.getRecordFactory();
    MappedRecord in = rf.createMappedRecord("IN");
    MappedRecord out = rf.createMappedRecord("OUT");

  • Assuming that we wish to create a file, the only parameters we would need to pass are the folder name in which to create the file and the file name. Initialize the records with the input parameters.

    in.put("SOURCE_DIR", "E:\\bea\\weblogic700\\FILES1");
    in.put("SOURCE_FILENAME", "create.txt");

  • Now the only information that needs to be passed is our intention of actually creating a file. This is specified when we instantiate an object of the class InteractionSpecImpl.

    interaction.execute(new
    InteractionSpecImpl(InteractionSpecImpl.CREATE), in, out);

  • Handle the exceptions and examine the OUT record for returned values.
  • Close all open resources.

    What Further?
    The proposed draft connector specification makes provisions for inbound communication through a Message Inflow Contract and also includes a Work Management Contract. The inclusion of the two would make it feasible for the adapter to work as a poller.

    The Work Management Contract would allow the adapter to monitor folders for incoming files. The activity can be performed by submitting work to the container. The contract would save the adapter the task of creating its own threads thus allowing the container to exercise better control over its runtime environment.

    The Message Inflow Contract would allow the adapter to trigger action in the event of receiving a file. The situation can be thought of as analogous to a JMS situation, with the adapter instead of an MDB delivering messages by polling on a folder instead of a queue.

    Better concurrency control can also be built into the adapter using the new improved I/O support found in newer Java runtimes, 1.4 onwards.

  • More Stories By Ashish Garg

    Ashish Garg is a Senior Technical Specialist with Infosys Technologies Ltd. He has more than 4 years of experience in J2EE Technologies.

    More Stories By Sandip Mane

    Sandip Mane is a senior technical specialist with Infosys Technologies Ltd. and specializes in WebLogic

    Comments (6) View Comments

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


    Most Recent Comments
    Michael Wolff 06/21/06 09:24:22 AM EDT

    Very interesting article, but it would be great to get the complete source code instead of the snippets. The links mentioned in the other comments does not work, they produce an HTTP 404 error.

    lperon 11/29/05 05:35:09 AM EST

    The source code is not available following indicated link:

    http://www.sys-con.com/magazine/?issueid=238&src=false
    http://www.sys-con.com/read/issues/archives/
    404 Not Found

    JDJ Editorial Staffer 06/02/04 11:04:52 AM EDT

    The source code is here:
    http://www.sys-con.com/magazine/?issueid=238&src=false

    Ernest Muljadi 06/02/04 12:40:15 AM EDT

    Please make the SOURCE code available or post it to my e-mail address

    skr 04/07/04 05:47:58 PM EDT

    Please post the source code.

    Peter Reinhardt 01/08/04 02:25:40 AM EST

    great article. it would have been nice if you provided a fully functional rar and not only the code snippets.

    Peter.

    @ThingsExpo Stories
    SYS-CON Events announced today the How to Create Angular 2 Clients for the Cloud Workshop, being held June 7, 2016, in conjunction with 18th Cloud Expo | @ThingsExpo, at the Javits Center in New York, NY. Angular 2 is a complete re-write of the popular framework AngularJS. Programming in Angular 2 is greatly simplified. Now it’s a component-based well-performing framework. The immersive one-day workshop led by Yakov Fain, a Java Champion and a co-founder of the IT consultancy Farata Systems and...
    What a difference a year makes. Organizations aren’t just talking about IoT possibilities, it is now baked into their core business strategy. With IoT, billions of devices generating data from different companies on different networks around the globe need to interact. From efficiency to better customer insights to completely new business models, IoT will turn traditional business models upside down. In the new customer-centric age, the key to success is delivering critical services and apps wit...
    Join us at Cloud Expo | @ThingsExpo 2016 – June 7-9 at the Javits Center in New York City and November 1-3 at the Santa Clara Convention Center in Santa Clara, CA – and deliver your unique message in a way that is striking and unforgettable by taking advantage of SYS-CON's unmatched high-impact, result-driven event / media packages.
    In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, will provide an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life ...
    SYS-CON Events announced today that BMC Software has been named "Siver Sponsor" of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. BMC is a global leader in innovative software solutions that help businesses transform into digital enterprises for the ultimate competitive advantage. BMC Digital Enterprise Management is a set of innovative IT solutions designed to make digital business fast, seamless, and optimized from mainframe to mo...
    SYS-CON Events announced today that MobiDev will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex software systems for startups and enterprises. Since 2009 it has grown from a small group of passionate engineers and business managers to a full-scale mobile software company with over 200 develope...
    As cloud and storage projections continue to rise, the number of organizations moving to the cloud is escalating and it is clear cloud storage is here to stay. However, is it secure? Data is the lifeblood for government entities, countries, cloud service providers and enterprises alike and losing or exposing that data can have disastrous results. There are new concepts for data storage on the horizon that will deliver secure solutions for storing and moving sensitive data around the world. ...
    SoftLayer operates a global cloud infrastructure platform built for Internet scale. With a global footprint of data centers and network points of presence, SoftLayer provides infrastructure as a service to leading-edge customers ranging from Web startups to global enterprises. SoftLayer's modular architecture, full-featured API, and sophisticated automation provide unparalleled performance and control. Its flexible unified platform seamlessly spans physical and virtual devices linked via a world...
    SYS-CON Events announced today that ContentMX, the marketing technology and services company with a singular mission to increase engagement and drive more conversations for enterprise, channel and SMB technology marketers, has been named “Sponsor & Exhibitor Lounge Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York City, New York. “CloudExpo is a great opportunity to start a conversation with new prospects, but what happens after the...
    Companies can harness IoT and predictive analytics to sustain business continuity; predict and manage site performance during emergencies; minimize expensive reactive maintenance; and forecast equipment and maintenance budgets and expenditures. Providing cost-effective, uninterrupted service is challenging, particularly for organizations with geographically dispersed operations.
    The IoTs will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, will demonstrate how to move beyond today's coding paradigm and share the must-have mindsets for removing complexity from the development proc...
    SYS-CON Events announced today TechTarget has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. TechTarget is the Web’s leading destination for serious technology buyers researching and making enterprise technology decisions. Its extensive global networ...
    SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management...
    The essence of data analysis involves setting up data pipelines that consist of several operations that are chained together – starting from data collection, data quality checks, data integration, data analysis and data visualization (including the setting up of interaction paths in that visualization). In our opinion, the challenges stem from the technology diversity at each stage of the data pipeline as well as the lack of process around the analysis.
    Designing IoT applications is complex, but deploying them in a scalable fashion is even more complex. A scalable, API first IaaS cloud is a good start, but in order to understand the various components specific to deploying IoT applications, one needs to understand the architecture of these applications and figure out how to scale these components independently. In his session at @ThingsExpo, Nara Rajagopalan is CEO of Accelerite, will discuss the fundamental architecture of IoT applications, ...
    A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, wh...
    SYS-CON Events announced today that Tintri Inc., a leading producer of VM-aware storage (VAS) for virtualization and cloud environments, will exhibit at the 18th International CloudExpo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, New York, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
    SYS-CON Events announced today that MangoApps will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. MangoApps provides modern company intranets and team collaboration software, allowing workers to stay connected and productive from anywhere in the world and from any device. For more information, please visit https://www.mangoapps.com/.
    SYS-CON Events announced today that Alert Logic, Inc., the leading provider of Security-as-a-Service solutions for the cloud, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Alert Logic, Inc., provides Security-as-a-Service for on-premises, cloud, and hybrid infrastructures, delivering deep security insight and continuous protection for customers at a lower cost than traditional security solutions. Ful...
    In his session at 18th Cloud Expo, Bruce Swann, Senior Product Marketing Manager at Adobe, will discuss how the Adobe Marketing Cloud can help marketers embrace opportunities for personalized, relevant and real-time customer engagement across offline (direct mail, point of sale, call center) and digital (email, website, SMS, mobile apps, social networks, connected objects). Bruce Swann has more than 15 years of experience working with digital marketing disciplines like web analytics, social med...