Welcome!

Java Authors: Liz McMillan, Trevor Parsons, Elizabeth White, Carmen Gonzalez, Plutora Blog

Related Topics: DevOps Journal, Java, SOA & WOA, Linux, Cloud Expo, Big Data Journal

DevOps Journal: Article

The DevOps Database | Part 4

A culture of continual experimentation and learning

In the final post in this series about bringing DevOps patterns to database change management, we’re going to discuss the Third Way.  Here’s a refresher on the Third Way from the introductory post in this series:

The Third Way: Culture of Continual Experimentation & Learning – This way emphasizes the benefits that can be realized through embracing experimentation, risk-taking, and learning from failure.  By adopting this kind of attitude, experimentation and risk-taking lead to innovation and improvement while embracing failure allows the organization to produce more resilient products and sharpen skills that allow teams to recover more quickly from unexpected failure when it does occur.

The Third Way is by far the most intriguing of the “The Ways” to me.  I’ve spent the lion’s share of my career in early stage start-ups where cycles of experimentation, learning, and failure are the norm.  When bringing a new product or service to market, your latest release is never your last.  It may not even be the last release this week.  You are constantly experimenting with new workflows and technology, learning about your target market, and getting more valuable information from your early failures than your early successes.  The Third Way is crucial to the success of an early stage company.

While the benefits of the Third Way still apply to more established companies and product lines, practicing it becomes more difficult.  The potential negatives of experimentation and risk-taking are much harder to stomach when you have a large base of paying customers with SLAs. This aversion to risk is most acute when you’re talking about your data platform where outages, performance problems and data loss are not an option. Complicating matters further is how difficult it can be to unwind the database changes that were affected to support a specific version of your app.  Application code can usually be uninstalled and replaced with the previous working version fairly simply should problems arise. Reverting the database changes that support that version of the application is more akin to defusing a bomb.  Database changes must be reverted delicately and meticulously to avoid errors and omissions that could negatively impact your data platform.

What DBAs and release managers need to facilitate experimentation and risk taking on the data platform is a special combination of tools and process.  This combination should make it easy to identify the root cause of issues, quickly remediate problems caused by application schema structure, and revert to a previous version of the schema safely.  When we started Datical, we spent several hours in conversation and at white boards exploring these unique needs and hammering out a path to usher the Third Way into regular database activity.

A Rollback Designed With Every Change
The biggest problem with experimentation in the data platform is how difficult it is to move backward and forward through your schema’s version history.  We feel the best way to bring more flexibility to the process of upgrading and reverting schema is an attitude shift.  Your rollback strategy for each database change must become as important as the change itself.  The best time to craft your rollback strategy is when the change itself is being designed.  When the motivation for the change is fresh in your mind and the dependencies of the object being created, dropped or altered are clearly mapped out, a developer or DBA can better craft a rollback strategy for which every contingency has been considered.  This leads to a stronger safety net and makes your application schema as agile and easily managed as your application code. The database is no longer preventing you from being bold but quickly and safely moving between versions to accommodate experimentation.

The Richness of Model Based Comparison & Remediation
I’m a native Texan, born and raised in Austin.  I know the jokes about how proud and vocal Texans can be about their home state.  That being said, I spend more time telling people about the model based approach Datical has applied to Database Change Management than I do telling them how wonderful it is that I was fortunate enough to be born in the best state in the country.  The advantages of the model based approach really shine through when it comes to experimentation and troubleshooting.  The model allows you to annotate all objects and modifications to your application’s schema with the business reason that prompted them.  This detailed history is invaluable when designing new changes or refactoring your schema as part of an experimental exercise.  You immediately know what objects are most crucial, what your dependencies are, what areas you need to tread lightly in, and what areas are ripe for experimentation due to the changing needs of your business.  Designing intelligently eliminates risk.

Troubleshooting with models is also dramatically faster and more reliable than other methods. Programmatically comparing models allows you to determine the differences between two databases much more quickly than manually comparing diagrams or SQL scripts.  You know with certainty exactly what has changed and what is missing in a fraction of the time that human review takes.

Once you have identified the differences, remediation is as simple plugging one, some or all of the determined differences into the model and deploying those changes to the non-compliant instance.

Flexible Quick Rollback
If you’ve taken our advice to implement your rollback strategy when you implement a change, recovering from disaster becomes testable, fast, and simple.  Before going to production or a sensitive environment, you should always test your rollback steps in dev, test and staging.  This will allow you to make any tweaks or changes to your rollback strategy before you are in a pinch.  Think of it like testing your smoke alarm.  Hopefully you’ll never need it, but it’s nice to know that it’ll work if you do.

Let’s say the worst happens. You deploy a new set of database changes and the application performance degrades or errors are logged.  The decision is made to revert the entire installation to the previous version.  Because you have carefully designed, tested and refined your rollback strategy, rolling back the database changes becomes a push button operation or a single invocation of a command line tool.  No more running disparate SQL scripts or undoing changes on the fly.  You can be confident that your database has been returned to the same state it was in before the upgrade.

Summary
The database has long been handled with kid gloves and for good reason.  Data is a precious resource for consumers and businesses.  Consumers provide data to businesses and trust that it will be kept safe and used to offer them better products and services.  Businesses rely on data to strategize, grow, and become more efficient and profitable.  It is the lifeblood of our economy.  As our ability to collect and process data becomes greater and greater, the rate at which an enterprise must move on what is learned from that data becomes linearly faster.  Data must be kept safe, but the database must become more agile to accommodate the growing pressure for faster value realization of business intelligence initiatives.  DevOps patterns hold the key to this necessary agility while maintaining or improving the security and integrity of data stores.  The database and DBAs need to be brought into the application design process earlier and must be treated as the first class stakeholder and commodity that they are. Companies that acknowledge this and move to adopt DevOps patterns and include their database teams will be at a distinct competitive advantage to those that don’t.  Don’t miss the boat!

More Stories By Pete Pickerill

Pete Pickerill is Vice President of Products and Co-founder of Datical. Pete is a software industry veteran who has built his career in Austin’s technology sector. Prior to co-founding Datical, he was employee number one at Phurnace Software and helped lead the company to a high profile acquisition by BMC Software, Inc. Pete has spent the majority of his career in successful startups and the companies that acquired them including Loop One (acquired by NeoPost Solutions), WholeSecurity (acquired by Symantec, Inc.) and Phurnace Software.

@ThingsExpo Stories
IoT is still a vague buzzword for many people. In his session at @ThingsExpo, Mike Kavis, Vice President & Principal Cloud Architect at Cloud Technology Partners, discussed the business value of IoT that goes far beyond the general public's perception that IoT is all about wearables and home consumer services. He also discussed how IoT is perceived by investors and how venture capitalist access this space. Other topics discussed were barriers to success, what is new, what is old, and what the future may hold. Mike Kavis is Vice President & Principal Cloud Architect at Cloud Technology Pa...
Dale Kim is the Director of Industry Solutions at MapR. His background includes a variety of technical and management roles at information technology companies. While his experience includes work with relational databases, much of his career pertains to non-relational data in the areas of search, content management, and NoSQL, and includes senior roles in technical marketing, sales engineering, and support engineering. Dale holds an MBA from Santa Clara University, and a BA in Computer Science from the University of California, Berkeley.
The Internet of Things (IoT) is rapidly in the process of breaking from its heretofore relatively obscure enterprise applications (such as plant floor control and supply chain management) and going mainstream into the consumer space. More and more creative folks are interconnecting everyday products such as household items, mobile devices, appliances and cars, and unleashing new and imaginative scenarios. We are seeing a lot of excitement around applications in home automation, personal fitness, and in-car entertainment and this excitement will bleed into other areas. On the commercial side, m...
The Industrial Internet revolution is now underway, enabled by connected machines and billions of devices that communicate and collaborate. The massive amounts of Big Data requiring real-time analysis is flooding legacy IT systems and giving way to cloud environments that can handle the unpredictable workloads. Yet many barriers remain until we can fully realize the opportunities and benefits from the convergence of machines and devices with Big Data and the cloud, including interoperability, data security and privacy.
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
"People are a lot more knowledgeable about APIs now. There are two types of people who work with APIs - IT people who want to use APIs for something internal and the product managers who want to do something outside APIs for people to connect to them," explained Roberto Medrano, Executive Vice President at SOA Software, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Performance is the intersection of power, agility, control, and choice. If you value performance, and more specifically consistent performance, you need to look beyond simple virtualized compute. Many factors need to be considered to create a truly performant environment. In his General Session at 15th Cloud Expo, Harold Hannon, Sr. Software Architect at SoftLayer, discussed how to take advantage of a multitude of compute options and platform features to make cloud the cornerstone of your online presence.
SYS-CON Media announced that Splunk, a provider of the leading software platform for real-time Operational Intelligence, has launched an ad campaign on Big Data Journal. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. The ads focus on delivering ROI - how improved uptime delivered $6M in annual ROI, improving customer operations by mining large volumes of unstructured data, and how data tracking delivers uptime when it matters most.
In this Women in Technology Power Panel at 15th Cloud Expo, moderated by Anne Plese, Senior Consultant, Cloud Product Marketing at Verizon Enterprise, Esmeralda Swartz, CMO at MetraTech; Evelyn de Souza, Data Privacy and Compliance Strategy Leader at Cisco Systems; Seema Jethani, Director of Product Management at Basho Technologies; Victoria Livschitz, CEO of Qubell Inc.; Anne Hungate, Senior Director of Software Quality at DIRECTV, discussed what path they took to find their spot within the technology industry and how do they see opportunities for other women in their area of expertise.
DevOps Summit 2015 New York, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity.
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, discussed how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money!
Connected devices and the Internet of Things are getting significant momentum in 2014. In his session at Internet of @ThingsExpo, Jim Hunter, Chief Scientist & Technology Evangelist at Greenwave Systems, examined three key elements that together will drive mass adoption of the IoT before the end of 2015. The first element is the recent advent of robust open source protocols (like AllJoyn and WebRTC) that facilitate M2M communication. The second is broad availability of flexible, cost-effective storage designed to handle the massive surge in back-end data in a world where timely analytics is e...
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
"There is a natural synchronization between the business models, the IoT is there to support ,” explained Brendan O'Brien, Co-founder and Chief Architect of Aria Systems, in this SYS-CON.tv interview at the 15th International Cloud Expo®, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
The Internet of Things will put IT to its ultimate test by creating infinite new opportunities to digitize products and services, generate and analyze new data to improve customer satisfaction, and discover new ways to gain a competitive advantage across nearly every industry. In order to help corporate business units to capitalize on the rapidly evolving IoT opportunities, IT must stand up to a new set of challenges. In his session at @ThingsExpo, Jeff Kaplan, Managing Director of THINKstrategies, will examine why IT must finally fulfill its role in support of its SBUs or face a new round of...
The BPM world is going through some evolution or changes where traditional business process management solutions really have nowhere to go in terms of development of the road map. In this demo at 15th Cloud Expo, Kyle Hansen, Director of Professional Services at AgilePoint, shows AgilePoint’s unique approach to dealing with this market circumstance by developing a rapid application composition or development framework.

ARMONK, N.Y., Nov. 20, 2014 /PRNewswire/ --  IBM (NYSE: IBM) today announced that it is bringing a greater level of control, security and flexibility to cloud-based application development and delivery with a single-tenant version of Bluemix, IBM's platform-as-a-service. The new platform enables developers to build ap...

“The age of the Internet of Things is upon us,” stated Thomas Svensson, senior vice-president and general manager EMEA, ThingWorx, “and working with forward-thinking companies, such as Elisa, enables us to deploy our leading technology so that customers can profit from complete, end-to-end solutions.” ThingWorx, a PTC® (Nasdaq: PTC) business and Internet of Things (IoT) platform provider, announced on Monday that Elisa, Finnish provider of mobile and fixed broadband subscriptions, will deploy ThingWorx® platform technology to enable a new Elisa IoT service in Finland and Estonia.
Advanced Persistent Threats (APTs) are increasing at an unprecedented rate. The threat landscape of today is drastically different than just a few years ago. Attacks are much more organized and sophisticated. They are harder to detect and even harder to anticipate. In the foreseeable future it's going to get a whole lot harder. Everything you know today will change. Keeping up with this changing landscape is already a daunting task. Your organization needs to use the latest tools, methods and expertise to guard against those threats. But will that be enough? In the foreseeable future attacks w...