Welcome!

Java IoT Authors: Pat Romanski, Yeshim Deniz, Liz McMillan, Zakia Bouachraoui, Elizabeth White

Related Topics: @DevOpsSummit, Java IoT, Microservices Expo, Linux Containers, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Blog Feed Post

Provisioning the Entire Stack | @DevOpsSummit #DevOps #Microservices

We’re currently in the process of stringing together applications at the various layers to do server provisioning

As we move into the world of complete data center automation, there is a whole new selection of issues that we are learning to resolve – from custom hardware to a variety of provisioning tools at each level of automation. These are not unexpected issues, but they certainly provide us with plenty to do while we’re trying to reduce the amount of busy work we have to do.

We’re currently in the process of stringing together applications at the various layers to do server provisioning, application provisioning, and (for internal apps at least) application deployment. The options out there are a pretty broad swath that runs from all-in-one solutions like Puppet with Razor to specialized solutions hooked together to run the entire automation process. The end result is the same, the differences are in implementation, vendor lock-in, and whether a full solution that can be implemented all at once is more important to an organization than a solution that has the most suited parts that can be implemented in parallel but have to be tied together.

Image complements of University Corporation for Atmospheric Research.

For those not familiar with myself or my employer StackIQ, the solutions available from my company run the gamut from specialized open source best-of-breed server provisioning (Stacki) to end-to-end Hadoop deployment. Which a given organization uses just depends upon which solution and what the organization’s relationship with StackIQ is.

As such, I am comfortable discussing the strengths and weaknesses of both styles of automation. After all, I’ve worked with both extensively, and can see the use cases that drive either one. There is no shortage of vendors in this space, and I will even mention a couple other than StackIQ, but my examples and use-cases will come from the various editions of Stacki, because I have worked on them more with that platform than others.

At the bottom level, pure-play server provisioning tools aim for best-of-breed usefulness. Open Source Stacki, for example, is focused on installing RHEL and CentoOS on target servers and configuring them to begin use. Of course few of the solutions on the market stop there, though a couple do. Stacki, for example, can be extended to install and configure agents from most of the application provisioning tools on the market. This means that server provisioning can hand off to application provisioning to achieve the full automation solution. It can also be extended to install specific applications as part of the OS install process. So a give server or set of servers can be configured to run Tomcat via installation, so that when Stacki finishes, the application server is installed and configured.

The drawback to this entry level is, of course, that such extensions must be developed by the user. While it is certainly possible to share extensions between users (for most tools anyway), and solicit feedback on the extensions being developed, there will still be work setting it up to work correctly. Some organizations – particularly those seeking a significant amount of control in their installations – find this to be an acceptable trade-off. Others would rather some of this foot work be done for them, or at least that they had ready access to people who specialize in this sort of thing.

That’s where the middle level comes in. This solution offers the functionality of the entry level, but adds either pre-built packages to install specific applications (like puppet agents or Tomcat) that can be easily marked as part of the install for a server or group of servers, or alternatively they offer the help of experts in setting these installations up – because once they’ve been configured, they (1) Don’t generally need to change much except to upgrade when new releases of the app come out and (2) Serve as examples that can be used to make other application customizations.

The issues with this solution are that it still takes time to get everything configured the first time, and it still requires a relatively intimate knowledge of the applications being deployed. For some shops these are not issues at all, for others they are show-stoppers that require significantly more investment than the organization is willing or able to make. That’s why there is the third level of integration for automation tools.

For some solutions in some organizations, the best answer is “push a button, get fully functional deployed servers running the software chosen.” A good example of this is big data. Deploying Hadoop by hand can be a daunting experience, yet timelines for Hadoop deployment far too often are tight. The third tier of integration is the system that knows what to do, and can juggle all the disparate pieces – hardware config, OS install, application install, configuration – at once. Push a button, reboot the machines intended for the Hadoop cluster, and a short time later there is a running Hadoop instance. I’m using Hadoop as an example here because of the glaring difference between when I’ve deployed it by hand and by automated provisioning (weeks to hours), but complex n-tier architectures stand to gain as much.

With these tools, an organization gets support built-in for a given set of applications, or gets the plug-ins to support a given application and adds them to one of the lower tiers. Then it is a simple case of defining (through command line, APIs, or spreadsheets) how the complex system should be deployed across target servers, and hitting a button. In a short time, the system is up and running with the target servers fully installed and doing their part. The issues with this solution are that there is a limit to the number of complex applications that have pre-packaged solutions, and of course, no internally developed application does. Professional services can help with this problem, but that adds expense and timeline to the project in a manner similar to the second tier of integration.

The best solution to these issues should be able to support all of the above integration tiers, but it should also be even more adaptable. There are several very good application provisioning tools on the market – Puppet, Chef, Saltstack, Ansible to name a few. A server provisioning tool, as the first step that runs in server-up datacenter automation, should be able to work reliably with all of these. Whether it enables users to hook up to them, or actually deploys the agents itself, the end result should be that a deployed server should be able to use the application provisioning that is currently in use at a given organization, and be able to change if that provisioning tool is replaced. Deeply integrated tools will work well with a single application provisioning tool, but tend toward vendor lock-in, because that tool is then harder to replace without replacing how servers are provisioned (which is no small chore in and of itself).

So when speccing out a datacenter automation or DevOps automation project, look for a server provisioning tool that can hand off to the application provisioning tool of choice, has some subset of application provisioning for complex systems (unless your application provisioning tool is already taking care of that of course), and can be fully automated. Partial automation is difficult at best, but it is the state of this growing market that some functionality is still not completely implemented in some products. It is easy enough to poke at products and determine if they suit the needs of your level of automation. A good example is that only about half of the products on the market make any attempt at RAID configuration as part of server provisioning. But if operators have to sit and configure RAID before an automated installation, then it’s not really fully automated, is it?

And of course I recommend that you consider automation. Partially because my employer plays in the space, but moreso because there are far more impactful things operations could be doing than sitting and watching an OS install, or typing in IP addresses, masks, and gateways.

More Stories By Don MacVittie

Don MacVittie is founder of Ingrained Technology, A technical advocacy and software development consultancy. He has experience in application development, architecture, infrastructure, technical writing,DevOps, and IT management. MacVittie holds a B.S. in Computer Science from Northern Michigan University, and an M.S. in Computer Science from Nova Southeastern University.

IoT & Smart Cities Stories
Early Bird Registration Discount Expires on August 31, 2018 Conference Registration Link ▸ HERE. Pick from all 200 sessions in all 10 tracks, plus 22 Keynotes & General Sessions! Lunch is served two days. EXPIRES AUGUST 31, 2018. Ticket prices: ($1,295-Aug 31) ($1,495-Oct 31) ($1,995-Nov 12) ($2,500-Walk-in)
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
Nicolas Fierro is CEO of MIMIR Blockchain Solutions. He is a programmer, technologist, and operations dev who has worked with Ethereum and blockchain since 2014. His knowledge in blockchain dates to when he performed dev ops services to the Ethereum Foundation as one the privileged few developers to work with the original core team in Switzerland.
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
All in Mobile is a place where we continually maximize their impact by fostering understanding, empathy, insights, creativity and joy. They believe that a truly useful and desirable mobile app doesn't need the brightest idea or the most advanced technology. A great product begins with understanding people. It's easy to think that customers will love your app, but can you justify it? They make sure your final app is something that users truly want and need. The only way to do this is by ...
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed...