Welcome!

Java IoT Authors: Pat Romanski, Yeshim Deniz, Liz McMillan, Zakia Bouachraoui, Elizabeth White

Related Topics: @DXWorldExpo, Java IoT, Microservices Expo, Linux Containers, SDN Journal

@DXWorldExpo: Article

Big Data – How We Got Here and Is It For Everyone? | @BigDataExpo

An organization needs to be data driven in order to understand the data’s value and lament its absence

Big Data is an umbrella term for a multitude of new capabilities that are being used for storage and computing operations at scale. These capabilities allow organizations to store massive amounts of data, in disparate formats, and perform both batch and real-time analyses upon them.

The forces driving Big Data into the mainstream are the ever-decreasing cost of storage and processing, coupled with the open source enhancements of distributed systems techniques and software. Companies have realized that data storage is on the verge of being limitless, and they no longer need to be as judicious about what kinds of data they store. This realization has led to the storage of all manner of data, in addition to the traditional structured data found in relational databases.

Sometimes unstructured or semi-structured, this type of data encompasses emails, social media feeds, clickstreams, sensor data, videos and more. Further, the questions companies can ask of their data to realize value have become more complex. But the time window for analysis completion has remained the same or shrunk due to the massively parallel computation Big Data systems provide.

To organize all this limitless data - structured and unstructured - new tools have emerged. We no longer have just one hammer in our toolkit - the relational database - with which to fashion data. There are now a myriad of systems, thanks to big organizations, many of them Internet giants (Google, Yahoo!, Amazon, Facebook, and LinkedIn to name a few). Out of a need to scale storage and compute tasks confronting them, they made new data systems to satisfy these specific use cases. The most common characteristics of these systems are that they are not row based or relational databases. The systems have horizontal linear scalability (just add more nodes), and they expect components and nodes to fail so they are fault tolerant.

Most of these tools have been open sourced and, years later, are being adopted by other businesses that recognize their value. Examples of these non-relational databases are key-value pair databases (e.g., Riak), document databases (e.g., CouchDB, MongoDB), columnar databases (e.g., HBase, Cassandra), graph databases (e.g., Titan, Neo4J), distributed queues (e.g., Kafka, Kestrel) and spatial databases. There are also new computing tools such as Hadoop and Spark that allow immense amounts of data to be processed, and Storm, Samza and Spark Streaming which analyze data in near real-time, something that was previously only possible with supercomputers.

The ability to store unlimited amounts of disparate data in order to perform endless analysis in batch and real time is the allure of Big Data. But is it for everyone?

Are you ready for the Big Data bandwagon?
The promise of Big Data is very real - more data + new ways to analyze data = better business intelligence. So every business should be rushing out to see what Big Data can do for them, right? Not so fast.

Although Big Data has the capability to provide unprecedented insight into business operations, it is not for everyone. At least not yet. Big Data tools are still immature enough, even eight years after we've started down this road, that companies really have to be highly motivated to take on the task.

In my experience, this motivation typically comes from pain. For example, it is the pain of dropping valuable data on the floor because long-term storage isn't feasible within the current infrastructure. Or possibly it's the pain of not being able to monetize collected data due to technical hurdles with existing data systems. This type of pain - and the knowledge that the only way to escape it is to embark on Big Data - provides the fortitude necessary to push through on what can be quite a challenging project. In fact, it was this kind of pain, as experienced by Internet giants who needed to find a way to store, analyze and, ultimately, monetize the massive amounts of customer data to which they had access, that birthed the Big Data movement in the first place.

In addition to being highly motivated to take on Big Data, an organization must also be data driven. Organizations that treat data as currency are better equipped to tackle the challenges of Big Data, because they inherently understand its value and will nurture the initiative, no matter how unruly it becomes. Without this level of buy-in from top-level management, the obstacles of Big Data may prove to be too large to overcome.

The combination of the pain of unexploited data with a data-driven culture goes hand in hand. You typically won't find one without the other. It is when these two factors come together that organizations can then successfully tackle the challenge of Big Data. Without the right motivation or executive support, Big Data endeavors often die before they even begin. Ultimately, an organization needs to be data driven in order to understand the data's value and lament its absence.

More Stories By Brad Anderson

As vice president of Big Data informatics, Brad Anderson is responsible for Liaison's Big Data solution implementation, leveraging the company's world-class cloud infrastructure. He is a 20-year data management veteran with expertise in enterprise data warehouses and building and using non-relational Big Data tools.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


IoT & Smart Cities Stories
Early Bird Registration Discount Expires on August 31, 2018 Conference Registration Link ▸ HERE. Pick from all 200 sessions in all 10 tracks, plus 22 Keynotes & General Sessions! Lunch is served two days. EXPIRES AUGUST 31, 2018. Ticket prices: ($1,295-Aug 31) ($1,495-Oct 31) ($1,995-Nov 12) ($2,500-Walk-in)
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
Nicolas Fierro is CEO of MIMIR Blockchain Solutions. He is a programmer, technologist, and operations dev who has worked with Ethereum and blockchain since 2014. His knowledge in blockchain dates to when he performed dev ops services to the Ethereum Foundation as one the privileged few developers to work with the original core team in Switzerland.
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
All in Mobile is a place where we continually maximize their impact by fostering understanding, empathy, insights, creativity and joy. They believe that a truly useful and desirable mobile app doesn't need the brightest idea or the most advanced technology. A great product begins with understanding people. It's easy to think that customers will love your app, but can you justify it? They make sure your final app is something that users truly want and need. The only way to do this is by ...
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed...