Welcome!

Java IoT Authors: Yeshim Deniz, Pat Romanski, Ian Khan, Karyn Jeffery, Anders Wallgren

Related Topics: @CloudExpo, Java IoT, Linux Containers, @BigDataExpo

@CloudExpo: Article

Monitoring and Analyzing AWS CloudTrail Data

Monitoring and Analyzing AWS CloudTrail data from multiple AWS regions

We recently released AWS CloudTrail integration with Logentries - and not surprisingly we've seen a significant uptick in adoption as one of our most popular integrations. My job as director of customer success is to make things as simple for our customers as possible. One question that consistently pops up, is how to collect AWS CloudTrail logs from multiple AWS regions.

We follow Amazon's best practices when it comes to integrating with, and receiving information from, CloudTrail. In short, this works as follows:

  • When configuring CloudTrail, it will write events to a S3 bucket.
  • You can configure Cloudtrail to send notifications to an Amazon SNS topic whenever new log events are recorded.
  • You can get updates sent to an Amazon Simple Queue Service (Amazon SQS) queue, which enables you to handle these notifications programmatically.
  • To configure Logentries to consume your Cloudtrail logs, simply add the URL of the SQS queue to the Logentries/Cloudtrail setup page.

Logentries speaks directly to the SQS queue inside of your AWS account, so an obvious question that presents itself is: If I'm running in multiple AWS regions, how do I get Logentries to pull from all of the regions?

The simple answer: you don't. Make AWS do the work for you!

Following the steps outlined below, you'll be able to monitor and analyze CloudTrail logs from any number of AWS regions all within one Logentries account.

Create an S3 Bucket
If you're new to the CloudTrail setup, the first requirement of CloudTrail logging is that the logs must go "somewhere." In AWS, this somewhere happens to be a S3 bucket which you should create. Simply navigate to the S3 service and select ‘Create Bucket'. By default, all permissions required are given to the bucket - i.e. there is no extra permissions/configuration necessary to configure CloudTrail logging with Logentries.
Screen Shot 2014-08-19 at 12.55.03 PM

Create an SQS Queue in a Primary Region
Next up, we need to create an SQS Queue to allow Logentries to consume your CloudTrail data. Create a new Queue and provide a ‘Queue Name' - default options are fine.
SQS_Management_Console_and_LogentriesCloudtrail_docx

Add permissions to the SQS Queue
Once the queue has been created, the correct permissions must be applied. When adding permissions to the SQS queue, you need to add your full account number/name (officially called the AWS User ARN).

To get the User ARN navigate to the IAM Service, select the user that you want to utilize and click ‘Summary'. When the user is created within the IAM section, make sure that the user has at least ‘Read-Only' access - so that the user has the relevant permissions to read the bucket. The string you need is available under User ARN in the ‘Summary' section and follows this format:

arn:aws:iam::<account_code>:user/

Next add ‘Receive', ‘Send', and ‘Delete' Actions to the SQS Queue(see below):
SQS_Management_Console
Enable CloudTrail in any region, and publish to an SNS topic

Once the above three steps have been complete, it's time to enable CloudTrail in the relevant regions. Navigate to the CloudTrail Service in your AWS Console and turn on CloudTrail. Do not create a new S3 bucket, but instead select the S3 Bucket created in step one above from the drop down menu. Once you've done this, click the Advanced link. For the first region you enable CloudTrail for, remember to include Global Services under Advanced options - this record API calls from any global AWS services such as IAM or AWS STS. Make sure that "SNS Notification for every log file delivery" is checked, and finally, specify a SNS Topic to publish to. A new SNS topic name should be given - and will be created by CloudTrail.

Follow these above steps for each region that you want to collect CloudTrail logs from. NOTE: when adding subsequent regions you will want to exclude Global Services to avoid duplicate log events being recorded for your Global Services.

Screen Shot 2014-08-19 at 2.48.02 PM
Subscribe the SQS Queue to the multiple SNS topics
Once each region has been setup, the last step in AWS is to subscribe your SQS Queue to each newly created SNS topic. Navigate to the SQS Service in your AWS Console and highlight the queue created in step 2 above. Under the ‘Queue Actions' menu at the top select ‘Subscribe Queue to SNS Topic'. Use the ‘Topic Region' drop down to select the region and the ‘Choose a Topic' drop down to select the topic created in the previous step. Hit the ‘Subscribe' button and wait for the confirmation that the queue has subscribed to that topic.

After selecting subscribe make sure to copy the SQS URL from the ‘Details' section on the page.
SQS_Management_Console1

Setup Logentries to Pull data from the SQS Queue
Login to your Logentries account and navigate to your AWS settings area (My Account -> AWS). Select enable CloudTrail, supply your IAM access key, secret key, and SQS URL that you have copied above. Hit Save! Note: your IAM access key and secret key are made available to your when you create a new IAM user and should be stored safely.

Log Data from CloudTrail will begin to stream in within approximately 15 minutes.

Account___Logentries
Sit back and let Logentries do it's magic!
Visit our CloudTrail documentation to see some of the other cool things you can do - in particular we provide out of the box tags and alerts for important CloudTrail events. Have questions or ideas how we can make our CloudTrail integration better? Reach out to me directly at [email protected].

 

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

@ThingsExpo Stories
Complete Internet of Things (IoT) embedded device security is not just about the device but involves the entire product’s identity, data and control integrity, and services traversing the cloud. A device can no longer be looked at as an island; it is a part of a system. In fact, given the cross-domain interactions enabled by IoT it could be a part of many systems. Also, depending on where the device is deployed, for example, in the office building versus a factory floor or oil field, security ha...
24Notion is full-service global creative digital marketing, technology and lifestyle agency that combines strategic ideas with customized tactical execution. With a broad understand of the art of traditional marketing, new media, communications and social influence, 24Notion uniquely understands how to connect your brand strategy with the right consumer. 24Notion ranked #12 on Corporate Social Responsibility - Book of List.
Fact is, enterprises have significant legacy voice infrastructure that’s costly to replace with pure IP solutions. How can we bring this analog infrastructure into our shiny new cloud applications? There are proven methods to bind both legacy voice applications and traditional PSTN audio into cloud-based applications and services at a carrier scale. Some of the most successful implementations leverage WebRTC, WebSockets, SIP and other open source technologies. In his session at @ThingsExpo, Da...
Businesses are struggling to manage the information flow and interactions between all of these new devices and things jumping on their network, and the apps and IT systems they control. The data businesses gather is only helpful if they can do something with it. In his session at @ThingsExpo, Chris Witeck, Principal Technology Strategist at Citrix, will discuss how different the impact of IoT will be for large businesses, expanding how IoT will allow large organizations to make their legacy ap...
What happens when the different parts of a vehicle become smarter than the vehicle itself? As we move toward the era of smart everything, hundreds of entities in a vehicle that communicate with each other, the vehicle and external systems create a need for identity orchestration so that all entities work as a conglomerate. Much like an orchestra without a conductor, without the ability to secure, control, and connect the link between a vehicle’s head unit, devices, and systems and to manage the ...
What does it look like when you have access to cloud infrastructure and platform under the same roof? Let’s talk about the different layers of Technology as a Service: who cares, what runs where, and how does it all fit together. In his session at 18th Cloud Expo, Phil Jackson, Lead Technology Evangelist at SoftLayer, an IBM company, spoke about the picture being painted by IBM Cloud and how the tools being crafted can help fill the gaps in your IT infrastructure.
For basic one-to-one voice or video calling solutions, WebRTC has proven to be a very powerful technology. Although WebRTC’s core functionality is to provide secure, real-time p2p media streaming, leveraging native platform features and server-side components brings up new communication capabilities for web and native mobile applications, allowing for advanced multi-user use cases such as video broadcasting, conferencing, and media recording.
In this strange new world where more and more power is drawn from business technology, companies are effectively straddling two paths on the road to innovation and transformation into digital enterprises. The first path is the heritage trail – with “legacy” technology forming the background. Here, extant technologies are transformed by core IT teams to provide more API-driven approaches. Legacy systems can restrict companies that are transitioning into digital enterprises. To truly become a lea...
SYS-CON Events announced today that Sheng Liang to Keynote at SYS-CON's 19th Cloud Expo, which will take place on November 1-3, 2016 at the Santa Clara Convention Center in Santa Clara, California.
In his session at @ThingsExpo, Kausik Sridharabalan, founder and CTO of Pulzze Systems, Inc., will focus on key challenges in building an Internet of Things solution infrastructure. He will shed light on efficient ways of defining interactions within IoT solutions, leading to cost and time reduction. He will also introduce ways to handle data and how one can develop IoT solutions that are lean, flexible and configurable, thus making IoT infrastructure agile and scalable.
Just over a week ago I received a long and loud sustained applause for a presentation I delivered at this year’s Cloud Expo in Santa Clara. I was extremely pleased with the turnout and had some very good conversations with many of the attendees. Over the next few days I had many more meaningful conversations and was not only happy with the results but also learned a few new things. Here is everything I learned in those three days distilled into three short points.
Cognitive Computing is becoming the foundation for a new generation of solutions that have the potential to transform business. Unlike traditional approaches to building solutions, a cognitive computing approach allows the data to help determine the way applications are designed. This contrasts with conventional software development that begins with defining logic based on the current way a business operates. In her session at 18th Cloud Expo, Judith S. Hurwitz, President and CEO of Hurwitz & ...
So, you bought into the current machine learning craze and went on to collect millions/billions of records from this promising new data source. Now, what do you do with them? Too often, the abundance of data quickly turns into an abundance of problems. How do you extract that "magic essence" from your data without falling into the common pitfalls? In her session at @ThingsExpo, Natalia Ponomareva, Software Engineer at Google, provided tips on how to be successful in large scale machine learning...
The Transparent Cloud-computing Consortium (abbreviation: T-Cloud Consortium) will conduct research activities into changes in the computing model as a result of collaboration between "device" and "cloud" and the creation of new value and markets through organic data processing High speed and high quality networks, and dramatic improvements in computer processing capabilities, have greatly changed the nature of applications and made the storing and processing of data on the network commonplace.
An IoT product’s log files speak volumes about what’s happening with your products in the field, pinpointing current and potential issues, and enabling you to predict failures and save millions of dollars in inventory. But until recently, no one knew how to listen. In his session at @ThingsExpo, Dan Gettens, Chief Research Officer at OnProcess, will discuss recent research by Massachusetts Institute of Technology and OnProcess Technology, where MIT created a new, breakthrough analytics model f...
The Internet of Things can drive efficiency for airlines and airports. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Sudip Majumder, senior director of development at Oracle, will discuss the technical details of the connected airline baggage and related social media solutions. These IoT applications will enhance travelers' journey experience and drive efficiency for the airlines and the airports. The session will include a working demo and a technical d...
Almost two-thirds of companies either have or soon will have IoT as the backbone of their business in 2016. However, IoT is far more complex than most firms expected. How can you not get trapped in the pitfalls? In his session at @ThingsExpo, Tony Shan, a renowned visionary and thought leader, will introduce a holistic method of IoTification, which is the process of IoTifying the existing technology and business models to adopt and leverage IoT. He will drill down to the components in this fra...
Digital transformation is too big and important for our future success to not understand the rules that apply to it. The first three rules for winning in this age of hyper-digital transformation are: Advantages in speed, analytics and operational tempos must be captured by implementing an optimized information logistics system (OILS) Real-time operational tempos (IT, people and business processes) must be achieved Businesses that can "analyze data and act and with speed" will dominate those t...
If you had a chance to enter on the ground level of the largest e-commerce market in the world – would you? China is the world’s most populated country with the second largest economy and the world’s fastest growing market. It is estimated that by 2018 the Chinese market will be reaching over $30 billion in gaming revenue alone. Admittedly for a foreign company, doing business in China can be challenging. Often changing laws, administrative regulations and the often inscrutable Chinese Interne...
I'm a lonely sensor. I spend all day telling the world how I'm feeling, but none of the other sensors seem to care. I want to be connected. I want to build relationships with other sensors to be more useful for my human. I want my human to understand that when my friends next door are too hot for a while, I'll soon be flaming. And when all my friends go outside without me, I may be left behind. Don't just log my data; use the relationship graph. In his session at @ThingsExpo, Ryan Boyd, Engi...