Study Guides/AWS Cybersecurity Notes/Logging and Monitoring

From Cramsession
Jump to navigationJump to search
✍️ Verified Author: Mflavell • Click to view professional profile & credentials

Logging and Monitoring

The use of Cloud Watch is a important consideration - This can capture logs from AWS services and metrics to enable automation.


Logging is of vital importance in diagnosis of security and performance issues in AWS.


S3 Logging

It is possible to capture all logs in an S3 bucket, however this comes with a condition:

The access logs for the logging S3 bucket cannot be stored inside itself.
You will therefore need a seperate bucket for storing your S3 logging access logs.
Access logging for the S3 logging bucket is highly recommended - an attacker may target logging to inject bad data or delete records.
Best practice is to create a access policy to ensure only the security team can read data from these buckets.


[Main Logging bucket]
          |
         \|/
writes access logs to
          |
         \|/
  [S3 logging bucket]


Delays in S3 logging

Data is pushed to the S3 logging bucket on a "best effort" process.

It can take a few hours to deliver s3 logs.
S3 Logs will not be in realtime.
AWS has a massive shared disk system - this is the reason behind the delay.
  • Personally I have seen delays over 24 hours!!

Enabling S3 access logs

This assumes you already have a bucket you want to log.

  • Create a bucket to log data to.

Crearte a JSON file to define where to place logs: 

{
  "LoggingEnabled": {
      "TargetBucket": "logging bucket",
      "TargetPrefix": "S3Logs/"
  }
}

Push the json file: 

aws s3api put-bucket-logging --bucket s3_bucket_to_log --bucket-logging-status file://s3logs.json


Access log naming

The access logs are text files.

Names start with a timestamp and a semi random value - You cannot change this.

You will end up with lots of small text files.


S3 Object logging

S3 Object logging intergrates with Clould Trail events.
CouldTrail records and tracks all API calls.
If object-level logging is turned on you must associated it with could trail.
S3 object logging is dependent on cloud trail.


VPC Flow Logs (Traffic Monitoring)

Flow logs allow the capture of IP traffic over network interfaces.

This could be many interfaces and. lots of data!!


Flow logs can be activated by:

The VPC
A subnet of the VPC
A network interface:
EC2 / ELB / Amazon RDS / ElastiCache / Redshift / Workspaces / Nat Gateway / Transit Gateway


Then can be logged to the S3 Bucket or CloudWatch Logs


Selecting CloudWatch or S3

S3 is good for long term log storage:

You may neeed to keep logs for years
This is where S3 is useful
S3 is low cost
S3 offters 11'9s of reliability.


Why CloudWatch?

Easier - One click setup.
Less mess - Groups the logs by day week and year.
Native search.


Cloudwatch Problems:

Storage is more expensive than S3
Limits to cloudwatch queries


Traffic Type

It's important understand the ---traffic-type flag

  • All - Capture both accepted and rejected.
  • Reject - Traffic that was rejected from the VPC (ie Saves!).
  • Accept - Traffic that accepted and passed.

Think about these options in exam questions!!


Information in the VPC flow logs

This is a text file, space separated. In order of appearance:

  • The version of the VPC flow log format.
  • AWS account ID.
  • ID of the ENI that captured the traffic
  • Source IP.
  • Destination IP.
  • IP protocol number (6 for TCP).
  • Number of packets.
  • Number of bytes.
  • Start time (unix).
  • End time (unix).
  • Action taken (Accept / Reject)
  • Reason for the action.


Flow log limitations

  • The VPC must be in your account - Peering dosen't count.
  • The configuration or format cannot be changed once enabled.
  • Delivery to S3 happens every 5 minutes.

This not a complete list!


Traffic Mirroring

Allows the duplication of traffic from an ENI.

Traffic can be sent to third party tools for inspection.
The destination could be anything - Load balancer / fleet of innstances / appliances / Another EC2


Elastic Load Balancer Access Logs

In the ELB service logging is optional, you will have to turn it on!

They are delivered again to an S3 bucket.

The files are compressed and will need decompression.


Uses of ELB logs

  • Troubleshooting
  • Performance & Request latency analysis.
  • Security - Detection of DDOS, Bots and bad actors.
  • Compliance and auditing.
  • Capacitiy planning.


Load Balancer log files

All requests are captures - even if they never hit the desintation.

This includes malformed requests.
Malformed requests can be security related - directory traversal attacks


What the logs contain

  • Type of request
  • Time of the request
  • The ELB that took the request
  • Cilent port / Target port
  • Processing time - how long the target took to process the request.
  • Status code for the ELB
  • Target response code


Web Application Firewall logs

Cloudwatch is usefull here:

  • Will provide real time metrics about the AWS WAF and rule matching.
  • Metrics are published for every rule.
  • This includes AWS managed rules and rules you define.
  • Metrics will still be captured when the WAF is is blocked / count mode


WAF Full logs

This provides details about the type of request that came in - more than just metrics.

Benfits of full logs:

  • JSON format
  • Sensative infomation can be removed.
  • They can be streamed - Amazon Kinesis Data Firehoze
  • All request headers are recorded along with the ID of the rule that matching the request.


These are used for comprehsive infomration.

Siplified requests don't paint a full picture:

  • They are limited to 500 logfiles and 5000 requests
  • This is not enough imofmration to diagnose a busy firewall
  • Security often means examining all traffic to gain full understanding.


Cloudwatch Logs

Most AWS services can send thier logs to CloudWatch Logs

It's important to remember what services cannot send to CloudWatch, these are:

  • CloudFrount
  • AWS Global Accelerator
  • Route 53 query resolver
  • VPC Flow logs


What do these have in common: Network related / Accellerator related.


CloudFront is limited to S3 logging only.


Required IAM permissions

Publising logs to CloudWatch

The user or service must have these IAM permissions:

  • logs:CreateLogDelivery
  • logs:CreateResourcePolicy
  • logs:DescribeResourcePolicies
  • logs:DescribeLogGroups


If logs are been sent to a log group the logs:CreateLogDelivery is important.


Publising logs to S3

For sending to S3 the following IAM permissions are required:

  • logs:CreateLogDelivery
  • s3:GetBucketPolicy
  • s3:PutBucketPolicy


Publising logs to Kinesis Data Firehose

When sending data to the Kinesis Data Firehose the following permissions must be enabled:

  • logs:CreateLog Delivery
  • firehose:TagDeliverySystem
  • iam:CreateServiceLinkedRole


Logging API Activity with CloudTrail

This allows for compliance, governance and auditing though the logging of API calls.

Logs calls made via:

  • AWS Management console
  • AWS CLI
  • AWS SDK's
  • Third party tools using the AWS CLI


Can be used in multiple regions if needed.

When AWS Organizations is used - logs from all accounts flow up into the audit account.


Components of cloudtrail

  • Trails - Contain the options you want monitor to track.
  • Events - When an API call is made this is an event in CoudTrail.
  • Logs files - Logs are saved every 5 minutes to a S3 bucket.
  • Activity filters - These provide a search function when checking API history.


Types of CloudTrail Events

  • Management events - Capture management operations.
  • Data events - Events reproted as part of a specfic trail.
There are costs associated with capturing data events.
Best to develop a statergy concerning the about of data and retention period.
  • Insight events - these are not logged by default but can be turned on.
When on these capture rate alarms & error rates.
These help determine what is normal and abnormal on the account.


Default CloudTrail Settings

  • Enabled for the AWS account when you create it.
  • Logs are encrytptred with SSE (Server side encryption) / Can be changed to KMS key encrypted.
  • Logs are published about every 5 minutes.
  • Tails can only be viewed in the AWS regions where events are logged.


Creating new trails

by Default could trail is enabled in accounts.

It's important to ensure the integrity of all cloudtail logs.
Creating an additional trail is a mitigation tactic:
If something happens to the data in the original trail - you have another.


This could be useful if both security and developers need access to could trail data.
Two seperate tails can be created that are isolated.


Data events in s3 buckets

  • These are much like S3 access logs.
They are stored in CloudTrail instead of S3


This setup is useful when using multiple accounts:

  • All cloud trails flow to a seperate logging account.
  • This can use a central account to track all s3 activity.
  • Prevents somone accessing a bucket turing off or deleteing logs.


Event history in CloudTrail

  • You can query 90 days maximum.
Beyond this you will need to use a third party tool or extact the compessed logs to S3.


AWS Cloud Trail Lake

This enables large scale ananyls of AWS cloud trail logs.

This uses three tools:

S3
Athena
AWS Glue

Together they create a data lake that can store logs for an extented period.


Important point for the exam: Cloud trail limits to 90days - you can use Cloud Trail Lake to go beyond this.


Why use Cloud Trail Lake:

  • Deeper insights - more data.
  • Auditing and compliance.
  • Provides a central log storage location.
  • Easier to perform ad-hoc queries.
  • Easier to create custom repots.
  • Provides the ability to query multiple accounts.


What format does it use:

Uses an Apache based ORC format - not JSON!
These re row-based - columnar format is effiecent (like SQL!)
Retrieved from "https://cramsession.net/index.php?title=Study_Guides/AWS_Cybersecurity_Notes/Logging_and_Monitoring&oldid=1670"