Study Guides/AWS Cybersecurity Notes/Logging and Monitoring

Logging and Monitoring

The use of Cloud Watch is a important consideration - This can capture logs from AWS services and metrics to enable automation.

Logging is of vital importance in diagnosis of security and performance issues in AWS.

S3 Logging

It is possible to capture all logs in an S3 bucket, however this comes with a condition:

The access logs for the logging S3 bucket cannot be stored inside itself.

You will therefore need a seperate bucket for storing your S3 logging access logs.

Access logging for the S3 logging bucket is highly recommended - an attacker may target logging to inject bad data or delete records.

Best practice is to create a access policy to ensure only the security team can read data from these buckets.

[Main Logging bucket]
          |
         \|/
writes access logs to
          |
         \|/
  [S3 logging bucket]

Delays in S3 logging

Data is pushed to the S3 logging bucket on a "best effort" process.

It can take a few hours to deliver s3 logs.

S3 Logs will not be in realtime.

AWS has a massive shared disk system - this is the reason behind the delay.

• Personally I have seen delays over 24 hours!!

Enabling S3 access logs

This assumes you already have a bucket you want to log.

• Create a bucket to log data to.

Crearte a JSON file to define where to place logs:

{
  "LoggingEnabled": {
      "TargetBucket": "logging bucket",
      "TargetPrefix": "S3Logs/"
  }
}

Push the json file:

aws s3api put-bucket-logging --bucket s3_bucket_to_log --bucket-logging-status file://s3logs.json

Access log naming

The access logs are text files.

Names start with a timestamp and a semi random value - You cannot change this.

You will end up with lots of small text files.

S3 Object logging

S3 Object logging intergrates with Clould Trail events.

CouldTrail records and tracks all API calls.

If object-level logging is turned on you must associated it with could trail.

S3 object logging is dependent on cloud trail.

VPC Flow Logs (Traffic Monitoring)

Flow logs allow the capture of IP traffic over network interfaces.

This could be many interfaces and. lots of data!!

Flow logs can be activated by:

The VPC

A subnet of the VPC

A network interface:

EC2 / ELB / Amazon RDS / ElastiCache / Redshift / Workspaces / Nat Gateway / Transit Gateway

Then can be logged to the S3 Bucket or CloudWatch Logs

Selecting CloudWatch or S3

S3 is good for long term log storage:

You may neeed to keep logs for years

This is where S3 is useful

S3 is low cost

S3 offters 11'9s of reliability.

Why CloudWatch?

Easier - One click setup.

Less mess - Groups the logs by day week and year.

Native search.

Cloudwatch Problems:

Storage is more expensive than S3

Limits to cloudwatch queries

Traffic Type

It's important understand the ---traffic-type flag

• All - Capture both accepted and rejected.

• Reject - Traffic that was rejected from the VPC (ie Saves!).

• Accept - Traffic that accepted and passed.

Think about these options in exam questions!!

Information in the VPC flow logs

This is a text file, space separated. In order of appearance:

• The version of the VPC flow log format.

• AWS account ID.

• ID of the ENI that captured the traffic

• Source IP.

• Destination IP.

• IP protocol number (6 for TCP).

• Number of packets.

• Number of bytes.

• Start time (unix).

• End time (unix).

• Action taken (Accept / Reject)

• Reason for the action.

Flow log limitations

• The VPC must be in your account - Peering dosen't count.

• The configuration or format cannot be changed once enabled.

• Delivery to S3 happens every 5 minutes.

This not a complete list!

Traffic Mirroring

Allows the duplication of traffic from an ENI.

Traffic can be sent to third party tools for inspection.

The destination could be anything - Load balancer / fleet of innstances / appliances / Another EC2

Elastic Load Balancer Access Logs

In the ELB service logging is optional, you will have to turn it on!

They are delivered again to an S3 bucket.

The files are compressed and will need decompression.

Uses of ELB logs

• Troubleshooting

• Performance & Request latency analysis.

• Security - Detection of DDOS, Bots and bad actors.

• Compliance and auditing.

• Capacitiy planning.

Load Balancer log files

All requests are captures - even if they never hit the desintation.

This includes malformed requests.

Malformed requests can be security related - directory traversal attacks

What the logs contain

• Type of request

• Time of the request

• The ELB that took the request

• Cilent port / Target port

• Processing time - how long the target took to process the request.

• Status code for the ELB

• Target response code