Study Guides/AWS Cybersecurity Notes/Parsing logs and events

Important points to consider

• Build a resillant storage solution.

• Store log files centrally.

• Ensure integirity is maintained.

• Know retention policies.

• Know the process for adding new logs.

• Who to grant read access to.

• Monitor storage.

Storing logs in S3

The obvious solution.

Provdies resiliant long term storage for data.

The problem with S3 is the ability to search files - it's just a file system not an RDB.

S3 has several storage tiers

S3 Standard

Highly available but most expensive.

Ideal for requent access to logs.

99.99% avalability and 99.999999999% ( 5 9's ) diability.

Can encrypt data at rest or in transit (SSL).

Standard Infrequent access

Designed for files >128kb

If a file is less than 128 kb your charged for 128 kb - not a big deal unless you have lots of small files.

Designed for extended storage peroids >30 days - Minimum charge is 30 days.

Storage is lower cost but there are higher charges for actions (GET / PUT / COPY / POST / LIST / SELECT).

No delays to access objects - realtime access.

basically the same as S3 Standard but lower storage costs / ideal for infrequent access.

S3 One Zone IA

Lowest cost point.

Same as Standard IA but only in one zone.

Ideal if you are already replicating the data.

Risk of data loss - all your data is on One Zone.

Resillancey and avlability is the same.

S3 Glacier Instant

Raipid access to data.

68% cost savings over S3 standard

Same 128 kb minimum charge.

One AZ can be destroyed without data loss.

Same Resillanacy and avalability.

S3 Glaicer Flexable

Minimum charge of 90 days

Several retrival speeds from 5 minutes to 12 hours

Data access isn't instant.

Objects can be locked.

Great for low cost storage for compliance - I have it, I hope I don't need it.

S3 Glacier Deep Archive

Ideal if you don't expect to access the data again.

Can take 12 hours to get to your data.

Deep savings, About $1 per TB per month!

An alternative to using tape.

Using S3 lifecycle with logs

This approach helps reduce the stoage costs for logs automatically.

Lifecycles can apply to all objects in a bucket or items with certain prefixes.

Tags can also be used to assgin a file to a lifecycle.

Idealy, use a bucket just for logs - then you can easily manage the lifecycle.

Lifecycle rules are crated in the bucket.

Storing logs in S3 vs Cloudwatch

• The most cost effective may not be the least expensive.

• Consider the requirements and the effectiveness of the solution.

• Think S3 is just text files! - S3 Glacier is slow text files - If instant searching is a requirement these won't cut it.

Cloudwatch log subscription filters

These allow you to stream log events from a cloudwatch log group.

Can stream to:

Lambda Function

Data Firehose

Fire

SNS Topic

Can be used only to send logs that contain a spefific message - example error.

A subscription filter can match a pattern to keywords.