Study Guides/AWS Cybersecurity Notes/Parsing logs and events

Important points to consider

• Build a resillant storage solution.

• Store log files centrally.

• Ensure integirity is maintained.

• Know retention policies.

• Know the process for adding new logs.

• Who to grant read access to.

• Monitor storage.

Storing logs in S3

The obvious solution.

Provdies resiliant long term storage for data.

The problem with S3 is the ability to search files - it's just a file system not an RDB.

S3 has several storage tiers

S3 Standard

Highly available but most expensive.

Ideal for requent access to logs.

99.99% avalability and 99.999999999% ( 5 9's ) diability.

Can encrypt data at rest or in transit (SSL).

Standard Infrequent access

Designed for files >128kb

If a file is less than 128 kb your charged for 128 kb - not a big deal unless you have lots of small files.

Designed for extended storage peroids >30 days - Minimum charge is 30 days.

Storage is lower cost but there are higher charges for actions (GET / PUT / COPY / POST / LIST / SELECT).

No delays to access objects - realtime access.

basically the same as S3 Standard but lower storage costs / ideal for infrequent access.

S3 One Zone IA

Lowest cost point.

Same as Standard IA but only in one zone.

Ideal if you are already replicating the data.

Risk of data loss - all your data is on One Zone.

Resillancey and avlability is the same.

S3 Glacier Instant

Raipid access to data.

68% cost savings over S3 standard

Same 128 kb minimum charge.

One AZ can be destroyed without data loss.

Same Resillanacy and avalability.

S3 Glaicer Flexable

Minimum charge of 90 days

Several retrival speeds from 5 minutes to 12 hours

Data access isn't instant.

Objects can be locked.

Great for low cost storage for compliance - I have it, I hope I don't need it.

S3 Glacier Deep Archive

Ideal if you don't expect to access the data again.

Can take 12 hours to get to your data.

Deep savings, About $1 per TB per month!

An alternative to using tape.

Using S3 lifecycle with logs

This approach helps reduce the stoage costs for logs automatically.

Lifecycles can apply to all objects in a bucket or items with certain prefixes.

Tags can also be used to assgin a file to a lifecycle.

Idealy, use a bucket just for logs - then you can easily manage the lifecycle.

Lifecycle rules are crated in the bucket.

Storing logs in S3 vs Cloudwatch

• The most cost effective may not be the least expensive.

• Consider the requirements and the effectiveness of the solution.

• Think S3 is just text files! - S3 Glacier is slow text files - If instant searching is a requirement these won't cut it.

Cloudwatch log subscription filters

These allow you to stream log events from a cloudwatch log group.

Can stream to:

Lambda Function

Data Firehose

Fire

SNS Topic

Can be used only to send logs that contain a spefific message - example error.

A subscription filter can match a pattern to keywords.

Using Amazon Kinesis with Logs

This is a fully managed service that can scale automatically.

The system scales up and down automatically.

Data can be encrypted.

Kinesis facts to remember:

• It is serverless.

• Can automatically scale.

• It can transform data without other services.

• Can deliver data to:

• Opensearch

• Redshift

• Splunk

• S3

The data transofrmation feature can be used to change the format of data, for example CSV to JSON.

Transformation is useful for sending to Opensearch.

Kenisis data firehose can be used to move logs between CloudWatch and S3.

Amazon Athena

Athena allows ad-hoc queries on data sets.

It is serverless and can analize data stored in S3 - A big plus for logging.

Queries can be written in SQL

Athena:

• Uses S3 storage - no need for compute storage (used for SQL database).

• Serverles

• Pay per scan - cost savings.

• Supports a few log formats:

Apache Web Logs

CSV / TSV files

JSON Files

Parquet

ORC

Transforming data

Amazon Glue Data Catalog can transform data for Athena

Storing and searching in Amazon Opensearch

Opensearch is commonly used on log protection and auditing.

Opensearch is probsably an exam answer!!!

Opensearch is:

A search an analyitics engine complatable wirth Elasticsearch

Can create dashboards - Opensearch dashboards.

Handles large amounts of data.

It will give you:

• Log analytics

• Full text searching

• Application monitoring