Summary

These are my notes, there are many like them, but these ones are mine.

• Summary
• Cheatsheets
  • API Architectural Styles
  • cap theorem
  • standard http headers
  • Latency Numbers Every SRE Should Know
  • Make Files
  • Neovim Cheatsheet
  • nmap
  • medusa
  • hydra
  • Ranger Cheatsheet
  • Regex Cheatsheet
  • SQL Cheat Sheet
  • yaml
• Clouds
  • AWS
    • DVA-C02-notes
      • Elastic Beanstalk
      • CloudFormation
      • CloudFront
      • Copilot
      • Elastic Container Registry
      • Elastic Container Service
      • Dynamodb
      • EC2
      • Elasticache
      • IAM
      • Kinesis
      • Lambda
      • CloudTrail
      • CloudWatch
      • x-ray
      • Aurora
      • RDS (Relational Database Service)
      • Route53
      • S3
      • Simple Notification System
      • Simple Queue System
      • VPC
    • AWS-Solutions-Architect-Associate-notes
      • AWS Certified Solutions Architect Associate Practice Exams
      • Table of Contents
      • Study-more
      • Tutorialsdojo Cheatsheets
        • Api-gateway-cheatsheet
        • Queueing (SQS)
        • Cloudfront-cheatsheet
        • CloudFront
        • What is Database ?
        • Disaster Recovery
        • Disaster-recovery-cheatsheet
        • Savings Plan
        • EC2
        • Elastic Load Balancer
        • Elastic File System (EFS)
        • What is ElastiCache for Redis?
        • Glue-cheatsheet
        • AWS Lambda
        • Ml-models
        • What is Amazon QuickSight ?
        • Aurora Cheatsheet
        • RDS
        • Amazon Redshift
        • DNS
        • IAM
        • AWS Certificate Manager
        • Storage-cheatsheet
        • Introduction to S3
        • Vpc-endpoint-cheatsheet
        • Vpc-flow-logs-cheatsheet
        • Introduction to VPC
  • Azure
    • Core Networking Infrastructure Checklist
      • Design-and-implement-core-network-infra
        • IPv4 and IPv6 Addressing
        • Azure DNS
        • Azure Virtual Network NAT
        • Subnets
        • Virtual Machine Scale Sets
        • Azure Virtual Network (VNet)
      • Design-and-implement-private-access
        • Private Link
        • Azure Service Endpoint
      • Design-and-implement-routing
        • Application Gateway
        • Azure Availability Sets
        • Azure Front Door
        • Azure Load Balancer
        • Azure Virtual Network Routing
        • Traffic Manager
      • Design-implement-and-manage-hybrid-networking
        • Azure Express Route
        • VPN
        • Azure Virtual WAN
      • Secure-and-monitor-networks
        • Application Security Groups
        • Azure Firewall
        • DDoS Protection
        • Network Watcher
        • Network Security Groups
        • Web Application Firewall
    • Assembly
    • The 4 Steps of Compilation with GCC
    • C Programming Notes
    • the bufio package
    • Go-projects
    • Immutability
    • Imperative-programming
• Computer Science
  • Algorithms
  • Computer Architecture
  • Data Structures
  • Euclid’s Algorithm
  • fizzbuzz
  • Graph Theory
  • Hashing
  • string algorithms
• DevOps
  • DevOps Principles
• Kubernetes
  • CKS
    • Certified Kubernetes Security Specialist (CKS) Notes
    • Kubernetes Security Specialist (CKS) Practice Scenarios
  • KCNA
    • Kubernetes Certified Native Associate (KCNA) Notes
  • KCSA
    • Kubernetes Certified Security Associate (KCSA) Notes
• Networking
  • Browser Networking
    • Chapter 1
    • Chapter 2
    • Chapter 3
    • Chapter 4
    • Chapter09
    • Chapter 10
    • Chapter 11
    • Chapter12
    • Chapter 13
    • Chapter 15
    • Chapter 16
    • Chapter 17
  • HTTP
    • Clean URLs
    • HTTP Persistent Connection
    • .–––––––– URN
  • Load Balancing
    • load balancing
  • Nginx
    • set header
  • Rate Limiting Algorithms
    • Fixed Window Counter Algorithm
    • Leaking Bucket Algorithm
    • TOKEN BUCKET ALGORITHM
• Redis
  • redis
• Systems
  • Linux Kernel Boot Process
  • Common Files and Directories
  • Dev Tools
  • devices
  • Disks
  • file systems
  • Groups
  • Hashing
  • Interrupts and Traps
  • kernel subsystems
  • Key Value Store
  • Hard and Soft Links
  • commands
  • Logging
  • lvm (logical volume manager)
  • make
  • memory
  • Memory Management
  • network manager
  • Networking
  • Linux Observability Sources
  • Pluggable Authentication Modules (PAM)
  • Per-Process Analysis
  • Permissions
  • processes
  • Scheduled Tasks
  • Bash Startup Files
  • Troubleshooting Storage
  • System Calls
  • System Wide Analysis
  • Systemd
  • The first 60 seconds
  • time
  • troubleshooting
  • Users and User Management
  • Bash
    • bash notes
    • Moving the cursor:
  • Commands
    • chgrp
    • Chmod
    • chown
    • dd
    • groups
    • ip
    • Job Control
    • Kill
    • lsscsi
    • passwd
    • ps
    • umask
  • Greybeard Qualification
    • Block Devices and File Systems
    • Memory Management
    • Execution and Scheduling of Processes and Threads
    • Process Structure and IPC
    • Startup and Init
  • Linux Kernel Development
    • Building the Linux Kernel
    • Kernel Modules
• Tools
  • Need to add/refine:
  • instructions for using dnspinger
  • nmap
  • medusa
  • hydra
• Troubleshooting
  • Performance Mantras
  • The Problem Statement
  • RED Method
  • USE Method
  • Linux
    • Troubleshooting Memory
  • Troubleshooting Playbook
    • Troubleshooting 503s for Apps in Kubernetes
    • General troubleshooting/notes
• What Happens When…
  • What happens when a CPU starts?
• Fun
  • Modify Machine Code

Cheatsheets

Directory Map

• api_architecture_styles
• cap_theorem
• http_headers
• latency_numbers
• make_files
• medusa
• neovim
• nmap
• ranger
• regex
• sql
• yaml
• hydra

API Architectural Styles

REST

Proposed in 2000, REST is the most used style. It is often used between front-end clients and back-end services. It is compliant with 6 architectural constraints. The payload format can be JSON, XML, HTML, or plain text.

GraphQL

GraphQL was proposed in 2015 by Meta. It provides a schema and type system, suitable for complex systems where the relationships between entities are graph-like. For example, in the diagram below, GraphQL can retrieve user and order information in one call, while in REST this needs multiple calls.

GraphQL is not a replacement for REST. It can be built upon existing REST services.

Web Socket

Web socket is a protocol that provides full-duplex communications over TCP. The clients establish web sockets to receive real-time updates from the back-end services. Unlike REST, which always “pulls” data, web socket enables data to be “pushed”.

Webhook

Webhooks are usually used by third-party asynchronous API calls. In the diagram below, for example, we use Stripe or Paypal for payment channels and register a webhook for payment results. When a third-party payment service is done, it notifies the payment service if the payment is successful or failed. Webhook calls are usually part of the system’s state machine.

gRPC

Released in 2016, gRPC is used for communications among microservices. gRPC library handles encoding/decoding and data transmission.

SOAP

SOAP stands for Simple Object Access Protocol. Its payload is XML only, suitable for communications between internal systems.

cap theorem

CAP theorem states that it is impossible for a distributed system to provide more than two of these guarantees: consistency, availability, and partition tolerance.

• Consistency
  • All clients see the same data at the same time from any node

• Availability

  • The ability for a system to respond to requests from users at all times

  

• Partition Tolerance

  • The ability for a system to continue operating even if there is a partition in the network

  

standard http headers

Latency Numbers Every SRE Should Know

nanosecond = 1/1,000,000,000 second microsecond = 1/1,000,000 second millisecond = 1/1000 second

Sub-Nanosecond Range

• Accessing CPU registers
• CPU Clock Cycle

1-10 Nanosecond Range

• L1/L2 cache
• Branch Misprediction in CPU pipelining

10-100 Nanosecond Range

• L3 cache
• Apple M1 referencing main memory (RAM)

100-1000 Nanosecond Range

• System call on Linux
• MD5 hash a 64-bit number

1-10 Microsecond Range

• Context switching between Linux threads

10-100 Microsecond Range

• Process a HTTP request
• Reading 1 megabyte of sequential data from RAM
• Read an 8k page from an ssd

100-1000 Microsecond Range

• SSD write Latency
• Intra-zone networking round trip in most cloud providers
• Memcache/Redis get operation

1-10 Millisecond Range

• Inter-zone networking Latency
• Seek time of a HDD

10-100 Millisecond Range

• Network round trip between US-west and US-east coast
• Read 1 megabyte sequentially from main memory

100-1000 Millisecond Range

• Some encryption/hashing algorithms
• TLS handshake
• Read 1 Gigabyte sequentially from an SSD

1 second+

• Transfer 1GB over a cloud network within the same region

Make Files

Why do Make files exist?

Make files are used for automation. Typically as a step in the software development lifecycle (compilation, builds, etc.). However, they can be used for any other task that can be automated via the shell.

Make files must be indented using tabs, not spaces

Makefile Syntax

Makefiles consist of a set of rules. Rules typically look like this:

targets: prerequisites
	command
	command
	command

• The targets are file names, separated by spaces. Typically, there is only 1 per rule.
• The commands are a series of steps typically used to make targets.
• The prerequisites are also file names, separated by spaces. These files need to exist before the commands for the target are run. These are dependencies to the targets.

Example

Let’s start with a hello world example:

hello:
	echo "Hello, World"
	echo "This line will print if the file hello does not exist."

There’s already a lot to take in here. Let’s break it down:

• We have one target called hello
• This target has two commands
• This target has no prerequisites

We’ll then run make hello. As long as the hello file does not exist, the commands will run. If hello does exist, no commands will run. It’s important to realize that I’m talking about hello as both a target and a file. That’s because the two are directly tied together. Typically, when a target is run (aka when the commands of a target are run), the commands will create a file with the same name as the target. In this case, the hello target does not create the hello file.

Let’s create a more typical Makefile - one that compiles a single C file. But before we do, make a file called blah.c that has the following contents:

// blah.c
int main() { return 0; }

Then create the Makefile (called Makefile, as always):

blah:
	cc blah.c -o blah

This time, try simply running make. Since there’s no target supplied as an argument to the make command, the first target is run. In this case, there’s only one target (blah). The first time you run this, blah will be created. The second time, you’ll see make: ‘blah’ is up to date. That’s because the blah file already exists. But there’s a problem: if we modify blah.c and then run make, nothing gets recompiled.

We solve this by adding a prerequisite:

blah: blah.c
	cc blah.c -o blah

When we run make again, the following set of steps happens:

• The first target is selected, because the first target is the default target
• This has a prerequisite of blah.c
• Make decides if it should run the blah target. It will only run if blah doesn’t exist, or blah.c is newer than blah

This last step is critical, and is the essence of make. What it’s attempting to do is decide if the prerequisites of blah have changed since blah was last compiled. That is, if blah.c is modified, running make should recompile the file. And conversely, if blah.c has not changed, then it should not be recompiled.

To make this happen, it uses the filesystem timestamps as a proxy to determine if something has changed. This is a reasonable heuristic, because file timestamps typically will only change if the files are modified. But it’s important to realize that this isn’t always the case. You could, for example, modify a file, and then change the modified timestamp of that file to something old. If you did, Make would incorrectly guess that the file hadn’t changed and thus could be ignored.

Make Clean

clean is often used as a target that removes the output of other targets, but it is not a special word in Make. You can run make and make clean on this to create and delete some_file.

Note that clean is doing two new things here:

• It’s a target that is not first (the default), and not a prerequisite. That means it’ll never run unless you explicitly call make clean
• It’s not intended to be a filename. If you happen to have a file named clean, this target won’t run, which is not what we want. See .PHONY later in this tutorial on how to fix this

some_file: 
	touch some_file

clean:
	rm -f some_file

Variables

Variables can only be strings. You’ll typically want to use :=, but = also works.

Here’s an example of using variables:

files := file1 file2
some_file: $(files)
	echo "Look at this variable: " $(files)
	touch some_file

file1:
	touch file1
file2:
	touch file2

clean:
	rm -f file1 file2 some_file

targets

The ‘all’ target

Making multiple targets and you want all of them to run? Make an all target. Since this is the first rule listed, it will run by default if make is called without specifying a target.

all: one two three

one:
	touch one
two:
	touch two
three:
	touch three

clean:
	rm -f one two three

Multiple targets

When there are multiple targets for a rule, the commands will be run for each target. $@ is an automatic variable that contains the target name.

all: f1.o f2.o

f1.o f2.o:
	echo $@
# Equivalent to:
# f1.o:
#	 echo f1.o
# f2.o:
#	 echo f2.o

Reference

Var assignment

foo  = "bar"
bar  = $(foo) foo  # dynamic (renewing) assignment
foo := "boo"       # one time assignment, $(bar) now is "boo foo"
foo ?= /usr/local  # safe assignment, $(foo) and $(bar) still the same
bar += world       # append, "boo foo world"
foo != echo fooo   # exec shell command and assign to foo
# $(bar) now is "fooo foo world"

= expressions are only evaluated when they’re being used.

Magic variables

out.o: src.c src.h
  $@   # "out.o" (target)
  $<   # "src.c" (first prerequisite)
  $^   # "src.c src.h" (all prerequisites)

%.o: %.c
  $*   # the 'stem' with which an implicit rule matches ("foo" in "foo.c")

also:
  $+   # prerequisites (all, with duplication)
  $?   # prerequisites (new ones)
  $|   # prerequisites (order-only?)

  $(@D) # target directory

Command prefixes

build:
    @echo "compiling"
    -gcc $< $@

-include .depend

Find files

js_files  := $(wildcard test/*.js)
all_files := $(shell find images -name "*")

Substitutions

file     = $(SOURCE:.cpp=.o)   # foo.cpp => foo.o
outputs  = $(files:src/%.coffee=lib/%.js)

outputs  = $(patsubst %.c, %.o, $(wildcard *.c))
assets   = $(patsubst images/%, assets/%, $(wildcard images/*))

More functions

$(strip $(string_var))

$(filter %.less, $(files))
$(filter-out %.less, $(files))

Building files

%.o: %.c
  ffmpeg -i $< > $@   # Input and output
  foo $^

Includes

-include foo.make

Options

make
  -e, --environment-overrides
  -B, --always-make
  -s, --silent
  -j, --jobs=N   # parallel processing

Conditionals

foo: $(objects)
ifeq ($(CC),gcc)
  $(CC) -o foo $(objects) $(libs_for_gcc)
else
  $(CC) -o foo $(objects) $(normal_libs)
endif

Recursive

deploy:
  $(MAKE) deploy2

Further reading

• isaacs’s Makefile
• Your Makefiles are wrong
• Manual

Neovim Cheatsheet

Mode Switching:

• i: Insert mode before cursor
• I: Insert mode at the beginning of line
• a: Insert mode after cursor
• A: Insert mode at the end of line
• v: Visual mode
• V: Visual line mode
• ^V (Ctrl + V): Visual block mode
• :q: Quit (add ! to force)
• :w: Save/write
• :wq or ZZ: Save and Quit

Cursor Movement:

• h: Left
• j: Down
• k: Up
• l: Right
• w: Jump by start of words
• e: Jump to end of words
• b: Jump backward by words
• 0: Start of line
• $: End of line
• G: Go to last line of document
• gg: Go to first line of document
• ^: First non-blank character of line
• : followed by a number: Go to that line number (e.g., :10)

Editing:

• u: Undo
• ^R (Ctrl + R): Redo
• yy or Y: Yank/copy line
• dd: Delete line
• D: Delete from cursor to end of line
• x: Delete character under cursor
• p: Paste after cursor
• P: Paste before cursor
• r followed by a character: Replace character under cursor with the new character
• cw: Change word

Search and Replace:

• / followed by a term: Search for term (press n to go to next and N for previous)
• :%s/old/new/g: Replace all occurrences of “old” with “new” in the entire file

Windows & Tabs:

• ^W (Ctrl + W) followed by h/j/k/l: Move cursor to another window
• :split or :sp: Split window horizontally
• :vsplit or :vsp: Split window vertically
• :tabnew or :tabn: Create a new tab
• gt: Move to next tab
• gT: Move to previous tab

Others:

• .: Repeat last command
• *: Search for word under cursor
• #: Search for word under cursor, backwards
• ~: Switch case of character under cursor
• o: Insert new line below and enter insert mode
• O: Insert new line above and enter insert mode

• : Indent line

• <<: Dedent line

Nmap Cheatsheet

Basic Scan Types

Port Selection

Important Flags

Port States

Useful Examples

Scan Top 10 TCP Ports

nmap --top-ports=10 <target>

Full TCP + UDP + Version + OS

nmap -sS -sU -sV -O <target>

Packet Trace Example

nmap -p 21 --packet-trace -Pn -n --disable-arp-ping <target>

Service Enumeration

nmap -sV -p <port> <target>

Medusa Cheatsheet

Medusa is a fast, massively parallel, and modular login brute-forcer designed to support a wide array of services that allow remote authentication.

Installation

sudo apt-get -y update
sudo apt-get -y install medusa

Command Syntax

medusa [target_options] [credential_options] -M module [module_options]

Parameters

Modules

Useful Examples

SSH Brute-Force Attack

medusa -h 192.168.0.100 -U usernames.txt -P passwords.txt -M ssh

Multiple Web Servers with Basic HTTP Authentication

medusa -H web_servers.txt -U usernames.txt -P passwords.txt -M http -m GET

Test for Empty or Default Passwords

medusa -h 10.0.0.5 -U usernames.txt -e ns -M ssh

HTTP POST Form Attack

medusa -M http -h www.example.com -U users.txt -P passwords.txt -m "POST /login.php HTTP/1.1\r\nContent-Length: 30\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\nusername=^USER^&password=^PASS^"

Fast Mode (Stop on First Success)

medusa -h 192.168.1.100 -U usernames.txt -P passwords.txt -M ssh -f

Custom Port SSH Attack

medusa -h 192.168.1.100 -n 2222 -U usernames.txt -P passwords.txt -M ssh

Verbose Output

medusa -h 192.168.1.100 -U usernames.txt -P passwords.txt -M ssh -v 4

Parallel Tasks

medusa -h 192.168.1.100 -U usernames.txt -P passwords.txt -M ssh -t 8

Hydra Cheatsheet

Basic Syntax

hydra [login_options] [password_options] [attack_options] [service_options] service://server

Login Options

Password Options

Attack Options

Common Services

Useful Examples

SSH Brute Force

hydra -l root -P /path/to/passwords.txt -t 4 ssh://192.168.1.100

FTP Brute Force

hydra -L usernames.txt -P passwords.txt ftp://192.168.1.100

HTTP POST Form Attack

hydra -l admin -P passwords.txt http-post-form "/login.php:user=^USER^&pass=^PASS^:F=incorrect" 192.168.1.100

RDP with Password Generation

hydra -l administrator -x 6:8:aA1 rdp://192.168.1.100

SSH on Non-Default Port

hydra -l admin -P passwords.txt -s 2222 ssh://192.168.1.100

Stop After First Success

hydra -l admin -P passwords.txt -f ssh://192.168.1.100

Verbose Output

hydra -l admin -P passwords.txt -v ssh://192.168.1.100

RDP with Custom Character Set

hydra -l administrator -x 6:8:abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 rdp://192.168.1.100

Ranger Cheatsheet

General

Movement

File Operations

Commands

Tabs

File substituting

Example for substitution

:bulkrename %s

Marker

thx to the comments section for additional shortcuts! post your suggestions there!

Regex Cheatsheet

Regular expressions (regex) are patterns used for string matching and manipulation. Here’s a quick reference guide for common regex syntax:

Basics

• .: Matches any character except a newline.
• ^: Matches the start of a string or line.
• $: Matches the end of a string or line.

Character Classes

• [abc]: Matches any character a, b, or c.
• [^abc]: Matches any character except a, b, or c.
• [a-z]: Matches any lowercase letter.
• [A-Z]: Matches any uppercase letter.
• [0-9]: Matches any digit.
• [a-zA-Z0-9]: Matches any alphanumeric character.
• \d: Matches any digit (short for [0-9]).
• \w: Matches any word character (alphanumeric + underscore).
• \s: Matches any whitespace character (space, tab, newline).

Quantifiers

• *: Matches the preceding element zero or more times.
• +: Matches the preceding element one or more times.
• ?: Matches the preceding element zero or one time.
• {n}: Matches the preceding element exactly n times.
• {n,}: Matches the preceding element n or more times.
• {n,m}: Matches the preceding element between n and m times.

Groups and Alternation

• (abc): Matches the group abc and captures it.
• (?:abc): Matches the group abc without capturing it.
• a|b: Matches either a or b.

Anchors

• \b: Matches a word boundary.
• \B: Matches a position that is not a word boundary.
• (?=...): Positive lookahead assertion.
• (?!...): Negative lookahead assertion.

Escaping Special Characters

• \\: Escapes a special character (e.g., \\. matches a literal period).

Flags (Depends on Language)

• i: Case-insensitive matching.
• g: Global match (find all occurrences).
• m: Multiline mode (^ and $ match the start/end of each line).
• s: Dot matches all, including newlines.
• u: Treat the pattern and input as UTF-16 or UTF-32.
• x: Ignore whitespace and allow comments.

Examples

• /\d{3}-\d{2}-\d{4}/: Matches a standard US social security number.
• /^[A-Za-z]+$/: Matches a string containing only letters.
• /https?:\/\/(www\.)?\w+\.\w+/: Matches URLs starting with http:// or https://.
• /(\d+)\s?-\s?\1/: Matches repeated numbers separated by a hyphen.
• ^(\/[^\/?]+)(\/[^\/?]+)?(\/[^\/?]+)?: Match all text up to the 3rd ‘/’ in a URL
• ^[a-z][a-z0-9+\-.]*://([a-z0-9\-._~%!$&'()*+,;=]+@)?([a-z0-9\-._~%]+|\[[a-z0-9\-._~%!$&'()*+,;=:]+\]): extract hostname from URL
• ^https?:\/\/(.*)(\/[^\/?]+)(\/[^\/?]+)?(\/[^\/?]+)?: Match protocol (http/s), hostname, and path (3 forward slashes)

Useful links

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions/Cheatsheet

SQL Cheat Sheet

Who has a role:

 select @@ServerName [Server Name], DB_NAME() [DB Name], u.name [DB Role], u2.name [Member Name]
    from sys.database_role_members m
    join sys.database_principals u on m.role_principal_id = u.principal_id
    join sys.database_principals u2 on m.member_principal_id = u2.principal_id
    where u.name = 'db_owner'
    order by [Member Name]

Who logged in as dbo:

#in user database run the command 
SELECT name, sid FROM sys.sysusers where name = 'dbo' . 
#in master database run the command 
SELECT name, sid FROM sys.sql_logins

multi-line strings

There are 9 (or 63*, depending how you count) different ways to write multi-line strings in YAML.

• Use > most of the time: interior line breaks are stripped out, although you get one at the end:

    key: >
    Your long
    string here.
  

• Use | if you want those linebreaks to be preserved as \n (for instance, embedded markdown with paragraphs):

    key: |
    ### Heading
  
    * Bullet
    * Points
  

• Use >- or |- instead if you don’t want a linebreak appended at the end.

• Use “…” if you need to split lines in the middle of words or want to literally type linebreaks as \n:

   key: "Hello\
   World!\n\nGet on it."
  

Clouds

Notes related to various cloud service providers.

Directory Map

• aws
• azure

AWS

Notes related to Amazon Web Services (AWS).

• AWS Certified Developer: Notes for the AWS Certified Developer - Associate (DVA-C02) exam.
• Solutions Architect - Associate: Notes for the AWS Certified Solutions Architect - Associate (SAA-C03) exam.

DVA-C02-notes

Directory Map

• beanstalk
• cloudformation
• cloudfront
• containers
  • copilot
  • ecr
  • ecs
• dynamodb
• ec2
• elasticache
• iam
• kinesis
• lambda
• monitoring
  • cloudtrail
  • cloudwatch
  • x-ray
• rds
  • aurora
  • rds
• route53
• s3
• sns
• sqs
• vpc

Elastic Beanstalk

Introduction

• Developer centric view of deploying an application on AWS
• Simplifies deploying EC2, ASG, ELB, RDS, etc.
• Fully managed by AWS

Beanstalk Components

• Application: A collection of Beanstalk components
• Application Version: an iteration of your application
• Environment:
  • collection of AWS resources running an application version
  • Tiers: web server environment and worker environment
  • You can create multiple environments (dev, prod, QA, etc.)

Supported Platforms

• Multiple languages supported: Go, Python, Java, .NET, Node, PHP, Ruby, Packer Builder,
• Supports single container docker, multi-container docker, etc.

Deployment Modes

• Single Instance
  • One EC2 instance
  • Great for dev environments
• HA with Load Balancer
  • Multiple EC2 instances in ASG
  • Great for prod

Deployment Options

• All at once
  • fastest, but instances aren’t available to service traffic for a bit (downtime)
• Rolling
  • A few instances at a time are taken offline and updated
• Rolling with additional batches
  • Like rolling, but spins up new instances to move the batch
• Immutable
  • Spins up new instances in a new ASG, deploys versions to these instances, and then swaps all the instances when the new are healthy
• Blue/Green
  • Create a new environment and switch over when ready
• Traffic Splitting
  • Like canary testing

Beanstalk Lifecycle Policy

• Beanstalk can store at most 1000 application versions
• Versions currently in use cannot be deleted

CloudFormation

Introduction

• Declarative language for deploying resources in AWS
• YAML or JSON
• CloudFormation templates can be visualized using Application Composer

Example

---
Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      AvailabilityZone: us-east-1a
      ImageId: ami-0a3c3a20c09d6f377
      InstanceType: t2.micro

CloudFormation Template Sections

• Resources

  • The only required section in a template
  • The resources section represent the AWS components that the CF template will deploy
  • Resource type identifiers are in this format:
    • service-provider::service-name::data-type-name

• Parameters

  • Provide input to your CF templates

• Mappings

  • Fixed variables in your CF template used to differentiate between different environment like dev vs prod, regions, AMI types, etc.

  • To access values in a map, use Fn::FindInMap:

        {
          ...
          "Mappings" : {
            "RegionMap" : {
              "us-east-1" : {
                "HVM64" : "ami-0ff8a91507f77f867", "HVMG2" : "ami-0a584ac55a7631c0c"
              },
              "us-west-1" : {
                "HVM64" : "ami-0bdb828fd58c52235", "HVMG2" : "ami-066ee5fd4a9ef77f1"
              },
              "eu-west-1" : {
                "HVM64" : "ami-047bb4163c506cd98", "HVMG2" : "ami-0a7c483d527806435"
              },
              "ap-southeast-1" : {
                "HVM64" : "ami-08569b978cc4dfa10", "HVMG2" : "ami-0be9df32ae9f92309"
              },
              "ap-northeast-1" : {
                "HVM64" : "ami-06cd52961ce9f0d85", "HVMG2" : "ami-053cdd503598e4a9d"
              }
            }
          },
    
          "Resources" : {
            "myEC2Instance" : {
              "Type" : "AWS::EC2::Instance",
              "Properties" : {
                "ImageId" : {
                  "Fn::FindInMap" : [
                    "RegionMap",
                    {
                      "Ref" : "AWS::Region"
                    },
                    "HVM64"
                  ]
                },
                "InstanceType" : "m1.small"
              }
            }
          }
        }
    

• Outputs

  • Optional
  • Output values can be referenced in other stacks
  • Use FN:ImportValue

• Conditions

  • Control the creation of resources or outputs based on a condition

Intrinsic Functions

• Fn::Ref - Get a references to a value of a paremeter, physical Id of a resource, etc.
• Fn::GetAtt - Get attributes from a resource
• Fn::FindInMap - Retrieve a value from a map
• Fn::ImportValue - Import an output value from another template
• Fn::Base64 - Convert a value to Base64 inside a template
• Condition Functions (Fn::If, Fn::Not, Fn::Equals, etc.)
• etc….

Service Roles

• IAM roles that allow CloudFormation to create/update/delete stack resources

CloudFront

Table of Contents

• Introduction
• CloudFront Core Components
• CloudFront Distributions
• Lambda@Edge
• CloudFront Protection
• Caching Policy
• Caching Behaviors
• Geo-Restriction
• CloudFront Signed URL
• Pricing

Introduction

• Content Distribution Network (CDN) creates cached copies of your website at various Edge locations around the world
• Content Delivery Network (CDN)
  • A CDN is a distributed network of servers which delivers web pages and content to users based on their geographical location, the origin of the webpage and a content delivery server
    • Can be used to deliver an entire website including static, dynamic and streaming
    • 216 points of presence globally
    • DDoS protection since it is a global service. Integrates with AWS Shield and AWS WAF
    • Requests for content are served from the nearest Edge Location for the best possible performance

CloudFront Core Components

• Origin
  • The location where all of original files are located. For example an S3 Bucket, EC2 Instance, ELB or Route53
• Edge Location
  • The location where web content will be cached. This is different than an AWS Region or AZ
• Distribution
  • A collection of Edge locations which defines how cached content should behave

CloudFront Distributions

• A distribution is a collection of Edge Location. You specific the Origin eg. S3, EC2, ELB, Route53
• It replicates copies based on your Price Class
• There are two types of Distributions
  1. Web (for Websites)
  2. RTMP (for streaming media)
• Behaviors
  • Redirect to HTTPs, Restrict HTTP Methods, Restrict Viewer Access, Set TTLs
• Invalidations
  • You can manually invalidate cache on specific files via Invalidations
• Error Pages
  • You can serve up custom error pages eg 404
• Restrictions
  • You can use Geo Restriction to blacklist or whitelist specific countries

Lambda@Edge

• Lambda@Edge functions are used to override the behavior of request and responses

• Lambda@Edge lets you run Lambda functions to customize the content that CloudFront delivers, executing the functions in AWS locations closer to the viewer.

• The functions run in response to CloudFront events, without provisioning or managing servers. You can use Lambda functions to change CloudFront requests and responses at the following points:

• The 4 Available Edge Functions

  1. Viewer Request
     • When CloudFront receives a request from a Viewer
  2. Origin request
     • Before CLoudFront forwards a request to the origin
  3. Origin response
     • When cloudfront receives a response from the origin
  4. Viewer response
     • Before CLoudFront returns the response to the viewer
  

CloudFront Protection

• By Default a Distribution allows everyone to have access
• Original Identity Access (OAI)
  • A virtual user identity that will be used to give your CloudFront Distribution permission to fetch a private object
• Inorder to use Signed URLs or Signed Cookies you need to have an OAI
• Signed URLs
  • (Not the same thing as S3 Presigned URL)
    • A url with provides temporary access to cached objects
• Signed Cookies
  • A cookie which is passed along with the request to CloudFront. The advantage of using a Cookie is you want to provide access to multiple restricted files. eg. Video Streaming

Caching Policy

• Each object in the cache will be identified by a cache key
• Maximize the cache-hit-ratio by minimizing requests to the origin
• Cache Key
  • a unique identifier for everything in the cache
  • by default, made up of the hostname and resource portion of the URL
  • The cache key can be customized by creating a CloudFront Cache Policy

Caching Behaviors

• Configure different settings for a given URL path pattern
• Example: configure a specific behavior for requests to /images/*.jpg
• Route to different kinds of origins/origin groups based on the content-type or path.
• Examples:
  • /images/* to S3
  • /login to EC2
  • /api to API Gateway

Geo-Restriction

• Restrict who can access your CloudFront distribution based on the country where the distribution was access from
• You can create an AllowList or a BlockList

CloudFront Signed URL

• Two types of signers:
  • Either a trusted key group (Recommended)
  • An AWS Account that contains a CloudFront Key Pair
• In your distribution, create one or more trusted key groups

Pricing

• You can reduce the number of edge locations for a cost savings
• Price Classes
  • Price Class All: All regions, best performance
  • Price Class 200: most regions, but excludes the most expensive regions
  • Price Class 100: only the least expensive regions

Copilot

Introduction

• CLI tool to build, release, and maintain production ready containerized apps
• Helps you focus on building apps rather than setting up infrastructure
• Automatically provisions all required infrastructure for a containerized app
• Automate deployments using one command with CodePipeline
• Deploy to ECS, Fargate, or App Runner

Elastic Container Registry

Introduction

• Store container images in AWS, similar to DockerHub
• Public or Private

Elastic Container Service

Introduction

• Launch container instances on AWS as ECS Tasks
• Launch Types:
  • EC2: You must provision and manage the infrastructure (EC2 instances)
    • Each EC2 instance must run the ECS Agent to register in the ECS cluster

  • Fargate: AWS provisions and manages the infrastructure

IAM Roles for ECS

• EC2 Instance Profile
  • Used by the ECS agent to make API calls to ECS Service, send container logs to CloudWatch, pull docker images from ECR, etc.
• ECS Tasks:
  • Each ECS tasks gets a role. Applies to both EC2 and Fargate launch types
  • Task role is defined in the Task Definition

Load Balancer Integrations

• ALB supports and works for most use cases
• NLB is recommended only for high-throughput use cases

Data Volumes

• Mount EFS file systems onto ECS Tasks
• Works for both EC2 and Fargate Launch Types
• Tasks running in any AZ will share the same data in the EFS file system
• EFS + Fargate = serverless
• S3 cannot be mounted on a file system in your ECS Tasks

ECS Service Auto-scaling

• You can scale on 3 metrics
  • CPU
  • Memory
  • ALB Request Count
• Types of Scaling:
  • Target Tracking: scale based on target value for a specific CloudWatch metric
  • Step Scaling: scale based on a specified CloudWatch Alarm
  • Scheduled Scaling: scale based on a specified date/time
• Auto Scaling EC2 instances
  • Use an Auto Scaling Group
    • Scale based on CPU usage
  • ECS Cluster Capacity Provider
    • Used to automatically provision and scale the infrastructure of your ECS Tasks
    • Capacity Provider is paired with an auto-scaling group
    • Add EC2 instances when you are out of usable capacity (CPU, RAM, etc…)

ECS Rolling Updates

• When updating versions of an ECS service, we can control how many tasks can be started and stopped, and in which order
• Rolling Updates

ECS Tasks Definitions

• Tasks definitions are metadata in JSON form to tell ECS how to run a container
• How can we add environment variables to an ECS Task?
  • Hardcoded - URLs for example
  • SSM Parameter Store - sensitive variables such as API keys
  • Secrets Manager - Sensitive variables
• Bind mounts
• Essential Container -If enabled: If one container in the task fails or stops, all the other containers in the task will stop

ECS Task Placement

• When a Task Definition for EC2 is created, ECS must determine where to schedule it

• When a service scales in, ECS must determine which tasks to kill

• To help with this, you can define a task placement strategy and task placement constraint

• Task placement strategies and constraints only work on EC2 launch types

• How does this work?

  • ECS will first determine where it is possible to place the task. Which nodes have enough resources?
  • ECS will then determine which EC2 instances satisfy the task placement constraints
  • ECS will then determine which EC2 instances satisfy the task placement strategy
  • ECS will then schedule the task

• Task Placement Strategies

  • BinPack: Schedule tasks based on the least available amount of CPU or memory
    • i.e. pack as many containers as possible on a node before scheduling containers on other nodes
  • Random: Place the task randomly
  • Spread: Spread instances across nodes based on a specified value (AZ, instanceId, etc.)

• Task Placement Constraints

  • distinctInstance: Each task should be placed on a separate EC2 instance
  • memberOf: Schedule tasks on instances that satisfy an expression written in cluster query language ()

Dynamodb

Introduction

• Fully managed, highly available with replication across AZs
• NoSQL Database
• Scales to massive workloads, highly distributed database
• Fully integrates with IAM
• Enable event driven programming with DynamoDB streams

Basics

• DynamoDB is made of tables
• Each table will have primary key (must be decided at creation time)
• Each table can have an infinite number of rows (rows are items)
  • Each row can have a max of 400KB of data

Primary keys

• how to choose a primary key?
• Two types of primary keys
  • partition keys
    • Partition key must be unique for each item
    • Partition key must be ‘diverse’ so that data is distributed
  • partition key + sort key
    • data is grouped by partition key
    • each combination must be unique per item

Read/Write Capacity Modes

• Provisioned Mode
  • Provision all capacity in advance by providing values for RCU and WCU
  • Pay up front
  • Throughput can temporarily be exceeded using a burst capacity
  • If you exhaust the burst capacity, you can retry using exponential backoff
• On-demand Mode
  • Reads/writes scale on-demand
  • Pay for what you use
• You can switch between the two modes once every 24 hours

Local Secondary Index

• Alternate sort key for your table
• Must be defined at table creation time
• Up to 5 local secondary indexes per table
• The sort key consists of one scaler attribute (string, number, or binary)

Global Secondary Index

• Alternative primary key from the base table

PartiQL

• Use SQL-like syntax to query DynamoDB tables

Optimistic Locking

• DynamoDB has a feature called “conditional writes” to ensure an item has not changed before writing to it
• Each item has an attribute that acts as a version number
• Useful for when you have multiple writers attempters to write to an item

DynamoDB Accelerator (DAX)

• Fully managed, highly available, seamless in-memory cache for DynamoDB
• Microseconds latency for cached reads and queries
• Does not require that you change any application code
• Solves the “hot key” problem. If you read a specific key (item) too many times, you may get throttled.
• 5 minutes TTL for cache (default)
• Up to 10 DAX nodes per cluster
• Multi-AZ (3 nodes minimum recommended for production)
• Secure (Encryption in transit and at rest)

DynamoDB Streams

• Streams are an ordered item-level modifications (create/update/delete) in a table
• Stream records can be:
  • Sent to Kinesis Data Streams
  • Read by AWS Lambda
  • Read by Kinesis Client Library Apps
• Data retention for up to 24 hours
• Use cases:
  • react to changes in real-time
  • Analytics
  • Insert into derivative tables
  • Insert into OpenSearch service

EC2

Table of Contents

• Introduction
• Budgets
• Instance Types
• Security Groups
• EC2 Purchasing Options
• EBS Volumes
• EBS Volume Snapshots
• AMI (AWS Machine Image)
• EC2 Instance Store
• EBS Volume Types
• EBS Multi-attach
• EFS
• Elastic Load Balancer (ELB)
• Autoscaling Groups
• IMDS

Introduction

• EC2 is Amazon’s Elastic Compute Cloud
• It is comprised of virtual machines, storage, load balancers, auto scaling VMs
• You can run Windows, Linux, or MacOS
• You can choose how many vCPUs and how much RAM you want
• You can choose how much storage space you want
• You can choose what type of network card and whether or not you need a public IP (Elastic IP)
• You can run a bootstrap script at launch time called “user data”. The script is only run once when the instance first starts.
• You can configure firewall rules via a Security Group

Budgets

• You can create a budget and alert to ensure you don’t go over a certain cost
• You need to enable “IAM user and role access to Billing information” in your account settings

Instance Types

• General Purpose
  • Balance between compute, memory, and networking
  • Great for diversity of workloads such as web servers or code repositories
• Compute Optimized
  • Optimized for compute intensive tasks
  • Examples: Machine Learning, batch processing, HPC, etc.
• Memory Optimized
  • Optimized for memory intensive tasks
  • High performance databases or caches
• Storage Optimized
  • Example use cases: OLTP, databases, caches, etc.
• Accelerated Computing
  • HPC

Example:

m5.2xlarge
| |   |
| |   +-- 2xlarge: size within instance class
| +------ 5: generation
+--------- m: instance class

Security Groups

• Security groups are like a firewall scoped to the EC2 instance
• Security groups only contain allow rules
• Security groups are stateful
• Security groups can have a source of IP/range or another security group
• An instance can have multiple security groups attached
• A security group can be attached to multiple instances
• Security groups are region-locked

Ports to know for the exam

• 21 = FTP
• 22 = SSH/sFTP
• 80 = HTTP
• 443 = HTTPS
• 3389 = RDP
• 5432 - Postgresql
• 3306 - MySQL / MariaDB
• 1521 - Oracle
• 1433 - MSSQL

EC2 Purchasing Options

• On-Demand
  • Short workload, predictable pricing
  • Linux or Windows, billed per second. Other operation systems, billed per hour
• Reserved
  • 1 to 3 years commitment
  • Used for long workloads
  • Up to a 72% discount compared to on-demand
  • Pay upfront, partially upfront, or no upfront
  • Scoped to a region or zone
  • You can buy or sell them in the Reserved Instances Marketplace
• Savings Plan
  • Up to a 72% discounted compared to on-demand
  • Commit to a certain type of usage (example: $10/hour for 1 to 3 years). Usage beyond the commitment is billed at the on-demand price
  • Locked to a specific instance family and AWS region (example: M5 in us-east-1)
  • 1 to 3 years commitment
  • Commit to an amount of usage
• Spot Instances
  • Short workloads, cheap, less reliable
  • The MOST cost efficient option
  • Workload must be resilient to failure
• Dedicated Hosts
  • Reserve an entire physical server, control instance placement
  • Allows you to address compliance or license requirements
  • Purchasing Options:
    • On-demand
    • Reserved for 1 to 3 years
  • The most expensive option
• Dedicated Instances
  • No other customers will share your hardware
  • You may share the hardware with other instances in the same account
  • No control over the instance placement
• Capacity Reservation
  • Reserve capacity in a specific AZ for any duration
  • You always have access to the EC2 capacity when you need it
  • No time reservations
  • Combine with regional reserved instances or a savings plan for cost savings
  • Even if you don’t launch instances, you still get charged

EBS Volumes

• AN EBS (Elastic Block Store) Volume is a network drive which you can attach to you instances while they run
• EBS volumes are bound to a specific Availability Zone
  • To move an EBS volume to another AZ, you must first snapshot it and then copy the snapshot
• EBS volumes have a provisioned capacity
  • You are billed for the provisioned capacity
• IOPS typically scale with capacity (i.e. larger volumes have better performance)
• EBS volumes have a “Delete on Termination” attribute. This is enabled for the root volume by default, but not for other volumes

EBS Volume Snapshots

• To move an EBS volume to another AZ, you must first snapshot it and then copy the snapshot
• EBS Snapshot Archive
  • Gives you the ability to move snapshots to the archive tier, which is up to 75% cheaper
  • Takes 24-72 hours to restore the snapshot
• EBS Snapshot Recycle Bin
  • Allows you to restore deleted snapshot
  • Retention can be 1 day to 1 year
• Fast Snapshot Restore
  • Force full initialization of snapshot to have to latency on first use
  • can be very expensive

AMI (AWS Machine Image)

• VM Image

• AMI’s are built for a specific region and can be copied to other regions

• AMI Types:

  • Private
  • Public
  • MarketPlace

• AMI Creation Process

  • Start instance and customize it
  • Stop the instance
  • Capture the AMI

EC2 Instance Store

• Storage mounted in an EC2 instance that is local to the physical host
• High performance
• The storage is wiped when the EC2 instance stops or is terminated
• Use cases: cache, temporary content, or scratch space

EBS Volume Types

• GP2/GP3 - General SSD

  • 1 GB up to 16 TB

• IO1/IO2 - High performance SSD

• ST1 (hdd) - low cost HDD volume

• SC1 (hdd) - Lowest cost HDD volume

• Only GP2/3 and IO1/2 can be used as root (bootable) volumes

EBS Multi-attach

• Attach the same EBS volume to multiple instances (up to 16) in the same AZ
• Only available for IO1/IO2 family of EBS volumes
• Each instance will have read/write access to the volume
• You must use a file system that is cluster aware

EFS

• Managed NFS (Network File System)

• Pay per use

• 3x more expensive than a GP2 EBS volume

• Can be mounted on different EC2 instances in different Availability Zones

• EFS Scale

  • 1000s of concurrent clients, 10 GB+ throughput
  • Grow to petabyte scale network file system, automatically
  • Performance Classes:
    • Performance Mode:
      • General purpose: latency sensitive use cases (web server, CMS, etc…)
      • Max I/O: higher latency, throughput, highly parallel (big data, media processing)
    • Throughput Mode:
      • Bursting: 1 TB= 50MB/s + burst up to 100MB/s
      • Provisioned - set your throughput regardless of storage size
      • Elastic - Automatically scales throughput up or down based on your workloads
  • Storage Classes:
    • Storage Tiers (move files to another tier after ‘x’ number of days)
      • Standard
      • Infrequent Access
      • Archive
    • Implement lifecycle policies to move files between tiers

Elastic Load Balancer (ELB)

• Load balancers forward traffic to multiple backend servers

• ELB is a managed load balancer

• ELB is integrated with many offerings and services ()

• ELB supports health checks to verify if a backend instance is working before forwarding traffic to it

• Types of load balancers on AWS:

  • Application Load Balancer

    • Layer 7
      • Support HTTP2 and WebSockets
    • Supports HTTP redirects
    • Supports URL path routing, hostname routing, query string routing, header routing
    • Backend instances are grouped into a Target Group
    • You get a fixed hostname
    • The app servers don’t see the IP of the client directly
      • If the app servers need to know the client IP/port/protocol, they can check the following headers:
        • X-Forwarded-For
        • X-Forwarded-Proto
        • X-Forwarded-Port

  • Network Load Balancer

    • Layer 4
      • Supports UDP and TCP
    • High performance
    • One static IP per availability zone

  • Gateway Load Balancer

    • Layer 3
    • Used for 3rd party network appliances on AWS, example: Firewalls
    • Extremely high performance
    • Supports the GENEVE protocol

  • Target Groups

    • EC2 Instances
    • ECS Tasks
    • Lambda Functions
    • Private IP addresses

  • Sticky Sessions

    • Cookie Names

      • Application-based cookies
        • Custom Cookie
          • Generated by the target
          • Can include any custom attributes required by the application
          • Cookie name must be specified individually for each target group
          • You cannot use AWSALB, AWSALBAPP, or AWSALBTG (these are reserved by the ELB)
        • Application Cookie
          • Generated by the load balancer
          • Cookie name is AWSALBAPP
      • Duration-based Cookie
        • Cookie generated by the load balancer
        • Cookie name is AWSALB for ALB, AWSELB for CLB

  • Cross-zone load balancing

    • Each load balancer instance distributes traffic evenly across all registered instances in all availability zones

Autoscaling Groups

• Scale out EC2 instances to match increased load or scale in to match a decreased load
• Specify parameters to have a minimum and maximum number of instances
• Automatically replace failed instances
• Uses a launch template

IMDS

• IMDSv1 vs. IMDSv2
  • IMDSv1 is accessing http://169.254.169.254/latest/meta-data directly
  • IMDSv2 is more secure and is done in two steps
    1. get a Session token
    2. Use session token in the IMDSv2 calls

Elasticache

Introduction

• Fully managed Redis or Memcached instances
• Caches are in-memory databases with high-performance and low latency
• Helps to reduce load from databases
• Helps to make your application stateless
• Requires that your application be architected with a cache in-mind

Redis

• Supports multi-AZ with auto-failover
• Supports read-replicas to scale out reads
• Backup and restore features
• Supports sets and sorted sets

Memcached

• Multi-node for partitioning of data (Sharding)
• None of the features that Redis supports

Caching Implementation Considerations

• Is it safe to cache the data?
  • Data may be out of date (eventually consistent)
• Is caching effective for that data?
  • Patterns: data changing slowly, few keys are frequently updated
  • Anti patterns: data changing rapidly, all large key space frequently needed
• Is data structured for caching?
  • example: key value caching or caching of aggregations result

Caching design patterns

Lazy Loading / Cache-Aside / Lazy Population

Write-through

Cache Evictions and TTL

• Cache eviction can occur in 3 ways:
  • You delete the item in the cache
  • Item is evicted because the memory is full and its not in use (LRU)
  • The TTL (time to live) has expired
    • TTL can range from a few seconds to days

IAM

Table of Contents

• Introduction
• Users and Groups
• IAM Policies
• Password Policies
• MFA
• IAM Roles
• IAM Security Tools

Introduction

• IAM is Identity and Access Management
• IAM is a global service
• Do not use the root account, create user accounts instead

Users and Groups

• Groups can only contain users, not other groups
• Users do not need to belong to a group. Users can belong to multiple groups

IAM Policies

• Users and groups can be assigned a policy called an IAM policy. IAM policies are JSON documents:

      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Sid": "EnableDisableHongKong",
                  "Effect": "Allow",
                  "Action": [
                      "account:EnableRegion",
                      "account:DisableRegion"
                  ],
                  "Resource": "*",
                  "Condition": {
                      "StringEquals": {"account:TargetRegion": "ap-east-1"}
                  }
              },
              {
                  "Sid": "ViewConsole",
                  "Effect": "Allow",
                  "Action": [
                      "account:ListRegions"
                  ],
                  "Resource": "*"
              }
          ]
      }
  

• IAM policy inheritance

  • inline policies are attached directly to users
  • If an IAM policy is attached to a group, any users in that group will inherit settings from the policy
  

Password Policy

MFA (Multi-factor authentication)

• Virtual MFA device
  • Google Authenticator
  • Authy
• Universal 2nd Factor Security Key
  • Ubikey
• Hardware Key Fob
  • Also has a special option for GovCloud

IAM Roles

• Allows AWS service to perform actions on your behalf. When creating the role, you choose which service the role will apply to (For example, EC2)
• Assign permissions to AWS services with IAM roles

IAM Security Tools

• IAM Credentials Report
  • A report that lists all user accounts and status of their credentials
• IAM Access Advisor
  • Access advisor shows the service permissions granted to a user and when those services were last accessed

Kinesis

Introduction

• Kinesis is a set of services provided by AWS
  • Kinesis Data Streams: capture, process, and store data streams
  • Kinesis Data Firehose: load data streams into AWS data stores
  • Kinesis Data Analytics: analyze data streams with SQL or Apache Flink
  • Kinesis Video Streams: Capture, process and store video streams

Kinesis Data Firehose

• Records up to 1 MB can be sent to Kinesis Data Firehose and Firehose will then batch writees to other resources in near real-time
• Fully managed by AWS, autoscales
• Pay only for the data going through Firehose
• Producers such as (Applications, Kinesis Agent, Kinesis Data Streams, CloudWatch, AWS IoT) can write to Firehose, and Firehose will then send the data to S3, RedShift, or OpenSearch
• Data Firehose can also send to 3rd parties such as Splunk, Datadog, etc.
• You can transform data using Lambda functions before sending it to the destination

Kinesis Streams

Collect and process large streams of data in real-time.

Use Cases:

• Fast (second/millisecond latency) processing of log events
• Real-time metrics and reporting
• Data analytics
• Complex stream processing

Kinesis Libraries / Tools:

Producing Data:

• Kinesis Producer Library (KPL)

  • Blog post: Implementing Efficient and Reliable Producers with the Amazon Kinesis Producer Library
  • Auto-retry configurable mechanism
  • Supports two complementary ways of batching:
    • Collection (of stream records):
      • Buffers/collects records to write multiple records to multiple shards in a single request.
      • RecordMaxBufferedTime: max time a record may be buffered before a request is sent. Larger = more throughput but higher latency.
    • Aggregation (of user records):
      • Combines multiple user records into a single Kinesis stream record (using PutRecords API request).
      • KCL integration (for deaggregating user records).

• Kinesis Agent

  • Standalone application that you can install on the servers you’re interested in.
  • Features:
    • Monitors file patterns and sends new data records to delivery streams
    • Handles file rotation, checkpointing, and retry upon failure
    • Delivers all data in a reliable, timely, and simpler manner
    • Emits CloudWatch metrics for monitoring and troubleshooting
    • Allows preprocessing data, e.g., converting multi-line record to single line, converting from delimiter to JSON, converting from log to JSON.

Kinesis Streams API:

Reading Data:

• Kinesis Client Library (KCL)
  • The KCL ensures there is a record processor running and processing each shard.
  • Uses a DynamoDB table to store control data. It creates one table per application that is processing data.
  • Creates a worker thread for each shard. Auto-assigns shards to workers (even workers on different EC2 instances).
  • KCL Checkpointing
    • Last processed record sequence number is stored in DynamoDB.
    • On worker failure, KCL restarts from last checkpointed record.
    • Supports deaggregation of records aggregated with KPL.
    • Note: KCL may be bottlenecked by DynamoDB table (throwing Provisioned Throughput Exceptions). Add more provisioned throughput to the DynamoDB table if needed.

Emitting Data:

• Kinesis Connector Library (Java) for KCL
  • Connectors for: DynamoDB, Redshift, S3, Elasticsearch.
  • Java library with the following steps/interfaces:
    • iTransformer: maps from stream records to user-defined data model.
    • iFilter: removes irrelevant records.
    • iBuffer: buffers based on size limit and total byte count.
    • iEmitter: sends the data in the buffer to AWS services.
    • S3Emitter: writes buffer to a single file.

Kinesis Stream API:

• PutRecord (single record per HTTP request)
• PutRecords (multiple records per single HTTP request). Recommended for higher throughput.
  • Single record failure does not stop the processing of subsequent records.
  • Will return HTTP 200 as long as some records succeed (even when others failed).
  • Retry requires application code in the producer to examine the PutRecordsResult object and retry whichever records failed.

Key Concepts

Kinesis Data Streams:

• Stream big data into AWS

• Kinesis Data Streams

• A stream is a set of shards. Each shard is a sequence of data records.

  • Shards are numbered (shard1, shard2, etc.)

• Each data record has a sequence number that is assigned automatically by the stream.

• A data record has 3 parts:

  • Sequence number
  • Partition key
  • Data blob (immutable sequence of bytes, up to 1000KB).

• Sequence number is only unique within its shard.

• Retention Period:

  • Retention for messages within a Data Stream can be set to 1 - 365 days
  • You pay more for longer retention periods.

• Consumers and Producers

  • producers send data (records) into data streams
    • records consist of a partition key and a data blob
    • producers can send 1MB/sec or 1000 msg/sec per shard
  • Consumers receive data from data streams
    • consumers can be apps, lambda functions, Kinesis Data Firehose, or Kinesis Data Analytics
    • Consumers can receive messages at 2 MB/sec (shared version, across all consumers) per shard or 2 MB/sec (enhanced version) per consumer per shard

• Once data is inserted into Kinesis, it cannot be deleted

• Capacity Modes:

  • Provisioned Mode
    • Choose the number of shards provisioner
    • Scale manually
    • Each shard gets 1 MB/s in and 2 MB/s out
    • Pay per shard provisioned per hour
  • On-demand Mode:
    • No need to provision or manage capacity
    • Auto-scaling
    • Pay per stream per hour and data in/out per GB

• Access control to Data Streams using IAM policies

• Encryption in flight with HTTPS and at rest with KMS

• Kinesis Data Streams support VPC Endpoints

• Monitor API calls using CloudTrail

Kinesis Application Name:

• Each application must have a unique name per (AWS account, region). The name is used to identify the DynamoDB and the namespace for CloudWatch metrics.

Partition Keys:

• Used to group data by shard within a stream. It must be present when writing to the stream.
• When writing to a stream, Kinesis separates data records into multiple shards based on each record’s partition key.
• Partition keys are Unicode strings with a maximum length of 256 bytes. An MD5 hash function is used to map partition keys to 128-bit integer values that define which shard records will end up in.

Kinesis Shard:

• Uniquely identified group of data records in a stream.
• Multiple shards in a stream are possible.
• Single shard capacity:
  • Write: 1 MB/sec input, 1000 writes/sec.
  • Read: 2 MB/sec output, 5 transaction reads/sec.
• Resharding:
  • Shard split: divide a shard into two shards.
    • Example using boto3:

      sinfo = kinesis.describe_stream("BotoDemo")
      hkey = int(sinfo["StreamDescription"]["Shards"][0]["HashKeyRange"]["EndingHashKey"])
      shard_id = 'shardId-000000000000'  # we only have one shard!
      kinesis.split_shard("BotoDemo", shard_id, str((hkey+0)/2))
      

  • Shard merge: merge two shards into one.

Kinesis Server-Side Encryption:

• Can automatically encrypt data written, using KMS master keys. Both producer and consumer must have permission to access the master key.
• Add kms:GenerateDataKey to producer’s role.
• Add kms:Decrypt to consumer’s role.

Kinesis Firehose:

• Managed service for loading data from streams directly into S3, Redshift, and Elasticsearch.
• Fully managed: scalability, sharding, and monitoring with zero admin.
• Secure.
• Methods to Load Data:
  • Use Kinesis Agent.
  • Use AWS SDK.
  • PutRecord and PutRecordBatch.
  • Firehose to S3:
    • Buffering of data before sending to S3. Sends whenever any of these conditions is met:
      • Buffer size (from 1 MB to 128 MB).
      • Buffer interval (from 60s to 900s).
    • Can invoke AWS Lambda for data transformation.
      • Data flow:
        1. Buffers incoming data up to 3 MB or buffering size specified, whichever is lowest.
        2. Firehose invokes Lambda function.
        3. Transformed data is sent from Lambda to Firehose for buffering.
        4. Transformed data is delivered to the destination.
      • Response from Lambda must include:
        • recordId: must be the same as prior to transformation.
        • result: status, one of: “Ok”, “Dropped”, “ProcessingFailed”.
        • data: Transformed data payload.
      • Failure handling of data transformation:
        • 3 retries.
        • Invocation errors logged in CloudWatch Logs.
        • Unsuccessful records are stored in processing_failed folder in S3.
        • It’s possible to store the source records in S3, prior to transformation (Backup S3 bucket).
• Data Delivery Speed:
  • S3: based on buffer size/buffer interval.
  • Redshift: depending on how fast the Redshift cluster finishes the COPY command.
  • Elasticsearch: depends on buffer size (1-100 MB) and buffer interval.
• Firehose Failure Handling:
  • S3: retries for up to 24 hrs.
  • Redshift: retry duration from 0-7200 sec (2 hrs) from S3.
    • Skips S3 objects on failure.
    • Writes failed objects in manifest file, which can be used manually to recover lost data (manual backfill).
  • ElasticSearch: retry duration 0-7200 sec.
    • On failure, skips index request and stores in index_failed folder in S3.
    • Manual backfill.

AWS Kinesis Overview:

• Enables real-time processing of streaming data at massive scale.
• Kinesis Streams:
  • Enables building custom applications that process or analyze streaming data for specialized needs.
  • Handles provisioning, deployment, ongoing-maintenance of hardware, software, or other services for the data streams.
  • Manages the infrastructure, storage, networking, and configuration needed to stream the data at the required data throughput level.
  • Synchronously replicates data across three facilities in an AWS Region, providing high availability and data durability.
  • Stores records of a stream for up to 24 hours, by default, from the time they are added to the stream. The limit can be raised to up to 7 days by enabling extended data retention.
  • Data such as clickstreams, application logs, social media, etc., can be added from multiple sources and within seconds is available for processing to the Amazon Kinesis Applications.
  • Provides ordering of records, as well as the ability to read and/or replay records in the same order to multiple Kinesis applications.
  • Useful for rapidly moving data off data producers and then continuously processing the data, whether it is to transform the data before emitting to a data store, run real-time metrics and analytics, or derive more complex data streams for further processing.

Use Cases:

• Accelerated log and data feed intake: Data producers can push data to Kinesis stream as soon as it is produced, preventing any data loss and making it available for processing within seconds.
• Real-time metrics and reporting: Metrics can be extracted and used to generate reports from data in real-time.
• Real-time data analytics: Run real-time streaming data analytics.
• Complex stream processing: Create Directed Acyclic Graphs (DAGs) of Kinesis Applications and data streams, with Kinesis applications adding to another Amazon Kinesis stream for further processing, enabling successive stages of stream processing.

Kinesis Limits:

• Stores records of a stream for up to 24 hours, by default, which can be extended to max 7 days.
• Maximum size of a data blob (the data payload before Base64-encoding) within one record is 1 megabyte (MB).
• Each shard can support up to 1000 PUT records per second.
• Each account can provision 10 shards per region, which can be increased further through request.
• Amazon Kinesis is designed to process streaming big data and the pricing model allows heavy PUTs rate.
• Amazon S3 is a cost-effective way to store your data but not designed to handle a stream of data in real-time.

Kinesis Streams Components:

Shard:

• Streams are made of shards and is the base throughput unit of a Kinesis stream.
• Each shard provides a capacity of 1MB/sec data input and 2MB/sec data output.
• Each shard can support up to 1000 PUT records per second.
• All data is stored for 24 hours.
• Replay data inside a 24-hour window.
• Capacity Limits: If the limits are exceeded, either by data throughput or the number of PUT records, the put data call will be rejected with a ProvisionedThroughputExceeded exception.
  • This can be handled by:
    • Implementing a retry on the data producer side, if this is due to a temporary rise of the stream’s input data rate.
    • Dynamically scaling the number of shared (resharding) to provide enough capacity for the put data calls to consistently succeed.

Record:

• A record is the unit of data stored in an Amazon Kinesis stream.
• A record is composed of a sequence number, partition key, and data blob.
  • Data blob is the data of interest your data producer adds to a stream.
  • Maximum size of a data blob (the data payload before Base64-encoding) is 1 MB.

Partition Key:

• Used to segregate and route records to different shards of a stream.
• Specified by your data producer while adding data to an Amazon Kinesis stream.

Sequence Number:

• A unique identifier for each record.
• Assigned by Amazon Kinesis when a data producer calls PutRecord or PutRecords operation to add data to an Amazon Kinesis stream.
• Sequence numbers for the same partition key generally increase over time; the longer the time period between PutRecord or PutRecords requests, the larger the sequence numbers become.

Data Producers:

• Data can be added to an Amazon Kinesis stream via PutRecord and PutRecords operations, Kinesis Producer Library (KPL), or Kinesis Agent.

Amazon Kinesis Agent:

• A pre-built Java application that offers an easy way to collect and send data to Amazon Kinesis stream.
• Can be installed on Linux-based server environments such as web servers, log servers, and database servers.
• Configured to monitor certain files on the disk and then continuously send new data to the Amazon Kinesis stream.

Amazon Kinesis Producer Library (KPL):

• An easy to use and highly configurable library that helps you put data into an Amazon Kinesis stream.
• Presents a simple, asynchronous, and reliable interface that enables you to quickly achieve high producer throughput with minimal client resources.

Amazon Kinesis Application:

• A data consumer that reads and processes data from an Amazon Kinesis stream.
• Can be built using either Amazon Kinesis API or Amazon Kinesis Client Library (KCL).

Amazon Kinesis Client Library (KCL):

• A pre-built library with multiple language support.
• Delivers all records for a given partition key to the same record processor.
• Makes it easier to build multiple applications reading from the same Kinesis stream (e.g., to perform counting, aggregation, and filtering).
• Handles complex issues such as adapting to changes in stream volume, load-balancing streaming data, coordinating distributed services, and processing data with fault-tolerance.

Amazon Kinesis Connector Library:

• A pre-built library that helps you easily integrate Amazon Kinesis Streams with other AWS services and third-party tools.
• Kinesis Client Library is required for Kinesis Connector Library.

Amazon Kinesis Storm Spout:

• A pre-built library that helps you easily integrate Amazon Kinesis Streams with Apache Storm.

Kinesis vs SQS:

• Kinesis Streams enables real-time processing of streaming big data while SQS offers a reliable, highly scalable hosted queue for storing messages and moving data between distributed application components.
• Kinesis provides ordering of records, as well as the ability to read and/or replay records in the same order to multiple Amazon Kinesis Applications, while SQS does not guarantee data ordering and provides at least once delivery of messages.
• Kinesis stores the data up to 24 hours, by default, and can be extended to 7 days, while SQS stores the message up to 4 days, by default, and can be configured from 1 minute to 14 days but clears the message once deleted by the consumer.
• Kinesis and SQS both guarantee at-least-once delivery of messages.
• Kinesis supports multiple consumers, while SQS allows the messages to be delivered to only one consumer at a time and requires multiple queues to deliver messages to multiple consumers.

Kinesis Use Case Requirements:

• Ordering of records.
• Ability to consume records in the same order a few hours later.
• Ability for multiple applications to consume the same stream concurrently.
• Routing related records to the same record processor (as in streaming MapReduce).

SQS Use Case Requirements:

• Messaging semantics like message-level ack/fail and visibility timeout.
• Leveraging SQS’s ability to scale transparently.
• Dynamically increasing concurrency/throughput at read time.
• Individual message delay, which can be delayed.

Lambda

Introduction

• Serverless, virtual functions
• Short executions up to 15 minutes
• Run on-demand
• Pay for number of invocations and compute time
• Works with many programming languages
  • Node.js, python, Java, c#, Go, Powershell, Ruby, and Custom Runtime API (which can run practically any language)
• You can provision up to 10GB of RAM per function

Lambda Integrations

• API Gateway
• Kinesis
• DynamoDB
• S3
• CloudFront
• CloudWatch Events / EventBridge
• CloudWatch Logs
• SNS
• SQS
• Cognito

Pricing

• Pay per call:
  • First 1,000,000 requests are free
  • .20 per 1 million requests after the first million
• Pay per duration
  • 400,000 GB-seconds of compute time per month for free

Synchronous Invocation

• When invoking the function from the CLI, SDK, API Gateway, or ALB, the call is synchronous, meaning the result is returned right away
• Error handling must happen on the client side (retires, exponential backoff, etc.)

Asynchronous Invocation

• S3, SNS, CloudWatch Events are all processed asynchronously
• The events are placed in an internal event queue
• The lambda function will read from the event queue and attempt to process the events
• Lambda will attempt to retry failures up to 3 times
  • This means that event may be processed multiple times, so make sure the lambda function is idempotent
  • If the function is retried, you will see duplicate entries in CloudWatch Logs
  • You can define a DLQ (dead-letter queue) (SNS or SQS) for failed processing
• Async invocations allow you to speed up the processing if you don’t need to wait for the result

S3 Event Notifications

• Run a Lambda function when a event in S3 is detected

Lambda Event Source Mapping

• Lambda will poll from the sources and be invoked synchronously
  • Kinesis Data Streams
  • SQS or SQS FIFO
  • DynamoDB Streams
• Two categories of Event Source Mapping:
  • Streams
    • Kinesis or DynamoDB Streams
    • One Lambda invokation per stream shard
    • If you use parallelization, up to 10 batches processed per shard simultaneously
  • Queues
    • Poll SQS using Long Polling

Lambda in VPC

• By default, Lambda functions are launched outside of your VPC. Therefore, it cannot access resources in your VPC.
• Lambda can create an Elastic Network Interface inside your VPC
  • You must define the VPC ID, subnets, and security groups
  • Lambda requires the AWSLambdaVPCAccessExecutionRole
• By default, a Lambda function in your VPC does not have internet access
  • Deploying a Lambda function in a public subnet does not give it internet access
  • Instead, you can deploy the Lambda function in a private subnet and give it internet access via a NAT Gateway / NAT Instance

Lambda Concurrency

• Concurrency limit up to 1000 concurrent executions
• each invocation over the concurrency limit will respond with a HTTP 429
• Cold starts and provisioned concurrency
  • If the init is large, cold start could take a long time. This may cause the first request to have high latency than the rest
  • To resolve the cold start issue, you can use Provisioned concurrency
    • With Provisioned Concurrency, concurrency is allocated before the function is invoked

Lambda Containers

• Deploy Lambda functions as container images up to 10GB from ECR

CloudTrail

Introduction

• Internal monitors of API calls being made
• Audit changes to AWS resources
• Enabled by default
• Event Types:
  • Management Events
  • Data Events
  • CloudTrail Insights Events
    • analyze events and try to detect unusual activity in your account
• Event Retention
  • Events are stored by default for 90 days
  • To keep events beyond this period, log them to S3 and use Athena to analyze them

CloudWatch

Introduction

• Metrics, Logs, Events, and Alarms

CloudWatch Metrics

• CloudWatch provides metrics for every service in AWS
• Metric is a variable to monitor (CPU Utilization, Network In, etc.)
• Metrics belong to namespaces
• Dimension is an attribute of a metric (instance id, environment, etc.)
• Up to 30 dimensions per metric
• Metrics have timestamp
• You can create dashboards of metrics

EC2 Detailed Monitoring

• By default, EC2 instance have metrics every 5 minutes
• If you enable detailed monitoring, you can get metrics every 1 minute``
• Use detailed monitoring if you want your ASG to scale faster
• The AWS Free tier allows us to have 10 detailed monitoring metrics
• EC2 memory usage is not pushed by default (you must push it from inside the instance as a custom metric)

CloudWatch Custom Metrics

• You can define your own custom metrics
• Use an API call PutMetricData

CloudWatch Logs

• Define log groups, usually representing an application
• Log Stream: instances within application /log files/ containers
• You can define log expiration policies
• You can send CloudWatch logs to
  • S3
  • Kinesis Data Streams
  • Kinesis Data Firehose
  • AWS Lambda
  • OpenSearch
• Logs are encrypted by default

Log Sources

• SDK, CloudWatch Logs Agent, CloudWatch Unified Agent

• BeanStalk: Collection of logs from the application

• ECS: Collection from containers

• AWS Lambda: collection from function logs

• VPC Flow Log’s

• API Gateway

• CloudTrail based on a filter

• Route53

• Use CloudWatch Logs Insights to query logs

CloudWatch Logs Subscriptions

• Get a real-time log events from CloudWatch Logs for processing and analysis
• Send to Kinesis Data Streams, Kinesis Data Firehose, or Lambda
• Subscription Filter - filter which logs are events delivered to your destination

CloudWatch Alarms

• Trigger notifications from any metric
• Alarm States
  • Ok
  • Insufficient Data
  • Alarm
• Targets
  • Actions on EC2 instances
  • Trigger autoscaling action
  • Send notification to SNS service
• Composite Alarms monitor the state of multiple other alarms
  • AND and OR conditions

CloudWatch Synthetics Canary

• Configurable script that can monitor your APIs, URLs, Websites, etc.
• Reproduce what your customers do programmatically to find issues before customers are impacted
• Blueprints
  • Heartbeat Monitor
  • API Canary
  • Broken Link Checker
  • Visual Monitoring
  • Canary Recorder
  • GUI Workflow Builder

Amazon Event Bridge

• React to events. Examples:

  • EC2 Instance started
  • Codebuild failed build
  • S3 upload object
  • schedule a cronjob
  • CloudTrail API call

• Event Buses can be accessed across AWS accounts using Resource-Based Policies
  • Resource policies allow you to manage permissions for an EventBus

x-ray

Introduction

• Troubleshooting application performance and errors
• Distributed tracing of Micro-services
• Compatible with
  • Lambda
  • Beanstalk
  • ECS
  • ELB
  • API Gateway
  • EC2 instances or any on-premises app server
• You can enable x-ray by:
  • Install the x-ray daemon (on a server) or enable x-ray integration (some AWS services such as lambda)
  • You can instrument x-ray in your code using the AWS SDK
    • python, java, go, .NET, node.js

X-Ray APIs

• Writes
  • PutTraceSegments - Uploads a segment document int x-ray
  • PutTelemetryRecords - Used by the AWS X-Ray daemon o upload telemetry
  • GetSamplingRules - Retrieve all sampling rules
• Reads
  • GetServiceGraph - main graph
  • BatchGetTraces - Retries a list of traces specified by Id
  • GetTraceSummaries - Retrieve Ids and annotations for traces available for a specified time frame using an optional filter
  • GetTraceGraph

X-Ray with Beanstalk

• Beanstalk includes the x-ray daemon
• You can run the daemon by setting an option in the Elastic Beanstalk console or with a configuration file (in .ebextensions/xray-daemon.confi)

  option_settings:
    aws:elasticbeanstalk:xray:
      XRayEnabled: true
  

• Make sure to give your instance profile the correct IAM permissions so that the x-ray daemon can function correctly
• You app code must still be instrumented with the X-Ray integration code

ECS + X-Ray

• Pattens:
  • Run the X-Ray daemon container on every EC2 instance
  • Run the X-Ray container as a sidecar for the app containers (the only way to get ECS with Fargate working with X-Ray)

Aurora

Introduction

• AWS propriety database technology compatible with Postgres and Mysql
• 5x performance improvement over MySql, 3x performance improvement over Postgres
• Storage automatically grows, starts at 10GB, grows up to 128 TB
• Up to 15 read replicas, replication process is faster than MySql
• Failover is instantaneous, HA is native to Aurora
• About 25% more expensive than RDS

Aurora High Availability

• 6 copies of your data across 3 AZ:
  • 4 copies out of 6 need to be available for writes
  • 3 copies out of 6 need to be available for reads
• Self-healing with peer-to-peer replication
• Stored is striped across 100 volumes
• Only one instance will take writes at a time, failover within 30 seconds
  • You can optionally enable “Local Write Forwarding” to forward writes from a read replica to a write replica
• Supports cross region replication
• Reader endpoint is load balanced across all read replicas. Writer endpoint points to the current writer instance

Aurora Security

• If the master is not encrypted, read replicas cannot be encrypted
• To encrypt an unencrypted database, create a db snapshot and restore as encrypted

RDS (Relational Database Service)

Introduction

• Fully managed relational database service
  • Automated provisioning and maintenance
  • Full backups and point in time restore
• Monitoring dashboards
• Multi AZ setup for DR
• Read replicas for improved performance
• Storage backed by EBS
• Vertical and horizontal scaling
• Supports
  • Postgres
  • MySql / MariaDB
  • Oracle
  • IBM DB2
  • MSSQL
  • Aurora (Postgres and MySQL)
• You cannot SSH or access the underlying database compute instance
• Storage auto-scaling

Read replicas

• Scale out database reads
  • read replicas only support select statements
• Improves performance for reads
• You can create up to 15 read replicas in the same AZ, cross AZ, or cross region
• Asynchronous replication, reads are eventually consistent
• You can promote a read replica to a full read/write instance
• Apps must update the connection string to use a read replica
• Network costs
  • If the replica is in the same region, there is no fee for network traffic
  • Cross region replicas will cost you for replication traffic

RDS Proxy

• Fully managed database proxy for RDS
• Allow apps to pool and share DB connections established with the database
• Improves database efficiency by reducing stress on the database resources by pooling connections at the proxy
• Serverless, autoscaling, HA
• Reduced RDS and Aurora failover time by up to 66%
• RDS proxy is only accessible from within the VPC, it is never publicly accessible
• RDS proxy is particularly helpful when you have auto-scaling lambda functions connecting to your database

Route53

Introduction

• A highly available, fully managed, scalable, authoritative DNS service provided by Amazon
• Also a domain registrar
• Supports health checks for resources registered with DNS names
• The only AWS service that provides 100% availability

Hosted Zones

• Public Hosted Zones
• contains records that specify how to route traffic on the internet
• Private Hosted Zones
• Only hosts within the VPC can resolve the DNS names
• You will pay 50 cents per month for each hosted zone
• Domain names will cost you $12/year

TTL

• Time to live
• i.e. how long a DNS record will be cached on a client machine

CNAME vs Alias

• lb l-1234.us-east-2.elb.amazonaws.com and you want myapp.mydomain.com
• CNAME:
• Points a hostname to any other hostname (app.domain.com => blabla.anything.com)
• You cannot create a CNAME for the Apex record (root domain)
• Alias:
• Points a hostname to an AWS Resource (app.mydomain.com => blabla.amazonaws.com)
• WORKS for ROOT DOMAIN and NON ROOT DOMAIN (aka, mydomain.com)
  • Free of charge
  • Native health check
  • Only supported for A and AAAA record types
  • Cannot set alias for an EC2 instance name

Routing Policies

• Simple

  • Typically, the simple type of routing policy will resolve to a single resource
  • If the record resolves to multiple values, the client will choose a random one
  • When using the Alias record type, the record can only resolve to one resource

• Weighted

  • Control the % of the requests that go to each specific resource.
  • Assign each record a relative weight
    • $ \text traffic {(%)} = {\displaystyle \text {weight for a specific record } \over \displaystyle \text {sum of all the weights for all records }} $
    • The sum of the weights of all records does not need to equal 100
  • DNS records must have the same name and type
  • Can be associated with Health Checks
  • Use cases: load balancing between regions, testing new application versions
    

• Latency

  • Redirect to the resource that has the least latency close to us
  • Super helpful when latency for users is a priority
  • Latency is based on traffic between users and AWS Regions
  • Germany users may be directed to the US (if that’s the lowest latency)
  • Can be associated with Health Checks (has a failover capability)

• Failover

• Geolocation

  • Different from latency based
  • This routing is based on user location
  • Should create a “Default” record (in case there’s no match on location)
  • Use cases: website localization, restrict content distribution, load balancing
  • Can be associated with Health Checks

• Geoproximity

  • Route traffic to your resources based on the location of users and resources
  • Ability to shift more traffic to resources based on the defined bias
  • To change the size of the geographic region, specify bias values:
    • To expand (1 to 99)- more traffic to the resource
    • To shrink (-1 to 99)- less traffic to the resource
  • Resources can be:
    • AWS resources (specify AWS region)
    • Non-AWS resources (specify Latitude and Longitude)
  • You must use Route 53 Traffic Flow to use this feature

• Health Checks

  • HTTP Health Checks are only for public resources. You must create a CloudWatch Metric and associate a CloudWatch Alarm, then create a Health Check that checks the alarm
  • 15 global health checkers
  • Health checks methods:
    • Monitor an endpoint
      • Healthy/unhealthy threshold - 3 (default)
      • Interval 30 seconds
      • Supports HTTP, HTTPS, and TCP
      • if > 18% of health checkers report the endpoint is healthy, Route53 considers it healthy.
      • You can choose which locations you want Route53 to use
      • You must configure the firewall to allow traffic from the health checkers
    • Calculated Health Checks
      • Combine the results of multiple health checks into a single health check

Configuring Amazon Route 53 to route traffic to an S3 Bucket

• An S3 bucket that is configured to host a static website
  • You can route traffic for a domain and its subdomains, such as example.com and www.example.com to a single bucket.
  • Choose the bucket that has the same name that you specified for Record name
  • The name of the bucket is the same as the name of the record that you are creating
  • The bucket is configured as a website endpoint

S3

Introduction

• Storage
• Files are stored in Buckets, the files are called objects
• Storage Accounts must have a globally unique DNS name
• Buckets are regional
• Bucket names must have no uppercase, no underscore, 3-63 characters long, not an IP address, must start with a lowercase letter or number
• Objects (files) have a key, which is the FULL path of the object:
  • Example of a prefix
    • bucket/folder1/subfolder1/mypic.jpg => prefix is /folder1/subfolder1/
• S3 Select
  • Use SQL like language to only retrieve the data you need from S3 using server-side filtering
• Max object size is 5TB
• If you upload a file larger than 5GB, you must use Multi-part Upload
• Objects can have metadata

S3 Security

• User-Based

  • IAM Policies - Which API calls are allowed for an IAM user

• Resource-Based

  • Bucket Policies- bucket wide rules form the S3 Console - allows cross account
  • Object ACL - Finer grained (can be disabled)
  • Bucket ACL - less common (can be disabled)

• An IAM Principal can access an S3 object if:

  • The user IAM permissions ALLOW it OR the resource policy allows it and there is no explicit Deny

• Bucket Policies - Bucket wide rules from the S3 console

  • JSON based policy

      {
          "Version": "2012-10-17",
          "Statement": [{
              "Sid": "AllowGetObject",
              "Principal": {
                  "AWS": "*"
              },
              "Effect": "Allow",
              "Action": "s3:GetObject",
              "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",
              "Condition": {
                  "StringEquals": {
                      "aws:PrincipalOrgID": ["o-aa111bb222"]
                  }
              }
          }]
      }
    

  • You can use the AWS Policy Generator to create JSON policies

S3 Static Website Hosting

• You must enable public reads on the bucket

S3 Versioning

• allows to version the object

• Stores all versions of an object in S3

• Once enabled it cannot be disabled, only suspended on the bucket

  

• Fully integrates with S3 Lifecycle rules

• MFA Delete feature provides extra protection against deletion of your data

  

S3 Cross-Region Replication or Same-Region Replication

• When enabled, any object that is uploaded will be Automatically replicate to another region or from source to destination buckets

  

• Must have versioning turned on both the source and destination buckets.

• Can have CRR replicate to another AWS account

• Replicate objects within the same region

• You must give proper IAM permissions to S3

• Buckets can be in different AWS accounts

• Only new objects are replicated after enabling replication. To replicate existing objects, you must use S3 Batch Replication

• For DELETE operations, you can optionally replicate delete markers. Delete Markers are not replicated by default.

• To replicate, you create a replication rule in the “Management” tab of the S3 bucket. You can choose to replicate all objects in the bucket, or create a rule scope

S3 Storage Classes

• AWS offers a range of S3 Storage classes that trade Retrieval, Time, Accessability and Durability for Cheaper Storage

(Descending from expensive to cheaper)

• S3 Standard (default)

  • Fast! 99.99 % Availability,
  • 11 9’s Durability. If you store 10,000,000 objects on S3, you can expect to lose a single object once every 10,000 years
  • Replicated across at least three AZs
    • S3 standard can sustain 2 concurrent facility failures

• S3 Intelligent Tiering

  • Uses ML to analyze object usage and determine the appropriate storage class
  • Data is moved to most cost-effective tier without any performance impact or added overhead

• S3 Standard-IA (Infrequent Access)

  • Still Fast! Cheaper if you access files less than once a month
  • Additional retrieval fee is applied. 50% less than standard (reduced availability)
  • 99.9% Availability

• S3 One-Zone-IA

  • Still fast! Objects only exist in one AZ.
  • Availability (is 99.5%). but cheaper than Standard IA by 20% less
  • reduces durability
  • Data could be destroyed
  • Retrieval fee is applied

• S3 Glacier Instant Retrieval

  • Millisecond retrieval, great for data accessed once a quarter
  • Minimum storage duration of 90 days

• S3 Glacier Flexible Retrieval

  • data retrieval: Expedited (1 to 5 minutes), Standard (3 to 5 hours), Bulk (5 to 12 hours) - free
  • minimum storage duration is 90 days
  • Retrieval of data can take minutes to hours but the off is very cheap storage

• S3 Glacier Deep Archive

  • The lowest cost storage class - Data retrieval time is 12 hours
  • standard (12 hours), bulk (48 hours)
  • Minimum storage duration is 180 days

• S3 Glacier Intelligent Tiering

• Storage class comparison

  

• S3 Guarantees:

  • Platform is built for 99.99% availability
  • Amazon guarantee 99.99% availability
  • Amazon guarantees 11’9s of durability

S3 LifeCycle Rules

• Types of rules:
  • Transition Actions
    • Move objects between storage classes automatically
  • Expiration Actions
    • Configure objects to expire (delete) after some time
    • Can be used to delete incomplete multi-part uploads
    • Delete access logs automatically
    • Can be used to delete old versions of files if versioning is enabled
• Rules can be specified for objects with a certain prefix or tag

Event Notifications

• Examples of events:
  • S3:ObjectCreated, S3:ObjectRemoved, S3:ObjectRestore
• Object name filtering is possible (*.jpg for example)
• Send a notification when an event occurs
• Uses SNS, Lambda, or SQS to send the notifications to
  • Requires a SNS Resource Policy, SQS Resource Policy, or a Lambda Resource Policy allowing S3 bucket to write to the resource
• You can also send events to EventBridge, which can then be used to send the events to 18 other AWS services

S3 Encryption

• 4 types of encryption in S3
  • Server side encryption with managed keys (SSE-S3)
    • Key is completely managed by AWS, you never see it
    • Object is encrypted server-side
    • Enabled by default
      • Uses AES-256, must set header "x-amz-server-side-encryption": "AES256"
  • Server side encryption with KMS keys stored in AWS KMS (SSE-KMS)
    • Manage the key yourself, store the key in KMS
    • You can audit the key use in CloudTrail
      • Uses AES-256, must set header "x-amz-server-side-encryption": "AWS:KMS"
    • Accessing the key counts toward your KMS Requests quota (5500, 10000, 30000 rps, based on region)
      • You can request a quota increase from AWS
  • Server Side Encryption with customer provided keys (SSE-C)
    • Can only be enabled/disabled from the AWS CLI
    • AWS doesn’t store the encryption key you provide
    • The key must be passed as part of the headers with every request you make
    • HTTPS must be used
  • CSE (Client side encryption)
    • Clients encrypt/decrypt all the data before sending any data to S3
    • Customer fully managed the keys and encryption lifecycle
• Encryption in Transit
  • Traffic between local host and S3 is achieved via SSL/TLS

MFA Delete

• MFA Delete ensures users cannot delete objects from a bucket unless they provide their MFA code.
• MFA delete can only be enabled under these conditions
  1. The AWS CLI must be used to turn on MFA delete
  2. The bucket must have versioning enabled
• Only the bucket owner logged in as Root User can DELETE objects from bucket

Presigned URLs

• Generates a URL which provides temporary access to an object to either upload or download object data.

• The pre-signed URL inherites the permission of the user that created the pre-signed URL

• Presigned Urls are commonly used to provide access to private objects

• Can use AWS CLI or AWS SDK to generate Presigned Urls

• If in case a web-application which need to allow users to download files from a password protected part of the web-app. Then the web-app generates presigned url which expires after 5 seconds. The user downloads the file.

Simple Notification System

Introduction

• Pub/sub system
• The “event producer” only sends messages to one SNS topic
• As many subscribers as we want to listen to the SNS topic notifications
• Each subscriber to the topic will get all of the messages (new feature to filter messages)
• Up to 12,500,000 subscriptions per topic
• 100,000 topic limit
• Subscribers can be:
  • SQS, Lambda, Kinesis Data Firehose, Emails, SMS, etc.
• Publishers can be:
  • CloudWatch, Budgets, S3 Event Notifications, any many more…

How to publish

• Topic Publish (using the SDK)
  • Create a topic
  • Create a subscription (or many)
  • Publish to the topic
• Direct Publish (for mobile apps SDK)
  • Create a platform application
  • Create a platform endpoint
  • Publish to the platform endpoint
  • Works with Google GCM, Apple APNS, Amazon ADM, etc.

SNS + SQS: Fan Out

• Concept: Push once in SNS, receive in all SQS queues that are subscribers

Simple Queue System

Introduction

• When you deploy an application, it will communicate in one of two ways
  • Synchronous: Applications talk directly to each other
  • Asynchronous: Applications use some type of ‘middle-man’ to communicate, such as a queue
• A queue can have multiple producers and multiple consumers
• SQS offers unlimited throughput and unlimited messages, with less than 10 ms latency
• SQS is the oldest service provided by AWS
• The default TTL of a message in the queue is 4 days and the maximum is 14 days
• Messages must be less than 256 KB
• SQS can have duplicate messages and messages may be delivered out of order

Producing Messages

• Messages are sent to SQS using the SendMessage API
• The message is persisted until a consumer deletes it, unless the TTL expires

Consuming Messages

• An application you write. Can be hosted anywhere (AWS, on-prem, etc.)
• The consumer will poll the queue for new messages and receive up to 10 messages at a time
• Consumers need to delete messages after processing them, otherwise other consumers may receive the messages
• You can create a EC2 ASG to pull the CloudWatch Metric “Queue Length” and scale in/out based on the value of the metric
  • This metric value is the number of messages in a queue

SQS Queue Access Policy

• You can allow an EC2 instance in a different AWS account to access a queue using an SQS access policy
• You can use an access policy to allow an S3 bucket to write to an SQS queue using Event Notifications

Message Visibility Timeout

• After a message is polled by a consumer, it becomes invisible to other consumers
• By default, the message is invisible to other consumers for 30 seconds
• If a message is not processed within the visibility timeout, it may be processed twice. Your application can change this behavior by calling the ChangeMessageVisibility API

Dead letter Queues

• If a consumer fails to process a message within the Visibility Timeout, the message goes back to the queue.
• We can set a threshold of how many times the message can go back into the queue
• After the MaximumRecieves threshold is exceeded, the message goes into a dead letter queue (DLQ)
• The Dead letter queue of a FIFO queue must also be a FIFO queue
• The dead letter queue of a standard queue must also be a standard queue

FIFO Queues

• First in, first out
• Messages are ordered in the queue, first message to arrive is the first message to leave
• The name of the queue must end in ‘.fifo’
• De-duplication
  • default de-duplication interval is 5 minutes
  • Two de-duplication methods:
    • Content based de-duplication: will hash the message body and compare
    • Explicitly provide a Message De-duplication Id
• Message Grouping

VPC

Introduction

• VPC is a private network within AWS
• VPC’s can contain one or more subnets
• A public subnet is a subnet that is accessible from the internet
• To define access to the internet and between subnets, use route tables

Internet Gateway and NAT Gateway

• Internet gateways help the VPC connect to the internet
• Public subnets have a route to the internet gateway
• NAT gateways and NAT instances (self-managed) allow your instances in your private subnet to access the internet while remaining private

Network ACL and Security Groups

• NACL is a firewall rule list which allows or denies traffic to and from a subnet
• NACL’s are attached at the subnet level
• NACL’s are stateless, meaning an inbound rule needs to have a matching outbound rule
• Security groups are a firewall rule list that controls traffic to and from an EC2 instance
• Security groups can only contain allow rules
• Security group rules can contain IP addresses/ranges or other Security Groups

VPC Flow Logs

• Flow logs log traffic into a VPC, subnet, or Elastic Network Interface
• 3 Types of flow logs
  • VPC Flow Logs
  • Subnet Flow Logs
  • ENI Flow Logs
• Log data can be sent to S3, CloudWatch Logs, and Kinesis Data Firehouse

VPC Peering

• Connect two VPC, privately over the AWS backbone network
• The two VPCs must not have overlapping CIDR blocks
• VPC peering is not transitive

VPC Endpoints

• Endpoints allow you to connect to AWS Services using a private network instead of the public network
• Gives you enhanced security and lower latency accessing AWS services

Site to Site VPC

• Establish a physical connection between AWS and on-premises
• Goes over the public internet

Direct Connect

• Establish a physical connection between AWS and on-premises
• Goes over a private network
• Requires infrastructure to be put in place

AWS-Solutions-Architect-Associate-notes

This is a collection of study material and follows through guidelines of the AWS Certified Solutions Architect - Associate exam (SAA-C03) exam.

Exam Guide

Directory Map

• APIGateway
  • api-gateway-cheatsheet
• Application-Integration
  • application-integration
• CloudFront
  • cloudfront
  • cloudfront-cheatsheet
• Database
  • database
• Disaster-Recovery-Migrations
  • disaster-recover-migrations
  • disaster-recovery-cheatsheet
• EC2
  • Ec2
  • ec2-pricing
  • elb
• EFS
  • efs
• ElastiCache
  • elasticache
• Glue
  • glue-cheatsheet
• Lambda
  • lambda
• Machine-Learning-Models
  • ml-models
• Monitoring-and-Audit
  • monitoring-and-audit
• Quicksight
  • quicksight
• RDS-and-Aurora
  • aurora-cheatsheet
  • rds-and-aurora
• Redshift
  • redshift
• Route53
  • route53
• security
  • iam
  • security
• Storage
  • storage
  • storage-cheatsheet
• VPC
  • nacl-cheatsheet
  • vpc
  • vpc-endpoint-cheatsheet
  • vpc-flow-logs-cheatsheet

Table of Contents

• Storage
• VPC
• CLoudFront
• AWS Lambda
• RDS and Aurora
• Redshift
• EC2
• EC2 Pricing
• EFS
• ElastiCache for Redis
• Application Integration
• QuickSight
• Disaster Recovery Migrations
• Route 53

CheatSheets

• Storage cheat sheet
• VPC Endpoint
• VPC FLow Logs
• NACL
• CloudFront
• Aurora
• Glue
• API Gateway
• Disaster Recovery Migrations
• Overview

AWS Certified Solutions Architect Associate Practice Exams

Metadata

• Title: AWS Certified Solutions Architect Associate Practice Exams
• URL: https://www.udemy.com/course/aws-certified-solutions-architect-associate-amazon-practice-exams-saa-c03/learn/quiz/4394972/result/1100741372

Highlights & Notes

• In Auto Scaling, the following statements are correct regarding the cooldown period: It ensures that the Auto Scaling group does not launch or terminate additional EC2 instances before the previous scaling activity takes effect. Its default value is 300 seconds. It is a configurable setting for your Auto Scaling group.

• You can use Amazon Data Lifecycle Manager (Amazon DLM) to automate the creation, retention, and deletion of snapshots taken to back up your Amazon EBS volumes. Automating snapshot management helps you to: - Protect valuable data by enforcing a regular backup schedule. - Retain backups as required by auditors or internal compliance. - Reduce storage costs by deleting outdated backups.

• AWS Global Accelerator and Amazon CloudFront are separate services that use the AWS global network and its edge locations around the world. CloudFront improves performance for both cacheable content (such as images and videos) and dynamic content (such as API acceleration and dynamic site delivery). Global Accelerator improves performance for a wide range of applications over TCP or UDP by proxying packets at the edge to applications running in one or more AWS Regions. Global Accelerator is a good fit for non-HTTP use cases, such as gaming (UDP), IoT (MQTT), or Voice over IP, as well as for HTTP use cases that specifically require static IP addresses or deterministic, fast regional failover. Both services integrate with AWS Shield for DDoS protection.

• A Gateway endpoint is a type of VPC endpoint that provides reliable connectivity to Amazon S3 and DynamoDB without requiring an internet gateway or a NAT device for your VPC. Instances in your VPC do not require public IP addresses to communicate with resources in the service.

• AWS DataSync makes it simple and fast to move large amounts of data online between on-premises storage and Amazon S3, Amazon Elastic File System (Amazon EFS), or Amazon FSx for Windows File Server. Manual tasks related to data transfers can slow down migrations and burden IT operations. DataSync eliminates or automatically handles many of these tasks, including scripting copy jobs, scheduling, and monitoring transfers, validating data, and optimizing network utilization. The DataSync software agent connects to your Network File System (NFS), Server Message Block (SMB) storage, and your self-managed object storage, so you don’t have to modify your applications. DataSync can transfer hundreds of terabytes and millions of files at speeds up to 10 times faster than open-source tools, over the Internet or AWS Direct Connect links. You can use DataSync to migrate active data sets or archives to AWS, transfer data to the cloud for timely analysis and processing, or replicate data to AWS for business continuity. Getting started with DataSync is easy: deploy the DataSync agent, connect it to your file system, select your AWS storage resources, and start moving data between them. You pay only for the data you move.

• Here is a list of important information about EBS Volumes:

  • When you create an EBS volume in an Availability Zone, it is automatically replicated within that zone to prevent data loss due to a failure of any single hardware component.

  • An EBS volume can only be attached to one EC2 instance at a time.

  • After you create a volume, you can attach it to any EC2 instance in the same Availability Zone

  • An EBS volume is off-instance storage that can persist independently from the life of an instance. You can specify not to terminate the EBS volume when you terminate the EC2 instance during instance creation.

  • EBS volumes support live configuration changes while in production which means that you can modify the volume type, volume size, and IOPS capacity without service interruptions.

  • Amazon EBS encryption uses 256-bit Advanced Encryption Standard algorithms (AES-256)

  • EBS Volumes offer 99.999% SLA.

Table of Contents

Table of Contents

• Amazon EBS
• Cloudwatch
• AWS Identity and Access Management
• RDS
• Athena
• Kinesis
• DynamoDB
• Storage Gateway
• Elastic Load Balancer
• Security Group
• Route 53
• AWS Transit Gateway
• Amazon EMR
• Auto Scaling
• S3
• Cloudfront
• Secrets Manager
• Textract
• RPO and RTO
• EC2
• Network Firewall
• Security

Amazon EBS

• Amazon EBS provides three volume types to best meet the needs of your workloads:

  • General Purpose (SSD)
    • General Purpose (SSD) volumes are suitable for a broad range of workloads, including small to medium-sized databases, development and test environments, and boot volumes.
  • Provisioned IOPS (SSD)
    • These volumes offer storage with consistent and low-latency performance and are designed for I/O intensive applications such as large relational or NoSQL databases.
  • Magnetic
    • for workloads where data are accessed infrequently, and applications where the lowest storage cost is important.

• Here is a list of important information about EBS Volumes:

  • When you create an EBS volume in an Availability Zone, it is automatically replicated within that zone to prevent data loss due to a failure of any single hardware component.

  • An EBS volume can only be attached to one EC2 instance at a time.

  • After you create a volume, you can attach it to any EC2 instance in the same Availability Zone

  • An EBS volume is off-instance storage that can persist independently from the life of an instance. You can specify not to terminate the EBS volume when you terminate the EC2 instance during instance creation.

  • EBS volumes support live configuration changes while in production which means that you can modify the volume type, volume size, and IOPS capacity without service interruptions.

  • Amazon EBS encryption uses 256-bit Advanced Encryption Standard algorithms (AES-256)

  • EBS Volumes offer 99.999% SLA. This

Cloudwatch

• Monitoring tool for your AWS resources and applications.
• Display metrics and create alarms that watch the metrics and send notifications or automatically make changes to the resources you are monitoring when a threshold is breached.

AWS Identity and Access Management

• You should always associate IAM role to EC2 instances not IAM user for the purpose of accessing other AWS services

• IAM roles are designed so that your application can securely make API requests from your instances, without requiring you to manage the security credentials that the application use.

  • Instead of creating and distributing your AWS credentials, you can delegate permission to make API requests using IAM roles

• AWS Organization is a service that allows you to manage multiple AWS accounts easily.

• AWS IAM Identity Center can be integrated with your corporate directory service for centralized authentication.

  • This means you can sign in to multiple AWS accounts with just one set of credentials.
  • This integration helps to streamline the authentication process and makes it easier for companies to switch between accounts.

• SCP you can also configure a service control policy (SCP) to manage your AWS accounts.

  • SCPs help you enforce policies across your organization and control the services and features accessible to your other account.
  • prevents unauthorized access

• Security Token Service (STS) is the service that you can use to create and provide trusted users with temporary security credentials that can control access to your AWS resources.

  • Temporary security credentials work almost identically to the long-term access key credentials that your IAM users can use.

• AWS Control Tower provides a single location to easily set up your new well-architected multi-account environment and govern your AWS workloads with rules for security,operations, and internal compliance.

  • You can automate the setup of your AWS environment with best-practices blueprints for multi-account structure, identity, access management, and account provisioning workflow.
  • offers “guardrails” for ongoing governance of your AWS environment.

• You can use an IAM role to specify permissions for users whose identity is federated from your organization or a third-party identity provider (IdP).

  • Federating users with SAML 2.0
    • If your organization already uses an identity provider software package that supports SAML 2.0 (Security Assertion Markup Language 2.0), you can create trust between your organization as an identity provider (IdP) and AWS as the service provider.
    • You can then use SAML to provide your users with federated single-sign on (SSO) to the AWS Management Console or federated access to call AWS API operations.
    • For example: if your company uses Microsoft Active Directory and Active Directory Federation Services, then you can federate using SAML 2.0
  • Federating users by creating a custom identity broker application

    • If your identity store is not compatible with SAML 2.0, then you can build a custom identity broker application to perform a similar function.

    • The broker application authenticates users, requests temporary credentials for users from AWS, and then provides them to the user to access AWS resources.

    • The application verifies that employees are signed into the existing corporate network’s identity and authentication system, which might use LDAP, Active Directory, or another system. The identity broker application then obtains temporary security credentials for the employees.

    • To get temporary security credentials, the identity broker application calls either AssumeRole or GetFederationToken to obtain temporary security credentials, depending on how you want to manage the policies for users and when the temporary credentials should expire.

    • The call returns temporary security credentials consisting of an AWS access key ID, a secret access key, and a session token. The identity broker application makes these temporary security credentials available to the internal company application.

      

    • This scenario has the following attributes:

      • The identity broker application has permissions to access IAM’s token service (STS) API to create temporary security credentials.

      • The identity broker application is able to verify that employees are authenticated within the existing authentication system.

      • Users are able to get a temporary URL that gives them access to the AWS Management Console (which is referred to as single sign-on).

RDS

• Supports Aurora, MySQL, MariaDB, PostgreSQL, Oracle, Microsoft SQL Server.
• DB Instance
  • For production OLTP use cases, use Multi-AZ deployments for enhanced fault tolerance with Provisioned IOPS storage for fast and predictable performance.
    • You can use PIOPS storage with Read Replicas for MySQL, MariaDB or PostgreSQL.
  • Magnetic
    • Doesn’t allow you to scale storage when using the SQL Server database engine.
      • Doesn’t support elastic volumes.
      • Limited to a maximum size of 3 TiB.
      • Limited to a maximum of 1,000 IOPS.
• RDS automatically performs a failover in the event of any of the following:
  1. Loss of availability in primary Availability Zone.
  2. Loss of network connectivity to primary.
  3. Compute unit failure on primary.
  4. Storage failure on primary.

Athena

• An interactive query service that makes it easy to analyze data directly in Amazon S3 and other data sources using SQL.
• Serverless
• Has a built-in query editor.
• highly available and durable
• integrates with Amazon QuickSight for easy data visualization.
• retains query history for 45 days.
• You pay only for the queries that you run. You are charged based on the amount of data scanned by each query.
• There are 2 types of cost controls:
  • Per-query limit
    • specifies a threshold for the total amount of data scanned per query.
    • Any query running in a workgroup is canceled once it exceeds the specified limit.
    • Only one per-query limit can be created
  • Per-workgroup limit
    • this limits the total amount of data scanned by all queries running within a specific time frame.

Kinesis

• A fully managed AWS service that you can use to stream live video from devices to the AWS Cloud, or build applications for real-time video processing or batch-oriented video analytics.
• Amazon Kinesis Data Streams enables real-time processing of streaming big data. It provides ordering of records, as well as the ability to read and/or replay records in the same order to multiple Amazon Kinesis Applications
• A Kinesis data stream is a set of shards that has a sequence of data records , and each data record has a sequence number that is assigned by Kinesis Data Streams.
  • Kinesis can also easily handle the high volume of messages being sent to the service.
  • durable
  • no missing of messages

DynamoDB

• How to choose the right partition key ?

  • What is partition key ?
    • DynamoDB supports 2 types of primary keys
      • Partition key: A simple primary key, composed of one attribute known as the partition key.
      • Partition key and Sort key: Referred to as Composite Primary Key, this type of key is composed of two attributes. 1st one is partition key and 2nd one is sort key
  • Why do I need a partition key?
    • DynamoDb stores data as groups of attributes, - Items
    • Items are similar to rows or records in other database systems.
    • DynamoDB stores and retrieves each item based on the primary key value which must be unique
    • DynamoDb uses the partition key’s value as an input to an internal hash function. The output from the hash function determines the partition in which the item is stored. Each item’s location is determined by the hash value of its partition key.
  • DynamoDB automatically supports access patterns using the throughput you have provisioned, or upto your account limits in the on-demand mode
  • Regardless of the capacity mode you choose if your access pattern exceeds 3000 RCU and 1000 WCU for a single partition key value, your requests might be throttled with a ProvisionedThroughputExceededException error
  • Recommended for Partition keys :
    • Use high-cardinality attributes. These are attributes that have distinct values for each item, like emailid, employee_no, customerid, sessionid, orderid
    • Use composite attributes Try to combine more tha one attribute to form a unique key, if that meets your access pattern
    • Cache the popular items when there is high volume of read traffic using DAX (DynamoDB Accelerator)
    • DAX is fully managed, in-memory cache for DynamoDB that doesn’t require developers to manage cache invalidation, data population or cluster management.
    • DAX also is compatible with DynamoDB API calls, so developer can incorporate it more easily into existing applications

Storage Gateway

• Connects an on-premise software appliance with cloud-based storage to provide seamless integration with data security features between your on-premises IT environment and the AWS storage infrastructure.

• You can use the service to store data in the AWS cloud for scalable and cost-effective storage that helps maintain

• It stores files as native S3 objects, archives virtual tapes in Amazon Glacier and stores EBS snapshots generated by the Volume Gateway with Amazon EBS.

  

Elastic Load Balancer

• Distributes incoming application or network traffic across multiple targets, such as EC2 instances containers (ECS), Lambda functions and IP addresses in multiple Availability zones

Security Group

• A security group acts as a virtual firewall for your instance to control inbound and outbound traffic.

  

Route 53

• A highly available and scalable Domain Name System (DNS) web service used for domain registration, DNS routing and health checking

AWS Transit Gateway

• A networking service that uses a hub and spoke model to connect the on-premises data centers and Amazon VPCs to a Single Gateway.
• With this service, customers only have to create and manage a single connection from the central gateway into each on-premises data center
• Features:
  • Inter-region peering
    • allows customers to route traffic
    • easy and cost-effective way
  • Multicast
    • allows customers to have fine-grain control on who can consume and produce multicast traffic
  • Automated provisioning
    • customers can automatically identify the Site-to-site VPN connections and on-premises resources with which they are associated using AWS Transit Gateway

Amazon EMR

• EMR (Elastic MapReduce)

• A managed cluster that simplifies running big data frameworks like Apache Hadoop and Apache Spark on AWS to process and analyze vast amounts of data.

• You can process data for analytics purposes and business intelligence workloads using EMR together with Apache Hive and Apache Pig

• You can use EMR to move large amounts of data in and out of other AWS data stores and databases like S3 and DynamoDB

  

• Purchasing options:

  • On-Demand:reliable, predictable, won’t be terminated
  • Reserved (min 1 year): cost savings (EMR will automatically use if available)

Auto Scaling

• Configure automatic scaling for the AWS resources quickly through a scaling plan that uses Dynamic Scaling and Predictive scaling .
• Useful for :
  • Cyclical traffic such as high use of resources during regular business hours and low use of resources
  • On and Off traffic such as batch processing, testing and periodic analysis
  • Variable traffic patterns, such as software for marketing growth with periods of spiky growth
• Dynamic Scaling
  • To add and remove capacity for resources to maintain resource at target value
• Predictive Scaling
  • To forecast the future load demands by analyzing your historical records for a metric
  • Allows schedule scaling by adding or removing capacity and controls maximum capacity
  • Only available for EC2 scaling groups
• In Auto Scaling, the following statements are correct regarding the cooldown period:
  • It ensures that the Auto Scaling group does not launch or terminate additional EC2 instances before the previous scaling activity takes effect.
  • Its default value is 300 seconds.
  • It is a configurable setting for your Auto Scaling group.

S3

• Server-side encryption (SSE) is about data encryption at rest-that is, Amazon S3 encrypts your data at the object level as it writes it to disks in its data centers and decrypts it for you when you access it.

  • You have three mutually exclusive options depending on how you choose to manage the encryption keys:

    1.Amazon S3-Managed Keys (SSE-S3)

    2. AWS KMS-Managed Keys (SSE-KMS)

    3. Customer-Provided Keys (SSE-C)

  • S3-Managed Encryption Keys (SSE-S3)

    • Amazon S3 will encrypt each object with a unique key and as an additional safeguard, it encrypts the key itself with a master key that it rotates regularly.

  • SSE-AES S3 handles the key, uses AES-256 algorithm

    • one of the strongest block ciphers available, 256-bit Advanced Encryption Standard (AES-256), to encrypt your data.
• Client-side Encryption using
  1. AWS KMS-managed customer master key
  2. client-side master key
• Cross-Account Access You can provide another AWS account access to an object that is stored in an S3 bucket.

  • These are the methods on how to grant cross-account access to objects that are stored in your own Amazon S3 bucket:

    • Resource-based policies and IAM policies
    • Resource-based Access Control List (ACL) and IAM policies

  • Cross-account IAM roles for programmatic and console access to S3 bucket objects

  • Supports failover controls for S3 Multi-Region access points.

• Requester Pays Buckets
  • Bucket owners pay for all of the Amazon S3 storage and data transfer costs associated with their bucket.

CloudFront

Secrets Manager

• Helps to manage, retrieve and rotate database credentials, application credentials, OAuth tokens, API keys and other secrets throughout their lifecycles
• Helps to improve security posture , because you no longer need hard-coded credentials in application source code.
  • Storing the credentials in Secrets Manager helps avoid possible compromise by anyone who can inspect the application or the components.
  • Replace hard-coded credentials with a runtime call to the Secrets Manager service to retrieve credentials with a runtime call to the Secrets Manager service to retrieve credentials dynamically when you need them.

Textract

• A fully managed document analysis service for detecting and extracting information from scanned documents
• Return extracted data as key-value pairs (e.g. Name: John Doe)
• Supports virtually any type of documents
• Pricing
  • Pay for what you use
  • Charges vary for Detect Document Text API and Analyze Document API with the later being more expensive

RPO and RTO

• RTO (Recovery Time Object)
  • measures how quickly the application should be available after an outage
• RPO (Recovery Point Object)
  • refers to how much data loss can the application can tolerate

- Data loss is measured from most recent backup to the point of disaster. Downtime is measured from the point of disaster until fully recovered and available for service.

EC2

Network Firewall

• AWS Network Firewall supports domain name stateful network traffic inspection
• Can create allow lists and deny lists with domain names that the stateful rules engine looks for in network traffic
• AWS Network Firewall is a stateful, managed network firewall and intrusion detection and prevention service for your virtual private cloud (VPC) that you created in Amazon Virtual Private Cloud (Amazon VPC).
  • With Network Firewall, you can filter traffic at the perimeter of your VPC.
  • This includes filtering traffic going to and coming from an internet gateway, NAT gateway, or over VPN or AWS Direct Connect.
• Network Firewall uses the open source intrusion prevention system (IPS), Suricata, for stateful inspection. Network Firewall supports Suricata compatible rules.

Security

• The security pillar includes the ability to protect data, systems, and assets to take advantage of cloud technologies to improve security

• Zero Trust security is a model where application components or microservices are considered discrete from each other and no component or microservice trusts any other.

  Design Principles

  1. Implement a strong identity foundation

  2. Enable traceability

  3. Apply security at all layers:

     • Apply a defense in depth approach with multiple security controls

     • Implementing security to multiple layers (for example, edge of network, VPC, load balancing, every instance and compute service, operating system, application, and code).

       

  4. Automate security best practices:

  5. Protect data in transit and at rest:

  6. Keep people away from data:

  7. Prepare for security events:

Study-more

[ ] Spot Fleets [ ] Reserved Instances vs. Dedicated Hosts

Tutorialsdojo Cheatsheets

• IAM
• AWS Directory Service
• S3
• RDS
• Aurora
• DynamoDB
• EBS
• Route53
• Auto Scaling
• KMS
• EFS
• cloudHSM (optional)

Api-gateway-cheatsheet

• Enables developers to create, publish, maintain, monitor, and secure APIs at any scale

• Create RESTful or WebSocket APIs

• HIPAA compliant service

• Allows creating, deploying and managing a RESTful API to expose backend HTTP endpoints, Lambda functions or other AWS services

• Concepts

  • API deployment
    • a point-in-time snapshot of your API Gateway API resource and methods. To be available for clients to use, the deployment must be associated with one or more API stages
  • API endpoints
    • host names APIs in API Gateway, which are deployed to a specific region and of the format: rest-api-id.execute-api.region.amazonaws.com
  • Usage Plan
    • Provides selected API clients with access to one or more deployed APISs. You can use a usage plan to configure throttling and quota limits, which are enforced on individual client API keys

• Features:

  • Amazon API Gateway provides throttling at multiple levels including global and by a service call. Throttling limits can be set for standard rates and bursts.
    • For example, API owners can set a rate limit of 1,000 requests per second for a specific method in their REST APIs, and also configure Amazon API Gateway to handle a burst of 2,000 requests per second for a few seconds.

• Endpoint Types

  • Edge-optimized: For global clients
    • Requests are routed through a Cloudfront Edge Location for improved latency
    • The API Gateway still only lives in one region
  • Regional: for clients within the same region
    • You could still manually combine with CloudFront for control over caching strategies
  • Private: Only accessible in our VPC
    • Use a resource policy to define access

• Stages

  • Create stages for different deployments of the API. Example: Production, Sandbox, QA, etc.
  • Switch between stages seamlessly
  • Similar to Azure Web App Deployment Slots
  • Use stage variables

Queueing (SQS)

• Queueing (SQS)
• SQS Message Visibility Timeout
• Long Polling
• Streaming and Kinesis
  • Kinesis Data Streams
  • Kinesis Data Firehose
  • Kinesis Data Streams vs Firehose
  • Kinesis Data Analytics
  • Kinesis Video Streams
• Pub-Sub and SNS
  • What is Pub/Sub ?
• SQS and SNS - Fan Out Pattern

Queueing (SQS)

• What is Messaging System ?

  • Used to provide asynchronous communication and decouple processes via messages / events from sender and receiver (producer and consumer)

• What is Queuing System ?

  • A queueing system is a messaging system that generally will delete messages once they are consumed .
  • Simple Communication
  • Not Real-time
  • Have to pull
  • Not reactive

• Simple Queuing System (SQS)

  • Fully managed queuing service that enables you to decouple and scale mircroservices, distributed systems, and serverless applications
  • Use Case: You need to queue up transaction emails to be sent
  • e.g. Signup, Reset Password
  • Default retention 4 Days and Max of 14 days
  • Limitation of 256 KB per message sent
  • Low Latency (<10ms on publish and receive)
  • Can have duplicate messages (at least once delivery, occasionally)
  • Unlimited Throughput
  
  • Encryption:
    • In-flight encryption using HTTPS API
    • At-rest encryption using KMS keys
    • Client-side encryption if the client wants to perform encryption/decryption itself

SQS Message Visibility Timeout

• After a message is polled by a customer it becomes invisible to other consumers
• By default the “ message visibility timeout“ is 30 seconds
• That means the message has 30 seconds to process
• If the message is not processed in the visibility timeout, it will be processed twice
• A consumer could call the ChangeMessageVisibility API to get more time
• If the visibility timeout is high(hours) and consumer crashes, reprocessing will take time
• If visibility timeout is too low (seconds), we may get duplicates

Long Polling

• When a consumer requests messages from the queue, it can optionally ‘wait’ for messages to arrive if there are none in the queue - Long Polling
• Long Polling decreases the number of API calls made to SQS while increasing the latency and efficiency of the application
• The wait time can be between 1 sec to 20 sec
• Long Polling is preferable to Short Polling
• Long Polling can be enabled at the queue level or at the API level using WaitTimeSeconds

Streaming and Kinesis

• What is Streaming ?

  • Multiple consumers can react to events (messages)

  • Events live in the stream for long periods of time, so complex operations can be applied

  • Real-time

  • Amazon-Kinesis

    • Amazon Kinesis is the AWS fully managed solution for collecting, processing and analyzing streaming data in the cloud
    

  Kinesis Data Streams

  • Capture,process and store data streams

    • Security:

      
      • Control access/ authorization using IAM policies
      • Encryption in flight using HTTPS endpoints
      • Encryption at rest using KMS
      • You can implement encryption/decryption of data on client-side (harder)
      • VPC endpoints available for Kinesis to access within VPC
      • Monitor API calls using CLoudTrail

  Kinesis Data Firehose

  • load data streams into AWS data stores
  • Pay for only data that is going through Firehose
  • Supports many data formats, conversions, transformations, compression

  Kinesis Data Streams vs Firehose

  Kinesis Data Streams	Kinesis Data Firehose
  - Streaming service for ingest at scale	- Load streaming data into S3 /Redshift /OpenSearch / 3rd Party /custom HTTP
  write Custom code (producer/consumer)	Fully managed
  Real-time (~200 ms)	Near real-time (buffer time min 60 sec)
  Managed scaling (shard splitting / merging)	Automatic scaling
  Data storage for 1 to 365 days	No data storage
  Supports replay capability	Doesn’t support Capability

  Kinesis Data Analytics

  • analyze data streams with SQL or Apache Flink

  Kinesis Video Streams

  • Capture, process and store video streams

Pub-Sub and SNS

• What is Pub / Sub ?

  • Publish-subscribe pattern commonly implemented in messaging systems.

  • In a pub/sub system the sender of messages (publishers) do not send their messages directly to receivers.

  • They instead send their messages to an event bus . The event bus categorizes their messages into groups.

  • The receivers of messages Subscribers subscribe to these groups

  • Whenever new messages appear within their subscription the messages are immediately delivered to them

    

  • Publisher have no knowledge of who their subscribers are

  • Subscribers do not pull for messages

  • Messages are instead automatically and immediately pushed to subscribers

  • Messages and events are interchangeable terms in pub/sub

  • Use case:

    • A real-time chat system
    • A web-hook system

• Simple Notification Service

  • It is a highly available, durable, secure, fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems and serverless applications
  

SQS and SNS - Fan Out Pattern

• Push once in SNS, receive in all SQS queues that are subscribers
• Fully decoupled : no data loss
• SNS - Message Filtering
  • JSON policy used to filter messages sent to SNS topic’s subscriptions
  • If a subscription doesn’t have a filter policy, it receives every message

Cloudfront-cheatsheet

• CloudFront is a CDN (Content Distribution Network). It makes website load fast by serving cached content that is nearby
• CloudFront distributes cached copy at Edge Locations
• Edge Locations aren’t just not read-only , you can write them eg. PUT objects
• TTL (Time to live) defines how long until the cache expires (refreshes cache)
• When you invalidate your cache, you are forcing it to immediately expire (refreshes cached data)
• Refreshing the cache costs money because of transfer costs to update Edge locations
• Origin is the address of where the original copies of your files reside eg. S3, EC2, ELB, Route53
• Distribution defines a collection of Edge Locations and behavior on how it should handle your cached content
• Distributions has 2 Types :
  • Web Distribution (statis website content)
  • RTMP (steaming media)
• Origin Identity Access (OAI) is used access private S3 buckets
• Access to cached content can be protected via Signed URLs or Signed Cookies
• Lambda@Edge allows you to pass each request through a Lambda to change the behavior of the response

CloudFront

• CloudFront
• CloudFront Core Components
• CloudFront Distributions
• Lambda@Edge
• CloudFront Protection

CloudFront

• Content Distribution Network (CDN) creates cached copies of your website at various Edge locations around the world

• Content Delivery Network (CDN)

  • A CDN is a distributed network of servers which delivers web pages and content to users based on their geographical location, the origin of the webpage and a content delivery server

    • Can be used to deliver an entire website including static, dynamic and streaming

    • 216 points of presence globally

    • DDoS protection since it is a global service. Integrates with AWS Shield and AWS WAF

    • Requests for content are served from the nearest Edge Location for the best possible performance

      

CloudFront Core Components

• Origin

  • The location where all of original files are located. For example an S3 Bucket, EC2 Instance, ELB or Route53

• Edge Location

  • The location where web content will be cached. This is different than an AWS Region or AZ

• Distribution

  • A collection of Edge locations which defines how cached content should behave

    

CloudFront Distributions

• A distribution is a collection of Edge Location. You specific the Origin eg. S3, EC2, ELB, Route53
• It replicates copies based on your Price Class
• There are two types of Distributions
  1. Web (for Websites)
  2. RTMP (for streaming media)
• Behaviors
  • Redirect to HTTPs, Restrict HTTP Methods, Restrict Viewer Access, Set TTLs
• Invalidations
  • You can manually invalidate cache on specific files via Invalidations
• Error Pages
  • You can serve up custom error pages eg 404
• Restrictions
  • You can use Geo Restriction to blacklist or whitelist specific countries

Lambda@Edge

• Lambda@Edge functions are used to override the behavior of request and responses

• Lambda@Edge lets you run Lambda functions to customize the content that CloudFront delivers, executing the functions in AWS locations closer to the viewer.

• The functions run in response to CloudFront events, without provisioning or managing servers. You can use Lambda functions to change CloudFront requests and responses at the following points:

• The 4 Available Edge Functions

  1. Viewer Request
     • When CloudFront receives a request from a Viewer
  2. Origin request
     • Before CLoudFront forwards a request to the origin
  3. Origin response
     • When cloudfront receives a response from the origin
  4. Viewer response
     • Before CLoudFront returns the response to the viewer
  

CloudFront Protection

• By Default a Distribution allows everyone to have access
• Original Identity Access (OAI)
  • A virtual user identity that will be used to give your CloudFront Distribution permission to fetch a private object
• Inorder to use Signed URLs or Signed Cookies you need to have an OAI
• Signed URLs
  • (Not the same thing as S3 Presigned URL)
    • A url with provides temporary access to cached objects
• Signed Cookies
  • A cookie which is passed along with the request to CloudFront. The advantage of using a Cookie is you want to provide access to multiple restricted files. eg. Video Streaming

What is Database ?

• What is Database
• What is Data Warehouse
• What is a key value store?
• What is a document database?
• NOSQL Database Services
  • DynamoDB
  • DocumentDB
  • Amazon Keyspaces
• Relational Database Service
• Other Database Services

What is Database ?

• A database is data-store that store semi-structured and structured data
• A database is more complex stores because it requires using formal design and modeling techniques
• Database types:
  • Relational Database
    • Structured data represents tabular data (tables,rows and columns)
  • Non-Relational Database
    • Semi-Structured that may or may not represent tabular data
• Set of functionality:
  • query
  • modeling strategies to optimize retrieval for different use cases
  • control over the transformation of the data into useful data structures or reports

What is Data Warehouse ?

• Relational Database : designed for analytic workloads and a column-oriented data-store
• Companies will have terabytes and millions of rows of data
• Data warehouses generally perform aggregation
  • aggregation is is grouping data eg. finding a total or average
  • Data warehouses are optimised around columns since they need quickly aggregate column data
• Data warehouses are generally designed be HOT
  • HOT means they can return queries very fast even though they have vast amounts of data
• Data warehouses are infrequently accessed
  • intended for real time reporting but maybe once or twice a day or once a week to generate business or user reports
• Data Warehouse needs to consume data from a relational database on a regular basis

What is a Key Value Store ?

• A key-value database is a type of non-relational database (NoSQL) that uses a simple key-value method to store data
  • Stores a unique key alongside a value
  • will interpret this data resembling a dictionary
  • can resemble tabular data, it does not have to have the consistent columns per row -Due to simple design so they can scale well beyond a relational database

What is a document database ?

• Document store - a NOSQL database that stores documents as its primary data-structure

  • it could be an XML but more commonly is JSON or JSON-like

  • they are sub-class of key/value stores

    

NOSQL Database Services

• DynamoDB

  • a serverless NOSQL key/Value and document database
  • designed to scale to billions of records with consistent data return in at least a second- millisecond latency
  • It is AWS’s flagship database service meaning it is cost-effective and very fast
  • DAX cluster for read cache, microsecond read latency
  • Event Processing: DynamoDB Streams to integrate with AWS Lambda, or Kinesis Data Streams
  • Global Table feature: active-active setup
  • Automated backups up to 35 days with PITR (restore to new table), or on-demand backups
  • Export to S3 without using RCU within the PITR window, import from S3 without using WCU
  • Great to rapidly evolve schema
  • It is a massively scalable database
  • Usecases: Serverless applications development (small documents 100s Kb), distributed serverless cache

• DocumentDB

  • A NOSQL document database that is “MongoDB compatible”
  • MongoDB is very popular NOSQL among developers there were open-source licensing issues around using open-source MongoDB , so aws got aorund it by just building their own MongoDB database
  • when you want a MongoDB database

• Amazon KeySpaces

  • A fully managed Apache Cassandra database
  • Cassandra is an open-source NOSQL key/value database similar to DynamoDB in that is columnar store database but has some additional functionality
  • when you want to use Apache Casandra

Relational Database Service

• Relational Database Services (RDS)

  • supports multiple SQL engines
  • Relational is synonymous with SQL and Online Transactional Processing (OLTP)
  • most commonly used type of database among tech companies and start ups
  • RDS supports the following SQL Engines:
    • MYSQL - Most popular open source SQL database that was purchased and now owned by Oracle
    • MariaDB - When Oracle bought MYSQL. MariaDB made a fork (copy) of MYSQL was made under a different open-source license
    • Postgres (PSQL) - Most popular open-source SQL database among developers. Has rich-features over MYSQL but at added complexity
    • Oracle - Oracle’s proprietary SQL database. Well used by Enterprise companies. Have to buy a license to use it
    • Microsoft SQL Server - Microsoft’s proprietary SQL database. Have to buy license to use it
    • Aurora - Fully managed database
      • Aurora
        • fully managed database,
        • database of either MYSQL (5X faster) and PSQL (3X faster) database
        • When you want a highly available, durable, scalable and secure relational database for Postgres or MySQL then Aurora is correct fit
    • Aurora Serverless - serverless on-demand version of Aurora. - When you want “most” of the benefits of Aurora but can trade to have cold-starts or you don’t have lots of traffic demand
    • RDS on VMware - allows you to deploy RDS supported engines to on-premise data center. - datacenter must be using VMware for server virtualization - when you want databases managed by RDS on your own datacenter

Other Database Services

• RedShift
  • petabyte-size data-warehouse
  • Data warehouses
    • are for Online Analytical Procesing (OLAP)
    • can be expensive because they are keeping data “hot”
    • “HOT” means we can run a very complex query and a large amount of data and get that data very fast
    • Usage: when you want to quickly generate analytics or reports from a large amount of data
• ElasticCache
  • a managed database of the in-memory and caching open-source databases
  • Redis or Memcached
  • Usage: when you want to improve the performance of application by adding a caching layer in-front of web-server or database
• Neptune
  • a managed graph database
  • Data is represented in interconnected nodes
  • Usage: when you need to understand the connections between data eg. Mapping Fraud Rings or Social Media Relationships
• Amazon Timestreams
  • a fully managed time series database
  • Related to Devices that send lot of data that are time-sensitive such as IOT devices
  • Usage: When you need to measure how things change over time
• Amazon Quantum Ledger Database
  • a fully managed ledger database that provides transparent, immutable and cryptographically variable transaction logs
  • Usage: when you need to record history of financial activities that can be trusted
• Database Migration Service
  • a database migration service
  • Can migrate from:
    • On-premise database to AWS
    • from two database in different or same AWS accounts using SQL engines
    • from a SQL to NOSQL database

Disaster Recovery

• RPO: how much data loss are you willing to accept during a disaster

• RTO: how much downtime can you accept

• Disaster Recovery in AWS

• Database Migration Service

  • Continuous Replication
  • Multi-AZ Deployment

• RDS & Aurora MySQL Migrations

Disaster Recovery in AWS

• Any event that has a negative impact on a company’s business continuity or finances is a disaster

• Disaster recovery (DR) is about preparing for and recovering from a disaster

• What kind of disaster recovery?

  • On-premise => On-Premise: traditional DR and very expensive
  • On-Premise => AWS cloud: hybrid recovery
  • AWS Cloud Region A => AWS Cloud Region B

• Disaster Recovery Strategies

  

  • Backup and Restore

    • High RPO

    • Cheap

    • Easy to implement

      

  • Pilot Light

    • small version of the app is always running in the cloud
    • Useful for the critical core components of the application (Pilot Light)
    • Very similar to Backup and Restore
    • Faster than Backup and Restore as critical systems are already up
    

  • Warm Standby

    • Full system is up and running, but at minimum size
    • Upon disaster we can scale to production load
    

  • Hot Site/ Multi Site Approach

    • Very low RTO (minutes or seconds) - very expensive
    • Full production scale is running AWS and On Premise
    

Database Migration Service

• Supports heterogeneous and homogeneous migrations
• You must create an EC2 instance to perform the replication tasks
• Sources can be on-prem databases or EC2-based databases, Azure SQL Databases, Amazon RDS, Amazon S3, and DocumentDB
• Targets can be on-prem databases, Amazon RDS, Redshift, DynamoDB, OpenSearch, Redis, Babelfish, DocumentDB, etc.
• AWS Schema Conversion Tool (SCT) can convert the database schema from one engine to another if you are migrating to a different database engine

Continuous Replication

Multi-AZ Deployment

• When Multi-AZ Enabled, DMS provisions and maintains a synchronously stand replica in a different AZ
  • Advantages:
    • Provide Data Redundancy
    • Eliminates I/O freezes
    • Minimizes latency spikes

RDS to Aurora Migration

• Options:
  • Snapshot RDS and migrate to Aurora
  • Create an Aurora Read REplica from RDS mySQL and when the replication lag is 0, promote it as it’s own DB Cluster
  • If MySQL is external to RDS, you can backup with Percona XtraBackup and import into Aurora
  • Use DMS if both databases are up and running

On-premise Strategies

• You can download Amazon Linux ISO and run on-prem hypervisors
• Import/export VMs for on-prem to AWS
• Use AWS Application Discovery Service to gather info about on-prem VMs and plan a migration
  • Track with AWS migration hub
  • Agentless Discovery
    • VM inventory, configuration, performance history, etc.
  • Agent-Based Discovery
    • System configuration, system performance history, running processes, network connection details, etc.
  • Use Application Migration Service (MGN) to lift-and-shift VMs to AWS
• AWS Database Migration Service
  • Migrate data across database engines
  • Migrate databases from on-prem to AWS
• AWS Server Migration
  • Incremental replication of on-prem servers to AWS
  • Converts on-prem servers to cloud-based servers

AWS Backup#

• Fully managed
• Centrally manage and automate backups across all AWS services
• AWS Backup supports cross-region backups and cross-account backups
• Backup policies are known as Backup Plans
• Vault Lock is used to enforce a Write-Once-Read-Many policy (WORM) to ensure backups in the Vault cannot be deleted. Even the root user cannot delete backups when enabled.

Disaster-recovery-cheatsheet

• Backup
  • EBS Snapshots, RDS automated backups/ Snapshots etc
  • Regular pushes to S3/ S3 IA/ Glacier, Lifecycle Policy, Cross Region Replication
  • From On-Premise:Snowball or Storage Gateway
• High availability
  • Use Route 53 to migrate DNS over Region to Region
  • RDS Multi-AZ, Elastic Cache Multi-AZ, EFS, S3
  • Site to Site VPN as a recovery from Direct Connect
• Replication
  • RDS Replication (Cross Region),AWS Aurora + Global Databases
  • Database replication from on-premise to RDS
  • Storage Gateway
• Automation
  • CloudFormation / Elastic Beanstalk to re-create a whole new environment
  • Recover / Reboot Ec2 instances with CloudWatch if alarms fail
  • AWS Lambda functions for customized automatons
• Chaos
  • Netflix has a “simian-army” randomly terminating EC2

Savings Plan

• Reserved Instances (RI)
  • Term
  • Class
  • Payment Options
• Reserved Instances Attributes
• Regional and Zonal RI
• RI Limits
• Capacity Reservations
• Standard vs Convertible RI
• RI Marketplace

Savings Plan

• Get a discount based on long term usage
• Commit to a certain amount of usage

Spot Instances

• Up to 90% discount
• Specify a max price you are willing to pay for your instances. If you go over the price, you lose the instance
• The MOST cost-efficient instance pricing
• Useful for workloads that are resilient to failure (batch jobs, etc.)

Dedicated Hosts

• A physical server with EC2 instance capacity dedicated to your use
• Allows you to address compliance or licensing requirements
• The most expensive option in AWS
• Purchasing OPtions
  • On-demand
  • Reserved Instances

Dedicated Instances

• Instances run on hardware dedicated to you
• You may share hardware with other instances in same account
• No control over instance placement.

Reserved Instances (RI)

• Designed for applications that have a steady state, predictable usage or require reserved capacity.
• Reduced Pricing is based on Term x Class Offering x Payment Option

  • Term

    • {The longer the term the greater the savings}
    • Commit to 1 year or 3 Year contract
    • Reserved Instances do not renew automatically
    • When it is expired it will use on-demand with no interruption to service

  • Class

    • {The less flexible the greater savings}
    • Standard
      • Up to 75% reduced pricing compared to on-demand
      • Can modify RI attributes
    • Convertible
      • Up to 54% reduced pricing compared to on-demand
      • can exchange RI based on RI attributes if greater or equal in value
    • Scheduled
      • AWS no longer offers Scheduled RI

  • Payment Options

    • {The greater upfront the greater savings}
    • All upfront : full payment at the start
    • Partial Upfront : A portion of the cost must be paid and remaining hours billed at a discounted hourly rate
    • No Upfront : billed at a discounted hourly rate for every hour within the term,regardless of whether the Reserved Instance is being used
  • RIs can be shared between multiple accounts within AWS organization
  • Unused RIs can be sold in the Reserved Instance Marketplace

Reserved Instance (RI) Attributes

• RI attributes
  • are limited based on class offering and can affect the final price of an RI instance
  • 4 RI attributes:
    • Instance Type:
      • eg. m4.Large. This is composed of the instance family (for example , m4) and the instance size (for example large)
    • Region:
      • The region in which the Reserved Instance is purchased
    • Tenancy:
      • Whether your instance runs on shared(default) or single-tenant (dedicated) hardware
    • Platform:
      • the operating system eg. Windows or Linux/Unix

Regional and Zonal RI

RI Limits

• There is a limit to the number of Reserved Instances that you can purchase per month
  • Per month you can purchase
    • 20 Regional Reserved Instances per Region
    • 20 Zonal Reserved Instances per AZ

Capacity Reservations

• EC2 instances are backed by different kind of hardware, and so there is a finite amount of servers available within an Availability Zone per instance type or family
• You go to launch a specific type of EC2 instance but AWS has ran out of that server
• Capacity reservation is a service of EC2 that allows you to request a reserve of EC2 instance type for a specific Region and AZ

Standard vs Convertible RI

RI Marketplace

• EC2 Reserved Instance Marketplace allows you to sell your unused Standard RI to recoup your RI spend for RI you do not intend or cannot use

• Reserved Instances can be sold after they have been active for at least 30 days and once AWS has received the upfront payment (if applicable)

• You must have a US bank account to sell Reserved Instances on the Reserved Instance Marketplace

• There must be at least one month remaining in the term of the Reserved Instance you are listing

• You will retain the pricing and capacity benefit of your reservation until it’s sold and the transaction is complete

• Your company name ( and address upon request) will be shared with the buyer for tax purposes.

• A seller can set only the upfront price for a Reserved Instance. The usage price and other configuration (eg. instance type, availability zone, platform) will remain the same as when the Reserved Instance was initially purchased

• The term length will be rounded down to the nearest month. For example, a reservation with 9 months and 15 days remaining will appear as 9 months on the Reserved Instance Marketplace.

• You can sell upto $20,000 in Reserved Instances per year. If you need to sell more Reserved Instances

• Reserved Instances in the GovCloud region cannot be sold on the Reserved Instance Marketplace

EC2

• Placement Groups
• Elastic Network Interfaces

EC2

• EC2 is not just virtual machines, it consists of VMs, EBS, EIP, ENI, etc.
• Use user-data to run a script at launch. This script is only run once at the instances first start and runs as root
• t2.micro is part of the free tier

EC2 Instance Types

• General Purpose (t)
• Compute Optimized (c)
• Memory Optimized (r)
• Storage Optimized (i,d,h1)

Security Groups

• Security Groups are like a firewall for EC2 instances
• Security groups only contain allow rules
• Security groups are stateful. Meaning if we have an inbound allow rule, we don’t need a corresponding outbound allow rule
• For the source of the traffic, Security Groups can reference an IP address, other security groups, and prefix lists
• Security Groups and VMs have a many-to-many relationship

Ports to know for the exam

• 21 = FTP
• 22 = SSH/sFTP
• 80 = HTTP
• 443 = HTTPS
• 3389 = RDP
• 5432 - Postgresql
• 3306 - MySQL
• Oracle RDS - 1521
• MSSQL - 1433
• MariaDB - 3306

Placement Groups

• Use Placement Groups when you want to control how your EC2 instances are scheduled on underlying infrastructure
• Placement Group strategies
  • Cluster
    • Scheduled EC2 instances into a low-latency group in a single Availability Zone
    • Use cases:
      • Big Data job that needs to complete fast
      • Application that needs extremely low latency and high network throughput
  • Spread
    • Pros:
      • Can span across Availability Zones (AZ)
      • Reduced risk is simultaneous failure
      • EC2 instances are on different physical hardware
    • Cons:
      • Limited to 7 instances per AZ per placement group
    • Use cases:
      • Application that needs to maximize high availability
      • Critical Applications where each instance must be isolated from failure from each other
  • Partition
    • Spreads instances across many different partitions (which rely on sets of racks) within an AZ. Scales to 100s of EC2 instances per group

Elastic Network Interfaces

• The ENI can have the following attributes:
  • Primary private IPV4, one or more secondary IPv4
  • One ELastic IP (IPv4) per private IPv4
  • One or more Security Groups
  • A MAC address
• You can create ENI independently and attach them on fly (move them) on EC2 instances for failover
• Bound to a specific availability zone (AZ)
• You can change the Termination Behavior so that if a VM is deleted the attached ENI is/isn’t deleted with it

Spot Instances

• Up to 90% discount
• Specify a max price you are willing to pay for your instances. If you go over the price, you have two options:
  • Two minute grace period
  • Stop the instance or terminate the instance
• If you don’t want AWS to reclaim the capacity, you can use a Spot Block to block AWS from reclaiming the instance for a specified time-frame (1-6 hours)
• The MOST cost-efficient instance pricing
• Useful for workloads that are resilient to failure (batch jobs, etc.)
• Persistent vs. One-Time Spot Requests. With a persistent spot request, if an instance is terminated, it will be restarted. With a one-time spot request, if an instance is terminated, it will NOT be restarted.

Spot Fleets

• Get a set of spot instances + On-Demand instances
• Strategies
  • Lowest Price: Spot Fleet will launch instances from the pool with the lowest price
  • Diversified: distributed across all pools
  • capacityOptimized: pool with optimal capacity for the number of instances
  • priceCapacityOptimized: Pools with highest capacity available, then select the pool with the lowest price

Elastic IPs

• When you start and stop an EC2 instance, the public IP won’t change
• You can only have 5 Elastic IP addresses in your AWS account by default. You can ask AWS to increase this limit.
• Try to avoid using EIP

EC2 Hibernate

• Store the RAM on disk when the OS is stopped.
• Faster startup
• The root EBS volume must be encrypted and it must have enough space to store the contents of RAM
• Instance RAM size must be less than 150GB
• Does not work for bare metal instances

EBS

• Bound to an AZ
• Can be attached/detached from instances on the fly
• EBS volumes can be mounted to multiple instances using ‘multi-attach’
  • Up to 16 instances at a time can be attached to a volume
• You can move an EBS volume across AZ by creating a snapshot and copying it to another region
• Snapshots
  • You can move a snapshot to an ‘archive tier’ that is 75% cheaper
  • Takes 24 to 72 hours to restore the snapshot from the archive
  • Recycle Bin
    • You can setup rules to retain deleted snapshots so you can easily recover them
    • Specify a retention for the recycle bin (from 1 day to 1 year)
  • Fast Snapshot Restore (FSR)
    • Force full initialization of your snapshot to have no latency on first use.
    • Expensive to use
• Encryption
  • data at rest and data in motion are both encrypted
  • all snapshots are encrypted
  • Copying an unencrytped snapshot enables encryption
  • How to encrypt an unencrypted volume
    1. Create a snapshot of the volume
    2. Encrypt the snapshot using the copy function
    3. Create new EBS volume from the snapshot (the volume will be encrypted)
    4. Attach the encrypted volume to the original instance
• Root volumes are automatically deleted (Termination Policy) when a EC2 instance is terminated. Other EBS volumes attached to the instance are not deleted unless their termination policy says to delete them on termination of the EC2 instance.

EC2 Instance Store

• Ephemeral storage
• High performance
• Use cases:
  • buffer
  • cache
• Data loss when the EC2 instance reboots

AMI

• VM Image
• Locked to a region, but can be copied across regions
• Types of AMIs:
  • Public (AWS Provided)
  • Private (created by you)
  • MarketPlace (3rd party vendor)

EBS Volume Types

• Types:

  • gp2/gp3 (SSD): General purpose SSD volume. Balance price and performance
  • io1/io2 Block Express (SSD): Highest performance SSD volume for mission-critical low-latency or high-throughput workloads
  • st1 (HDD): Low cost HDD volume designed for frequently accessed, throughput-intensive workloads
    • Cannot be a boot volume
    • 125GB to 16 TB
  • sc1 (HDD): Lowest cost HDD volume designed for less frequently accessed workloads

• Only GP2/GP3 and IO1/IO2 can be used as a boot volumes

• With GP3, you can independently set IOPS and throughput. With GP2, they are linked together

Provisioned IOPS

• Provisioned IOPS volumes are used for critical business applications with sustained IOPS performance
• Great for database workloads
• io1 Provisioned IOPS:
• If you want to get over 32000 IOPS, you need Nitro 1 or Nitro 2

Auto Scaling Group (ASG)

• Automatically scale out EC2 instances to meet traffic demand. You can scale based on a CloudWatch Alarm (metric), schedule,
• Set a minimum capacity, desired capacity, and max capacity
• The ASG itself is free
• Create a launch template, which specifies how to launch instances within the ASG
• Scaling Policies
  • Dynamic Scaling
    • Target Tracking Scaling
      • Simple, example: keep CPU usage around 50%
      • Target Tracking will create CloudWatch Alarms for you
    • Simple / Step Scaling
      • When a CloudWatch Alarm is triggered, add 2 instances
  • Scheduled Scaling
    • Scale based on a schedule
  • Predictive Scaling
    • Forecast load and scale ahead of time
• Scaling cooldown (default 300 seconds). The ASG will not launch or terminate instances

Elastic Load Balancer

• Spread load of traffic across multiple downstream instances

• Health check downstream instances

• SSL Termination
• High Availability across zones
• Add backend instances to a “Target Group”

Types of ELB

• Application Load Balancer (Layer 7)

  • Allows you to route to multiple instances in a Target Group (aka Backend Pool in Azure)
  • Supports HTTP/2 and websocket
  • Route based on the path in the URL, hostname, query strings, and headers
  • Extra headers added by ALB
    • x-forwarded-for
    • x-forwarded-proto
    • x-forwarded-port
  • ALB has a WAF capability that can be enabled

• Network Load Balancer (Layer 4)

  • High performance, millions of requests per second, and less latency ~100 ms
  • NLB has one static IP address per AZ, and supports assigning an Elastic IP
  • Not compatibly with the free tier

• Gateway Load Balancer (Layer 3)

  • Use cases: Send all traffic to a firewall, IDS, IPS, etc.
  • Supports the GENEVE protocol on port udp/6081

Sticky Sessions

• Same client is forwarded to the same instance, rather than spreading traffic amongst all instances
• Supported by the ALB and NLB
• Cookie is set on the client with has an expiration date you control
  • Cookies:
    • Two types of cookie are supported:
      • Application Based Cookie:
        • Custom cookie:
          • Generated by the target
          • Can include any custom attributes required by the application
          • The cookie name must be specified individually per target group
          • You cannot use AWSALB, AWSALBAPP, or AWSALBTG. These are all reserved by AWS
        • Application Cookie:
          • Generated by the LB itself
          • Cookie will be AWSALBAPP
      • Duration-based Cookie
        • Cookie is generated by the load balancer itself
        • Cookie name is AWSALB for ALB

Cross-Zone Load Balancing

• Each load balancer instance distributes traffic evenly across all registered instances in all AZ
• For the ALB, cross-zone load balancing can be enabled/disabled at the target group level. It is enabled by default and there are no additional charges
• Can be enabled for NLB and GLB, but additional charges will apply. It is disabled by default.

SNI

• Works with ALB, NLB, and CloudFront

Deregistration Delay

• AKA Connection Draining
• Stop sending new requests to the instance that is being deregistered
• Allows the instance to complete in-flight requests before being terminated
• 1 to 3600 seconds (default 300 seconds)
• Can be disabled (set to 0 seconds)
• Set to a low value if your requests are short-lived

Elastic File System (EFS)

• Elastic File System (EFS)
• Introduction to ELastic File System (EFS)

Elastic File System (EFS)

• Scalable, elastic, Cloud-Native NFS File System
• Attach a single file system to multiple EC2 Instances
• Don’t worry about running out or managing disk space

Introduction to Elastic File System (EFS)

• EFS is a file storage service for EC2 instances
• Storage capacity grows (upto petabytes) and shrinks automatically based on data stored (elastic)
• Multiple EC2 instances in same VPC can mount a single EFS Volume (Volume must be in same VPC)
• EC2 instances install the NFSv4.1 client and can then mount the EFS volume
• EFS is using Network File System version 4 (NFSv4) protocol
• EFS creates multiple mount targets in all your VPC subnets
• Pay only for the storage you use, starting at $0.30 GB / month
• You create a security group to control access to EFS
• Encryption at rest using KMS
• 1000s of concurrent NFS clients, 10 GB+ /s throughput
• Grow to petabyte scale storage

EFS Performance Settings

• Performance Mode
  • General Purpose (Default) - latency sensitive use cases
  • Max IO - higher latency, throughput, highly parallel
• Throughput Mode:
  • Bursting - 1 TB storage = 50 MB/s + burst up to 100 MB/s
  • Enhanced - Provides more flexibility and higher throughput levels for workloads with a range of performance requirements
    • Provisioned - set your throughput regardless of storage size
    • Elastic - scale throughput up and down based on workload. Useful for unpredictable workloads

EFS Storage Tiers

• Standard
• Infrequent (EFS-IA)
• Archive
• Implement lifecycle policies to move files between tiers

What is ElastiCache for Redis?

• What is ElastiCache for Redis?
• Comparing Memcached and Redis
• Authenticating with Redis AUTH command

What is ElastiCache for Redis?

• ElastiCache is a web service that makes it easy to set up, manage and scale a distributed in-memory data store or cache environment in the cloud.
• Features:
  • Automatic detection of and recovery from cache node failures
  • Multi-AZ for a failed primary cluster to a read replica in Redis cluster
  • Redis (cluster mode enabled) supports partitioning your data across up to 500 shards
• ElastiCache works with both the Redis and Memcached engines.

Comparing Memcached and Redis

• Redis supports:

  • Multi-AZ with Auto-Failover
  • Read replicas
  • data durability using AOF persistence
  • Backup and restore feautres
  • Supports sets and sorted sets
    • Sorted sets guarantees both uniqueness and element ordering. Useful when creating something like a gaming leaderboard. Each time a new element is added, its ordered automatically.
  • Supports IAM for authentication

• Memcached support:

  • None of what Redis supports
  • Supports SASL based authentication

Authenticating with Redis AUTH command

• Users enter a token (password) on a token-protected Redis server.
• Include the parameter --auth-token (API: AuthToken) with the correct token to create the replication group or cluster.
• Key Parameters:
  • --engine - Must be redis
  • –engine-version - Must be 3.2.6,4.0.10 or later
  • –transit-encryption-enabled - Required for authentication and HIPAA eligibility
  • –auth-token - Required for HIPAA eligibility. This value must be correct token for this token-protected Redis-server
  • –cache-subnet-group - Required fro HIPAA eligibility

Glue-cheatsheet

• A fully managed service to extract, transform and load (ETL) your data for analytics

• Discover and search across different AWS data sets without moving your data

• AWS Glue retrieves data from sources and writes data to targets stored and transported in various data formats

  • If your data is stored or transported in Parquet data format, this document introduces you available features for using your data in AWS Glue

• AWS glue consists of

  • Central metadata repository
  • ETL engine
  • Flexible scheduler

• Use Cases:

  • Run queries against an Amazon S3 data lake
    • You can use AWS Glue to make your data available for analytics without moving your data
  • Analyze the log data in your data warehouse
    • Create ETL transcripts to transform, flatten and enrich the data from source to target

• Integration with AWS Glue

  • To create database and table schema in the AWS Glue Data Catalog, you can run an AWS Glue crawler from within Athena on a data source, or you can run Data Definition Language (DDL) queries directly in the Athena Query Editor.
  • Then, using the database and table schema that you created, you can use Data Manipulation (DML) queries in Athena to query the data.

• Set up AWS Glue Crawlers using S3 event notifications

AWS Lambda

• AWS Lambda
• How it works
• File Processing Architecture
• Stream Processing Architecture
• Use Cases

AWS Lambda

• Run code without thinking about servers or clusters
• Run code without provisioning or managing infrastructure. Simply write and upload code as a .zip file or container image
• Automatically respond to code execution requests at any scale, from a dozen events per day to hundreds of thousands per second
• Save costs by paying only for the compute time you use by per millisecond instead of provisioning infrastructure upfront for peak capacity
• Optimize code execution time and performance with the right function memory size. Respond to high demand in double-digit milliseconds with Provisioned Concurrency.

How it works

• AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers.
• You can trigger Lambda over 200 AWS services and software as a service (Saas) applications and only pay for what you use

File Processing Architecture

Stream Processing Architecture

Use Cases

• Quickly process data at scale
  • Meet resource-intensive and unpredictable demand by using AWS Lambda to instantly scale out to more than 18K vCPUs.
  • Build processing workflows quickly and easily with suite of other serverless offerings and event triggers
• Run interactive web and mobile backends
• Enable powerful ML insights
• Create event-driven applications

Ml-models

• Sage Maker

What is Amazon QuickSight ?

• What is Amazon QuickSight ?
• Benefits

What is Amazon QuickSight ?

• Amazon Quicksight is a very fast, easy-to-use, cloud -powered business analytics service that makes it easy for all employees within an organization to build visualizations, perform ad-hoc analysis, and quickly get business insights from their data, anytime on any device.
• 1/10th the cost of traditional BI Solutions
• With QuickSight all users can meet varying analytic needs from the same source of truth through modern interactive dashboards, paginated reports, embedded analytics and natural language queries

Benefits

• Pay only for what you use
• Scale to tens of thousands of users
• Easily embed analytics to differentiate your applications
• Enable BI for everyone with QuickSight Q
• Can get data insights in minutes from AWS services (e.g. Redshift, RDS, Athena, S3)
• Can choose QuickSight to keep the data in SPICE up-yo-date as the data in the underlying sources change
• SPICE :
  • Amazon QuickSight is built with SPICE - a super-fast, parallel, In-memory calculation Engine.
  • SPICE uses a combination of columnar storage, in-memory technologies enabled through the latest hardware innovations and machine code generation to run interactive queries on large datasets and get rapid responses

Aurora Cheatsheet

Aurora Cheatsheet

• When you need a fully-managed Postgres or MySQL database that needs to scale, automate backups, high availability and fault tolerance think Aurora
• Aurora can run MySQL or Postgres database engines
• Aurora MySQL is 5x faster over regular MySQL
• AUrora Postgres is 3x faster over regular Postgres
• Aurora is 1/10 the cost over its competitors with similar performance and availability options
• Aurora replicates 6 copies for your database across 3 availability zones
• Aurora is allowed up to 15 Aurora Replicas
• An Aurora database can span multiple regions via Aurora Global Database
• Aurora Serverless allows you to stop and start Aurora and scale automatically while keeping costs low
• Aurora Serverless is ideal for new projects or projects with infrequent database usage

RDS

• RDS
• RDS Proxy
• Aurora
• Introduction to Aurora
• Aurora Availability
• Fault Tolerance and Durability
• Aurora Replicas
• Aurora Serverless

RDS

• Relational Database Service
• Database service for database engines that use SQL as a query language
• Engines:
  • Postgresql
  • Mysql / Mariadb
  • Oracle
  • MSSQL
  • IBM DB2
  • Aurora (Proprietary AWS Relational Database)
• You cannot access the underlying compute instances for RDS unless you are using RDS Custom
  • RDS Custom supports Oracle and Microsoft SQL Server
• RDS scales storage automatically
  • You have to set the Maximum Storage Threshold (max amount of storage to use)
  • Supports all RDS engines

Read Replicas

• Scale out read operations
• Create up to 15 read replicas within the same AZ or across AZ’s or across regions
• Replication is asynchronous
• You can promote a read-replica to it’s own database capable of full writes
• Network Costs
  • If the read replicas is in the same region, there is no cost for replication traffic
  • For cross-region replication traffic, there is a cost
• You can setup read-replicas as Multi-AZ for fault tolerance

RDS Multi-AZ

• Mainly used for disaster recovery
• synchronous replication to a standby database
• One DNS name for both databases with automatic failover
• Multi-AZ replicas cannot be read or written to until they are promoted to the primary instance
• Converting from single-AZ to multi-AZ requires no downtime. This can be done in the “modify” section of the RDS database
  • A snapshot is taken and restored into a new standby database. Then a full sync of the database is initiated.

RDS Proxy

• Fully managed database proxy for RDS
• Allows apps to pool and share DB connections established with the database
• Improving database efficiency by reducing the stress on database resources (e.g CPU, RAM)and minimize open connections (and timeouts)
• Serverless, autoscaling, highly available (multi-AZ)
• Reduced RDS and Aurora failover time by up 66%
• Supports RDS (MySQL,PostgreSQL, MariaDB, MS SQL Server) and Aurora (MySQL, PostgreSQL)
• No code changes required for most apps
• Enforce IAM authentication for DB, and securely store credentials in AWS Secrets Manager
• RDS proxy is never publicly accessible
• RDS Proxy is useful for highly scaling lambda functions that open database connections

RDS Backups

• Daily full backup of the database
• Transaction logs are backed up every 5 minutes
• 1 to 35 days of backup retention, set to 0 to disable

Aurora

• Fully managed Postgres or MySQL compatible database designed by default to scale and fine-tuned to be really fast
• Aurora automatically grows in increments of 10GB, up to 128 TB
• 5x performance over MySQL, 3x performance over Postgres

Introduction to Aurora

• Combines the speed and availability of high-end databases with the simplicity and cost-effectiveness of open source databases
• Aurora can run either MySQL or Postgres compatible engines
• Aurora MYSQL is 5x better performance than traditional MySQL
• Aurora Postgres is 3x better performance than traditional Postgres
• Aurora Costs more than RDS (20% more) but is more efficient

Aurora Availability

• 6 copies of your data in 3 AZ:
  • Needs only 4 out of 6 copies for writes (so if one AZ is down then it is fine)
  • Need only 3 out of 6 for reads
  • self healing with peer-to-peer replication
  • Storage is striped across 100s of volumes
• Automated failover for master happens in less than 30 seconds
• Master + up to 15 Aurora Read Replicas serve reads. You can autoscale the read replicas. Clients connect to the “Reader Endpoint”, which will point to any of the read instances

Fault Tolerance and Durability

• Aurora Backup and Failover is handled automatically

• Aurora has a feature called Backtrack that allows you to restore to any point in time without restoring from backups

• Snapshots of data can be shared with other AWS accounts

  

• Storage is self-healing , in that data blocks and disks are continuously scanned for errors and repaired automatically

Aurora Replicas

Aurora Serverless

• Aurora except the database will automatically start up, shut down, and scale capacity up or down based on your application’s needs
• Apps used a few minutes several times per day or week, eg. low-volume blog site
• pay for database storage and the database capacity and I/O your database consumes while it is active

Aurora Backups

• 1 to 35 days rention
• Cannot be disabled
• Point in time recovery
• Manual snaphots
  • Retain manually created snapshots for any amount of time

Aurora Database Cloning

• Clone an existing Aurora database into a new database
• Uses copy-on-write, so it’s very fast

Amazon Redshift

• Amazon Redshift
• What is Data Warehouse ?
• Introduction to Redshift
• Redshift Use Case
• Redshift Columnar Storage
• Redshift Configurations

Amazon Redshift

• Fully managed Petabyte-size Data warehouse .
• Analyze (Run complex SQL queries) on massive amounts of data Columnar Store database

What is Data Warehouse ?

• What is Data Warehouse ?
  • A transaction symbolizes unit of work performed within a database management system
  • eg. reads and writes

Introduction to Redshift

• AWS Redshift is the AWS managed, petabyte-scale solution for Data Warehousing
• Pricing starts at just $0.25 per hour with no upfront costs or commitments.
• Scale up to petabytes for $1000 per terabyte , per year
• Redshift price is less than 1/10 cost of most similar services
• Redshift is used for Business Intelligence
• Redshift uses OLAP (Online Analytics Processing System)
• Redshift is Columnar Storage Database
• Columnar Storage for database tables is an important factor in optimizing analytic query performance because it drastically reduces the overall disk I/O requirements and reduces the amount of data you need to load from disk

Redshift Use Case

• We want to continuously COPY data from

  1. EMR
  2. S3 and
  3. DynamoDB
  • to power a customer Business Intelligence tool

• Using a third-party library we can connect and query Redshift for data.

  

Redshift Columnar Storage

• Columnar Storage stores data together as columns instead of rows

  

• OLAP applications look at multiple records at the same time. You save memory because you fetch just the columns of data you need instead of whole rows

• Since data is stored via column, that means all data is of the same data-type allowing for easy compression

Redshift Configurations

• Single Node
  • Nodes come in sizes of 160Gb. You can launch a single node to get started with Redshift
• Multi-Node
  • You can launch a cluster of nodes with Multi Node mode
• Leader Node
  • Manages client connections and receiving

DNS

• DNS
• Records TTL
• CNAME vs Alias
• Routing Policies
• Configuring Amazon Route 53 to route traffic to an S3 Bucket

DNS

• Domain Name System which translates the human friendly hostnames into the machine IP addresses.
• www.google.com =>172.217.18.36
• Any zone costs 50 cents/month
• Public vs Private hosted zones

Records TTL

• TTL - Time to live
• High TTL - e.g. 24 hr
  • less traffic on Route 53
  • Possibly outdated records
• Low TTL - e.g. 60s
  • More traffic on Route 53 ($$)
  • Records are outdated for less time
  • Easy to change records

• Except for Alias records, TTL is mandatory for each DNS record

CNAME vs Alias

• AWS resources (Load Balancer, CLoudFront..) expose an AWS hostname:
  • lb l-1234.us-east-2.elb.amazonaws.com and you want myapp.mydomain.com
• CNAME:
  • Points a hostname to any other hostname (app.domain.com => blabla.anything.com)
  • You cannot create a CNAME for the Apex record (root domain)
• Alias:
  • Points a hostname to an AWS Resource (app.mydomain.com => blabla.amazonaws.com)
  • WORKS for ROOT DOMAIN and NON ROOT DOMAIN (aka, mydomain.com)
  • Free of charge
  • Native health check
  • Only supported for A and AAAA record types
  • Cannot set alias for an EC2 instance name

Routing Policies

• Simple

  • Typically, the simple type of routing policy will resolve to a single resource
  • If the record resolves to multiple values, the client will choose a random one
  • When using the Alias record type, the record can only resolve to one resource

• Weighted

  • Control the % of the requests that go to each specific resource.
  • Assign each record a relative weight
    • $ \text traffic {(%)} = {\displaystyle \text {weight for a specific record } \over \displaystyle \text {sum of all the weights for all records }} $
    • The sum of the weights of all records does not need to equal 100
  • DNS records must have the same name and type
  • Can be associated with Health Checks
  • Use cases: load balancing between regions, testing new application versions
    

• Latency

  • Redirect to the resource that has the least latency close to us
  • Super helpful when latency for users is a priority
  • Latency is based on traffic between users and AWS Regions
  • Germany users may be directed to the US (if that’s the lowest latency)
  • Can be associated with Health Checks (has a failover capability)

• Failover

• Geolocation

  • Different from latency based
  • This routing is based on user location
  • Should create a “Default” record (in case there’s no match on location)
  • Use cases: website localization, restrict content distribution, load balancing
  • Can be associated with Health Checks

• Geoproximity

  • Route traffic to your resources based on the location of users and resources
  • Ability to shift more traffic to resources based on the defined bias
  • To change the size of the geographic region, specify bias values:
    • To expand (1 to 99)- more traffic to the resource
    • To shrink (-1 to 99)- less traffic to the resource
  • Resources can be:
    • AWS resources (specify AWS region)
    • Non-AWS resources (specify Latitude and Longitude)
  • You must use Route 53 Traffic Flow to use this feature

• Health Checks

  • HTTP Health Checks are only for public resources. You must create a CloudWatch Metric and associate a CloudWatch Alarm, then create a Health Check that checks the alarm
  • 15 global health checkers
  • Health checks methods:
    • Monitor an endpoint
      • Healthy/unhealthy threshold - 3 (default)
      • Interval 30 seconds
      • Supports HTTP, HTTPS, and TCP
      • if > 18% of health checkers report the endpoint is healthy, Route53 considers it healthy.
      • You can choose which locations you want Route53 to use
      • You must configure the firewall to allow traffic from the health checkers
    • Calculated Health Checks
      • Combine the results of multiple health checks into a single health check

Configuring Amazon Route 53 to route traffic to an S3 Bucket

• An S3 bucket that is configured to host a static website
  • You can route traffic for a domain and its subdomains, such as example.com and www.example.com to a single bucket.
  • Choose the bucket that has the same name that you specified for Record name
  • The name of the bucket is the same as the name of the record that you are creating
  • The bucket is configured as a website endpoint

IAM

Groups

• Groups can only contain users, not other groups

User Permissions

• Permission Boundaries can be set for a user account. They control the maximum permissions for the user. This can be helpful to delegate permission management to other users.
• Permissions can be defined on a user account using a built-in policy or by adding the user to a group with defined permissions
• You can create an access key for a user that can be used to access AWS APIs via the CLI, an Application, third party service, etc.
• Permission policies are defined in JSON documents known as IAM policies:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": "ec2:*",
                "Resource": "*",
                "Effect": "Allow",
                "Condition": {
                    "StringEquals": {
                        "ec2:Region": "us-east-2"
                    }
                }
            }
        ]
    }
  

• Modifying custom IAM policies creates a new version of that policy

Password Policy

• You can define a password policy in IAM
• Typical password policy settings

MFA

• MFA Device Options
  • Virtual MFA Device
    • Google Authenticator
    • Authy
  • Universal 2nd Factor (U2F) Security Key
    • Yubikey
  • Hardware Key Fob
    • Provided By Gemalto (3rd party)
  • Hardware Device for AWS GovCloud
    • Provided by SurePassID

Roles

• Used to provide access to AWS services
• For example, provide an EC2 instance access to an S3 bucket

Security Tools in IAM

• Credential Report: Generates a CSV file contains details about user accounts
• Security Access Advisor: Accessible from an individual account in IAM. Shows what AWS services the AWS account is accessing.

AWS Certificate Manager

• Integration with API Gateway
  • Create a custom domain name in API Gateway
  • For edge optimized API Gateways, The TLS certificate must be in the same region as CloudFront
  • For regional API Gateways, The TLS certificate must be imported on API gateway, in the same region as the API Gateway

AWS WAF (Web Application Firewall)

• Protection at Layer 7 of the OSI Model
• Can be deployed on ALB, CloudFront, API Gateway, AppSync GraphQL API, Cognito User Pool
• After deploying the firewall, you create a Web ACL rule:
  • Filter based on IP address, HTTP Headers, HTTP body, URI strings, Message Size, geo-match, and rate-based rules
  • Web ACL’s are regional. Except for in CloudFront where they are global
• How can we get a fixed IP while using WAF with ALB? Use a Global Accelerator in front of the ALB. The Global Accelerator will provide the static IP address, since an ALB cannot have a static IP.

AWS Shield

• Protect from DDoS attacks
• Standard and Advanced SKUs
  • Standard is free and included/enabled on all VPCs
  • Advanced is $3000/month per organization. Protection from more sophisticated DDoS attacks on EC2, ELB, CloudFront, Global Accelerator, and Route 53. Advanced also included 24/7 access to the DDoS Response Team. Shield Advanced will automatically create WAF rules for you.

AWS Firewall Manager

• Manage rules for multiple firewalls in an AWS organization
• Can be used with WAF
• Policies are created at the regional level
• Rules are applied to new resources when they are created automatically

GuardDuty

• Use ML to protect your AWS account
• Uses CloudTrail Event Logs, VPC Flow logs, and DNS Logs. Optional EKS audit logs, RDS and Aurora logs, EBS, Lambda, and S3 data events
• Can setup EventBridge rules to be notified in case of findings.
• Can protect against Crypto Currency attacks

Inspector

• Automated security assessments on EC2 instances. Use AWS SSM Agent to scan the instance
• Automated scans of container images pushed to ACR for CVEs
• Lambda Functions can be scanned for vulnerabilities in code and package dependencies
• Report findings in Security Hub or send findings via EventBridge

Macie

• Use ML and pattern matching to discover and protect sensitive data in AWS in S3
• Notify you through EventBridge when PII is found

Storage-cheatsheet

• Simple Storage Service (S3) Object-based storage. Store unlimited amount of data without worry of underlying storage infrastructure
• S3 replicates data across at least 3 AZs to ensure 99.99% Availability and 11’9s of durability
• Objects contain data (they’re like files)
• • Objects can be size anywhere from 0 Bytes up to 5 Terabytes
• Buckets contain objects. Buckets can also contain folders which can in turn can contain objects
• Bucket names are unique across all AWS accounts. Like a domain name
• When you upload a file to S3 successfully you’ll receive a HTTP 200 code . Lifecycle Management Objects can be moved between storage classes or objects can be deleted automatically based on schedule
• Versioning Objects are given a Version ID. When new objects are uploaded the old objects are kept. You can access any object version. When you delete an object the previous object is restored. Once Versioning is turned on it cannot be turned off, only suspended.
• MFA DELETE enforce DELETE operations to require MFA token in order to delete an object. Must have verioning turned on to use. Can only turn on MFA delete from the AWS CLI. Root Account is only allowed to delete objects
• All new buckets are private by default Logging can be turned to on a bucket to log to track operations performed on objects
• Access Control is configured using Bucket Policies and Access Control Lists (ACL)
• Bucket Policies are JSON documents which let you write complex control access
• ACLs are the legacy method (not depracated) where you grant access to objects and buckets with simple actions
• Security in Transit Uploading is done over SSL
• SSE stands for Server Side Encryption , S3 has 3 options for SSE
• SSE-AES S3 handles the key, uses AES-256 algorithm
• SSE-KMS Envelope encryption via AWS KMS and you manage the keys
• SSE-C Customer provided key (you manage the key)
• Client Side Encryption You must encrypt your own files before uploading them to S3
• Cross Region Replication (CRR) allows you to replicate files across regions for greater durability.You must have versioning turned on in the source and destination bucket. You can have CRR replicate to bucket in another AWS account
• Transfer Acceleration Provide faster and secure uploads from anywhere in the world. Data is uploaded via distinct url to an Edge location. Data is then transported to your S3 bucket via AWS backbone network.
• Presigned Urls is a URL generated via the AWS CLI and SDK. It provides temporary access to write or download object data. Presigned URLs are commonly used to access private objects.
• S3 has 6 different storage classes
  • Standard Fast 99.99% Availability, 11 9’s Durability. Replicated across at least three AZs
  • Intelligent Tiering Uses ML to analyze your object usage and determine the appropriate storage class. Data is moved to the most cost-effective access tier, without any performance impact or added overhead.
  • Standard Infrequently Accessed (IA) n Still fast! Cheaper if you access files less than once a month. Additional retrieval fee is applied. 50 % less than Standard (reduced availability )
  • One Zone IA Still fast! Objects only exist in one AZ. Availability (is 99.5%). but cheaper then standard IA by 20% less (Reduce durability ) Data could get destroyed. A retrieval fee is applied.
  • Glacier For long term cold storage. Retrieval of data can take minutes to hours but the off is very cheap storage
  • Glacier Deep Archive The lowest cost storage class. Data retrieval time is 12 hours

Introduction to S3

• Introduction to S3
• S3 Storage Classes
• Storage class comparison
• S3 Security
• S3 Encryption
• S3 Objects
• S3 Data Consistency
• S3 Cross-Region Replication
• S3 Versioning
• Lifecycle Management
• S3 Transfer Acceleration
• Presigned URLs
• MFA Delete
• AWS Snow Family
• Storage Services
• Storage Gateway
• Amazon FSx vs EFS
• S3 Object Lock
  • Governance mode
  • Compliance mode

Introduction to S3

• What is Object Storage (Object-based storage)?
• data storage architecture that manages data as objects, as opposed to other storage architectures:
  • file systems: which manages data as files and fire hierarchy
  • block storage- which manages data as blocks within sectors and tracks
    • S3 provides with Unlimited storage
    • Need not think about underlying infrastructure
    • S3 console provides an interface for you to upload and access your data
    • Individual Object can be store form 0 Bytes to 5 Terabytes in size
    • Files larger than 5GB must be uploaded using multi-part upload. It’s recommended to use multi-part upload for files larger than 100MB
• Baseline Performance
  • 3500 PUT/COPY/POST/DELETE or 5500 GET/HEAD requests per seconds per prefix in a bucket
  • There are no limits to the number of prefixes in a bucket
  • Example of a prefix
    • bucket/folder1/subfolder1/mypic.jpg => prefix is /folder1/subfolder1/
• S3 Select
  • Use SQL like language to only retrieve the data you need from S3 using server-side filtering

S3 Storage Classes

• AWS offers a range of S3 Storage classes that trade Retrieval, Time, Accessability and Durability for Cheaper Storage

(Descending from expensive to cheaper)

• S3 Standard (default)

  • Fast! 99.99 % Availability,
  • 11 9’s Durability. If you store 10,000,000 objects on S3, you can expect to lose a single object once every 10,000 years
  • Replicated across at least three AZs
    • S3 standard can sustain 2 concurrent facility failures

• S3 Intelligent Tiering

  • Uses ML to analyze object usage and determine the appropriate storage class
  • Data is moved to most cost-effective tier without any performance impact or added overhead

• S3 Standard-IA (Infrequent Access)

  • Still Fast! Cheaper if you access files less than once a month
  • Additional retrieval fee is applied. 50% less than standard (reduced availability)
  • 99.9% Availability

• S3 One-Zone-IA

  • Still fast! Objects only exist in one AZ.
  • Availability (is 99.5%). but cheaper than Standard IA by 20% less
  • reduces durability
  • Data could be destroyed
  • Retrieval fee is applied

• S3 Glacier Instant Retrieval

  • Millisecond retrieval, great for data accessed once a quarter
  • Minimum storage duration of 90 days

• S3 Glacier Flexible Retrieval

  • data retrieval: Expedited (1 to 5 minutes), Standard (3 to 5 hours), Bulk (5 to 12 hours) - free
  • minimum storage duration is 90 days
  • Retrieval of data can take minutes to hours but the off is very cheap storage

• S3 Glacier Deep Archive

  • The lowest cost storage class - Data retrieval time is 12 hours
  • standard (12 hours), bulk (48 hours)
  • Minimum storage duration is 180 days

• S3 Glacier Intelligent Tiering