Cloud2web

—VPC

Can peer 1 VPC with another using private IP addresses. Must peer directly with VPC no hopping

Can have 5 VPC’s per region

Network Access control lists are stateless unlike security groups which are stateful.

You can only assign 1 internet gateway to a VPC

Security groups do not span VPC’s

//Route tables -> Assigned to subnets

default route 0.0.0.0/0 -> Internet gateway

—IGW Internet gateways

Assigned to VPC, can only have 1

—NAT instance

If we do not want to add a default route to a internet gateway

a) Create AMI NAT

b) Disable source/Dst check on the instance

c) Place a default route to NAT-INSTANCE

—NAT Gateway

Should be created for each availability zone

Scales to 10gbps

Route needs to be added to the NAT Gateway

—IAM

Local user database, can be tied to AD

No permissions by default.

Roles -> Allowed EC2 actions

Policies -> permissions for users

AWS Management Console

username + Password

//Programmic languages. Lost after creation, you will need to regenerate the key.

Access Key ID

Secret Access Key

–Instance backed Storage

Hdd stored in the same server – Fastest

Ephemeral, when shut down hard drive gets delete and creates again when started.

—Elastic IP’s

Required if you do not want the public IP to change.

Alternatively you can use dns with an ELB

–EBS-Backed

Hdd stored in elastic cloud storage.

Persistent storage. Can be marked as do not delete root volume.

—Edge Location

Offers caching for content, so that users from the local region, dont need to go all the way to the Datacentre Region

Horizontal vs Vertical Scalling

—EC2

Must be stopped to change resources.

Terminate -> Shutdown and delete

–S3

S3 buckets are configured at a global level,

Communication happens to the edge servers, and communicates back to the region

Can be an website

Can be a file directory 5TB max

Can hold Instance data

Objects go inside a bucket, buckets are associated to accounts.

You can have 100 buckets, each bucket has unlimited storage

S3 Bucket Storage options

Glacier  – Tape backup cheap

Standard – Most expensive

Infrequent Access – Only pay for ppl downloading, retrieval fee each time

Reduced Redundancy storage (RRS) – Less uptime/availability, can use for data you can regenerate, cheaper.

//Region replication

Must be both settings for versioning

only new files are replicated

Copy existing files to the new bucket

1. Create IAM user with programmic access
2. Download AWS cli tools
3. AWS configure -> Login
4. aws s3 cp –recursive s3://bucket1 s3://bucket2

—SSH 

Default username “ec2-user”

—Shared Tenancy

Shared -> Hardware is shared with other companies

Dedicated host -> Own hardware cannot specify hardware specs

Dedicated instance -> Specify a reserved hardware allocation

—Placement Groups

If you place your instances closer together, we can achieve greater throughput performance.

There are 2 different modes. Should be in the same availability zone, should be same instance spec (Compute Graphics, etc)

Required for 10gbps throughput.

-> Cluster (Close together)

-> Spread (Separated hardware)

—Lambda

Run scripts without having server instances.

Blueprints can be used to create templates.

Can be used to report on events from an S3 bucket.

—Auto Scaling

//Scale in – remove an EC2 instance from the pool

//Scale out – Add an EC2 instance to the pool

Manual

Dynamic

Sheduled scaling

Cloudwatch

Auto-Scale groups -> Settings for your instances how many, scale in/out settings

Launch configurations -> Resources, AMI, security, 

—Elastic Load Balancer ELB

Does not have a public IP that is static, must use DNS.

Cross zone availability zones – Can weight 80% traffic to AZ1 20% Traffic to AZ2

Application load balancer -> Used for internal services, not internet facing.

–Cloudfront

Distribute your S3 bucket across all edge locations

Can be media type for Adobe flash streaming files.

—Cloudformation

Graphic gui which builds a JSON script for creating your infrastructure.

If you delete the stack you are deleting your EC2 Instances

Templates deploy EC2, ELB, Security groups, Databases all setup and ready to go!

Make sure you change the security group ports.

—Elastic Container service.

Rather than spin up an EC2 instance, just pay for the service to be generated/hosted.

—Storage Gateways

An AWS Storage VM is installed onsite and communicates to AWS via (Internet,Direct,VPC)

4 Different types of storage

-File Gateway, store NFS files in an S3 bucket

-Tape Library, Integrate your backup solution to be stored in the cloud

-Volume storage iSCUS storage is stored both onprem and in the cloud.

-Cached Storage, iSCUS storage is stored in the cloud, files are cached onsite for commonly accessed files.

—Snowball

Previously called Import Export, users were sending in hard drives to upload large amounts of data to AWS.

Snowball is a secured hard drive to send in terrabytes of data to AWS

Snowball Edge is a secured hard drive with compte functionality that can run lambda, good for Airplanes.

Snowmobile Semi truck that can do peta and exabyte transfers.

—Security Groups

Stateful sessions, even if you delete the outbound rule. Traffic hitting the inbound will be allowed to return.

—EBS storage volumes

Must be in the same availability zones as the instance.

To move a Instance to another availability zone region,

a) Create a snapshot of the root volume

b) Copy snapshot to region/AZ

c) Create a volume from snapshot

For RAID arrays before we can take a snapshot either

-Shutdown instance OR

-Freeze filesrver

-Unmount RAID Array

—Metadata

High level information about an instance which can be queried and stored in a text file

example

curl http://169.254.169.254/latest/meta-data/public-ipv4 > txt.html

—Flowlogs

Can match interesting traffic to the VPC, Subnet or Instance interface to view data logs

//Configuration

1. Create the log group Cloudwatch -> Logs -> Create
2. VPC -> Create flowlogs
3. Create your IAM Role

—Bash

Scripts which run on startup/creatation eg

#!/Bin/bash

yum install httpd -y

—Elastic File System

Storage array which can be mounted on multiple instances.

NOTE: Must have same security group on EFS + INSTANCES (eg Default)

—Databases

//RDS – Multi-AZ for active/standby nodes

SQL

MySQL

Aurora    

– Read-only Replication nodes for scaled IOPS

//dynamoDB – NoSQL Can scale up IOPS without downtime

//Auroa – NoSQL Write nodes and read nodes across availability zones, when the cluster fails dns updates on the read node

//Redshift – Run reports on Databases Single AZ, stores data in collums not rows, Good compression

//Elastacache – Web service that caches items from the database in the memory. saves reading from a slow disk

//DMS – Migrate your oracle Database to another database type.

–Route53

For best results, pay for your DNS domain name through Amazon

Loadbalancing DNS resolution to ELB’s

Simple – Round robbin distribution

Geographic – Different continents/countries go to different ELB’s

Latency – Depending on the latency between you and the server take the shortest path

weighted – Can do 80% ELB1 20% ELB2

Failover – Active/Standby ELB’s. configure a health check to track up/down

SNS – Simple notification service -> Push emails, SMS, HTTP notifications eg when an autoscale group changes. Comprises Topics and Subscribers

SQS – Pull notitications from servers, queue based system which is polled by EC2 Instances.

SFS – Workflow based queues

—Athena and Cloudtrail.

1. Create the table for S3 Bucket in Athena

CREATE EXTERNAL TABLE cloudtrail_logs (

eventversion STRING,

userIdentity STRUCT<

  type:STRING,

  principalid:STRING,

  arn:STRING,

  accountid:STRING,

  invokedby:STRING,

  accesskeyid:STRING,

  userName:STRING,

  sessioncontext:STRUCT<

    attributes:STRUCT<

      mfaauthenticated:STRING,

      creationdate:STRING>,

    sessionIssuer:STRUCT<

      type:STRING,

      principalId:STRING,

      arn:STRING,

      accountId:STRING,

      userName:STRING>>>,

eventTime STRING,

eventSource STRING,

eventName STRING,

awsRegion STRING,

sourceIpAddress STRING,

userAgent STRING,

errorCode STRING,

errorMessage STRING,

requestParameters STRING,

responseElements STRING,

additionalEventData STRING,

requestId STRING,

eventId STRING,

resources ARRAY<STRUCT<

  ARN:STRING,accountId:

  STRING,type:STRING>>,

eventType STRING,

apiVersion STRING,

readOnly STRING,

recipientAccountId STRING,

serviceEventDetails STRING,

sharedEventID STRING,

vpcEndpointId STRING

)

ROW FORMAT SERDE ‘com.amazon.emr.hive.serde.CloudTrailSerde’

STORED AS INPUTFORMAT ‘com.amazon.emr.cloudtrail.CloudTrailInputFormat’

OUTPUTFORMAT ‘org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat’

LOCATION ‘s3://vxxx-auditing-s3/xxx-audit/AWSLogs/919153754842/’;

2. Query the table

SELECT * from cloudtrail_logs WHERE eventtime like ‘2018-07-10%’

OR 

SELECT * from cloudtrail_logs WHERE eventtime like ‘2018-07-10%’

and userIdentity.principalid like ‘%Calem.Blake2%’

Sample lines without full syntax

where eventtime >= ‘2017-01-01T00:00:00Z’

and eventtime >= ‘2017-02-17T00:00:00Z’

and eventtime < ‘2017-02-17T05:00:00Z’);

—Forward Route53 sub-domain to DDNS

1. Create the hosted zone in Route 53 vpn.xxxx.com.au
2. Edit the NS Records to be the following ns1.no-ip.com ns2.no-ip.com ns3.no-ip.com ns4.no-ip.com
3. Edit the SOA Record for No-IP ns2.no-ip.com. hostmaster.no-ip.com. 2036909809 600 300 604800 600
4. Delegate the subdomain in the parent Route53 -> xxx.com.au Create NS Record “vpn.xxx.com.au”  Value ns3.no-ip.com. ns2.no-ip.com. ns1.no-ip.com. ns4.no-ip.com.