My AWS Scribbles

Friday, 1 July 2016

Resizing a filesystem in Linux to reflect new EBS volume size

Resizing the filesystem on linux

[root@ip-10-0-0-242 opt]# fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/xvda: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt


#         Start          End    Size  Type            Name
 1         4096     16777182      8G  Linux filesyste Linux
128         2048         4095      1M  BIOS boot parti BIOS Boot Partition

Disk /dev/xvdb: 536.9 GB, 536870912000 bytes, 1048576000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[root@ip-10-0-0-242 opt]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  2.9G  4.8G  38% /
devtmpfs         15G   64K   15G   1% /dev
tmpfs            15G     0   15G   0% /dev/shm
/dev/xvdb       197G  173G   15G  93% /vol


The /dev/xvdb still shows 200GB even though it has been upgraded to 500 GB (fdisk)

Following are the commands to resizing the filesystem


[root@ip-10-0-0-242 opt]# mount
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
/dev/xvda1 on / type ext4 (rw,noatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=15427064k,nr_inodes=3856766,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime)
/dev/xvdb on /vol type ext4 (rw,noatime,data=ordered)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)

[root@ip-10-0-0-242 opt]# mkfs.ext4 /dev/xvdb
mke2fs 1.42.12 (29-Aug-2014)
/dev/xvdb contains a ext4 file system
 last mounted on /vol on Fri Jul  1 10:59:32 2016
Proceed anyway? (y,n) y
/dev/xvdb is mounted; will not make a filesystem here!

[root@ip-10-0-0-242 opt]# umount /dev/xvdb
[root@ip-10-0-0-242 opt]# resize2fs /dev/xvdb
resize2fs 1.42.12 (29-Aug-2014)
Please run 'e2fsck -f /dev/xvdb' first.

[root@ip-10-0-0-242 opt]# e2fsck -f /dev/xvdb
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/xvdb: 73444/13107200 files (2.1% non-contiguous), 46060290/52428800 blocks

[root@ip-10-0-0-242 opt]# resize2fs /dev/xvdb
resize2fs 1.42.12 (29-Aug-2014)
Resizing the filesystem on /dev/xvdb to 131072000 (4k) blocks.
The filesystem on /dev/xvdb is now 131072000 (4k) blocks long.

[root@ip-10-0-0-242 opt]# mount -a
[root@ip-10-0-0-242 opt]# mount
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
/dev/xvda1 on / type ext4 (rw,noatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=15427064k,nr_inodes=3856766,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
/dev/xvdb on /vol type ext4 (rw,noatime,data=ordered)

[root@ip-10-0-0-242 opt]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  2.9G  4.8G  38% /
devtmpfs         15G   64K   15G   1% /dev
tmpfs            15G     0   15G   0% /dev/shm
/dev/xvdb       493G  173G  295G  37% /vol
[root@ip-10-0-0-242 opt]#

Wednesday, 8 June 2016

AWS - Things to remember

DynamoDB
Query:

DynamoDB supports the Query operation on table, a local secondary index, or a global secondary index that have a composite primary key (partition key and sort key). You can also filter query results.
result set size limit of 1 MB
Global secondary indexes support eventually consistent reads only

BatchGetItem:

returns the attributes of one or more items from one or more tables
can retrieve up to 16 MB of data, which can contain as many as 100 items

S3:

100 buckets per account (not region)

Bucket Naming convention:

Bucket names must be at least 3 and no more than 63 characters long.
Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.). Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number.
Bucket names must not be formatted as an IP address (e.g., 192.168.5.4).
When using virtual hosted–style buckets with SSL, the SSL wildcard certificate only matches buckets that do not contain periods. To work around this, use HTTP or write your own certificate verification logic. We recommend that you do not use periods (".") in bucket names.
No Upper case characters allowed

ELB:
Request Routing

Before a client sends a request to your load balancer, it resolves the load balancer's domain name using a Domain Name System (DNS) server. The DNS entry is controlled by Amazon, because your instances are in theamazonaws.com domain. The Amazon DNS servers return one or more IP addresses to the client. These are the IP addresses of the load balancer nodes for your load balancer. As traffic to your application changes over time, Elastic Load Balancing scales your load balancer and updates the DNS entry. Note that the DNS entry also specifies the time-to-live (TTL) as 60 seconds, which ensures that the IP addresses can be remapped quickly in response to changing traffic.

The client uses DNS round robin to determine which IP address to use to send the request to the load balancer. The load balancer node that receives the request uses a routing algorithm to select a healthy instance. It uses the round robin routing algorithm for TCP listeners, and the least outstanding requests routing algorithm (favors the instances with the fewest outstanding requests) for HTTP and HTTPS listeners.

The cross-zone load balancing setting also determines how the load balancer selects an instance. If cross-zone load balancing is disabled, the load balancer node selects the instance from the same Availability Zone that it is in. If cross-zone load balancing is enabled, the load balancer node selects the instance regardless of Availability Zone. The load balancer node routes the client request to the selected instance using its private IP address.

Ref: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/how-elb-works.html

Monday, 23 May 2016

Automatically assign EIP via Lambda on launch of an ASG instance

Pre-requisite: EIP Allocation Id.

The following lambda code automatically assigns a previously created EIP to a new instance launched in a Autoscaling group. You need to define "Lifecycle hooks" for an ASG for the lambda function to be triggered. To view the lifecycle hook:

-> ~ aws autoscaling describe-lifecycle-hooks --auto-scaling-group-name Lambda_EIP_TEST
{
    "LifecycleHooks": [
        {
            "GlobalTimeout": 172800, 
            "HeartbeatTimeout": 3600, 
            "AutoScalingGroupName": "Lambda_EIP_TEST", 
            "LifecycleHookName": "Lambda-OpenVPN-Hook", 
            "DefaultResult": "ABANDON", 
            "LifecycleTransition": "autoscaling:EC2_INSTANCE_LAUNCHING"
        }
    ]
}

Lambda code:

console.log('Loading function');
var AWS = require('aws-sdk');

exports.handler = function(event, context) {
console.log('AutoScalingLaunchTerminate()');
console.log('Here is the event:', JSON.stringify(event, null, 2));
console.log('Event.Instance Id: ', event.detail.EC2InstanceId);

var Lifecycleparams = {
   AutoScalingGroupName: event.detail.AutoScalingGroupName, /* required */
   LifecycleActionResult: 'CONTINUE', /* required */
   LifecycleHookName: event.detail.LifecycleHookName, /* required */
   LifecycleActionToken: event.detail.LifecycleActionToken
};
var autoscaling = new AWS.AutoScaling();
console.log('Before completeLifeCycleAction .....:', Lifecycleparams.LifecycleActionToken);
autoscaling.completeLifecycleAction(Lifecycleparams, function(err, data) {
   if (err) {
      console.log(err, err.stack);
   } // an error occurred
   else {
      console.log(data);
      var params = {
         AllocationId: 'eipalloc-d65c42b3',
         AllowReassociation: true,
         DryRun: false,
         InstanceId: event.detail.EC2InstanceId
   };
   var ec2 = new AWS.EC2();
   console.log('Before associateAddress .....:', params.InstanceId);
   ec2.associateAddress(params, function(err, data) {
      if (err) {console.log(err, err.stack);} // an error occurred
      else { console.log(data); context.succeed('Ready');} // successful response
   });
   console.log('After associateAddress .....:', params.InstanceId);
   } // successful response
});
console.log('After completeLifeCycleAction .....:', Lifecycleparams.LifecycleActionToken);
};

Make sure that the Lambda function has access to the AWS resources (execution role: http://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html). In this case it is autoscaling CompleteLifecycleAction and EC2 AssociateAddress. During the creation of the Lambda function you have an option to choose or create a one-click IAM role.
Another thing to note is the association of the lambda function with a VPC. By default, lambda function runs in a default AWS VPC. You can also choose to run in your provided VPC. In this case, choose "No VPC", which will execute the function in the AWS VPC.

Event

The lambda function get executed on an Event. There are loads of events on which lambda can be triggered. In this case, it is the EC2 Launch event of autoscaling. For this to work, you need to enable ASG Lifecycle hooks. Follow steps in Scenario 4 of the Lambda example to configure the ASG Event rule and associate it with the Lambda function.

Logs:

START RequestId: c1a18c88-1cef-11e6-bf82-394cfdb4d343 Version: $LATEST
2016-05-18T11:57:55.397Z c1a18c88-1cef-11e6-bf82-394cfdb4d343 AutoScalingLaunchTerminate()
2016-05-18T11:57:55.397Z c1a18c88-1cef-11e6-bf82-394cfdb4d343 Here is the event:
{
"version": "0",
"id": "a2e7cb9c-8c6f-4ec3-9bc1-161122b32718",
"detail-type": "EC2 Instance-launch Lifecycle Action",
"source": "aws.autoscaling",
"account": "XXXXXXXXX",
"time": "2016-05-18T11:57:54Z",
"region": "eu-west-1",
"resources": [
"arn:aws:autoscaling:eu-west-1:XXXXXXXX:autoScalingGroup:38c88f38-9b16-4919-a641-1a2c2901b2d1:autoScalingGroupName/Lambda_EIP_TEST"
],
"detail": {
"LifecycleActionToken": "31089eef-6611-4cdb-8ad3-2e9a1abc3508",
"AutoScalingGroupName": "Lambda_EIP_TEST",
"LifecycleHookName": "Lambda-OpenVPN-Hook",
"EC2InstanceId": "i-80931d0c",
"LifecycleTransition": "autoscaling:EC2_INSTANCE_LAUNCHING"
}
}
2016-05-18T11:57:55.397Z c1a18c88-1cef-11e6-bf82-394cfdb4d343 Event.Instance Id: i-80931d0c
2016-05-18T11:57:55.398Z c1a18c88-1cef-11e6-bf82-394cfdb4d343 Before completeLifeCycleAction .....: 31089eef-6611-4cdb-8ad3-2e9a1abc3508
2016-05-18T11:57:55.405Z c1a18c88-1cef-11e6-bf82-394cfdb4d343 After completeLifeCycleAction .....: 31089eef-6611-4cdb-8ad3-2e9a1abc3508
2016-05-18T11:57:55.625Z c1a18c88-1cef-11e6-bf82-394cfdb4d343 { ResponseMetadata: { RequestId: 'c1c4565f-1cef-11e6-8f64-cd1610bf3f3a' } }
2016-05-18T11:57:55.805Z c1a18c88-1cef-11e6-bf82-394cfdb4d343 Before associateAddress .....: i-80931d0c
2016-05-18T11:57:55.863Z c1a18c88-1cef-11e6-bf82-394cfdb4d343 After associateAddress .....: i-80931d0c
2016-05-18T11:57:56.179Z c1a18c88-1cef-11e6-bf82-394cfdb4d343 { AssociationId: 'eipassoc-ca9bbcad' }
END RequestId: c1a18c88-1cef-11e6-bf82-394cfdb4d343
REPORT RequestId: c1a18c88-1cef-11e6-bf82-394cfdb4d343 Duration: 782.79 ms Billed Duration: 800 ms Memory Size: 128 MB Max Memory Used: 57 MB

NOTE: In case you are not using the inline Node.js editor on the AWS console and have a CI setup which stores the code in GitHub or any other CVS and you then reference the code in Lambda via a ZIP file, MAKE SURE to set the HANDLER to the name of the ZIP.

Eg. If the zip file being uploaded via Terraform or AWS CLI is "assign-eip-eni.zip" then the handler should be set to: "assign-eip-eni.handler".