BD & BA – ISfBD Practical AWS

BD&BA – ISfBD Practical

Create AWS account

http://aws.amazon.com

Screenshot_2016-04-01_11-13-36.png

Continue with the signup process, choosing the “Basic (Free)” support plan. It will ask for a credit card number, which will not be billed during this course. Once you’re done, send the email address you have used to sign up for AWS to

Stefan.Manegold@cwi.nl

to have charges taken over by ABS.

Accessing AWS console

Select the “US East” region

Configure Firewall

Go to EC2 in AWS Console

First, create a key pair for protected authorized access. Go to “Key Pair

Screenshot_2016-04-01_12-50-28.png

Click on “Create Key Pair” and choose a name for your key pair

READ ALSO
Kubernetes interview questions 2

Screenshot_2016-04-01_12-50-53.png Screenshot_2016-04-01_12-51-32.png

And save the key pair to your local machine following your browser’s download dialog — recall where, i.e., in which directory/location, you / your browser save(s) the text file containing your key pair; you’ll need it later.

Then, go to “Security Groups” and click “Create Security Group

 

The Hue Web interface runs on TCP port 8888, hence we need to create a firewall rule that allows access to this port. Please note that this is not suitable for a production environment. Name and description are of course arbitrary.

Screenshot_2016-04-01_11-34-19.png

Start EMR Cluster

Go to EMR in AWS Console

Create Cluster

READ ALSO
SSH Keys for Amazon EC2 instances

Select “advanced options“. We need to set some configuration.

This is the “Advanced cluster configuration screen“:

Screenshot_2016-04-01_12-37-21.png

Step 1: Software & Steps

Select Amazon Software Release 3.9.0

Screenshot_2016-04-01_12-39-42.png

Select Hue to be installed — most probably this is nowadays done automatically with software release 3.9.0; see the tick with “Hue 3.7.1” in above screenshot.

Step 2: Hardware:

Select instance type (m2.xlarge for now) and cluster size (2 in total, 1 master, 1 core).

Screenshot_2016-04-01_12-41-19.png

Step 3: General Cluster Settings:

Just click “Next

Step 4: Security

First select the key pair you created earlier:

Screenshot_2016-04-01_13-32-53.png

Then assign the security group we created earlier to the Master node:

READ ALSO
Introduction to AWS CLI

Screenshot_2016-04-01_13-36-27.png

Screenshot_2016-04-01_13-35-39.png

This is how this should look like. Note that your security group ID will be different.

Screenshot_2016-04-01_13-37-18.png

And finally, go for launch: click on “Create cluster”.

You will then see a screen like this:

Screenshot_2016-04-01_13-53-59.png

You can check the “Hardware” tab to see instances launching. You can also check the EC2 service to see your instances starting.

Now you need to wait till you see “Waiting”. This can take a while, 15 mins are not unheard of.

While you’re waiting, calculate how much running this cluster will cost per hour

https://aws.amazon.com/elasticmapreduce/pricing/

Hue

Since we opened the Firewall, we can access the Hue web interface directly. The address is the Master public DNS shown above, which will be different in your case. In this example,

READ ALSO
AWS Interview Questions 7

http://ec2-52-28-188-141.eu-central-1.compute.amazonaws.com:8888

You will see the Hue Web interface login screen:

Coming up with an arbitrary (simple, no special characters such as @) user name and a complex enough password is left as an exercise to the reader.

Afterwards, you will see the following screen.

Uploading data

We are going to use the King James version of the Bible as sample data. Download the text file from

http://homepages.cwi.nl/~manegold/ISfBD/kjv.txt

Go to the file browser and upload the file

You will see your file uploaded if all goes well.

Running your first EMR Pig job – Wordcount

We are going to find the most common words in the Bible. To this end, we are going to run an Apache Pig script on the kjv.txt file using MapReduce.

READ ALSO
Migrating AWS Server

 

Enter the following script:

a = LOAD ‘kjv.txt’;

b = FOREACH a GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word;

c = GROUP b BY word;

d = FOREACH c GENERATE COUNT(b) AS ct, group;

e = ORDER d BY ct DESC;

STORE e INTO ‘bible_wordcount’;

Then execute the script (what does it do? Can you tell?) by clicking the triangle:

The script will dutifully start executing and show the log:

Downloading results

Once your job has finished, you can inspect the results using the File Browser again. You will see a new folder, bible_wordcount.

Inside, there are files part-r-…, which have the word counts in it. Tadaa.

Before you shut down, you could look around a bit (if there is time)

  • Check the Hue job log to see the MapReduce jobs created from your Pig script. What were their statistics?
  • Check the EC2/EMR interfaces to see the various performance parameters
  • If you have time, you could try to change the pig script such that only the 100 most frequent words are shown and words with a length smaller than three are omitted.
READ ALSO
AWS Report

Cluster Shutdown

Do not forget to shut down your EMR cluster again, since it will create costs otherwise. Go back to the AWS Web interface, select the EMR service, select your cluster, and click “Terminate”.

Shutdown Verification

It is important to make sure no EC2 instances / EMR clusters at the end. So check the EC2 console to make sure 0 instances are running. Also check the EMR console to make sure no clusters are running.

 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.