BD & BA – ISfBD Practical AWS
Continue with the signup process, choosing the “Basic (Free)” support plan. It will ask for a credit card number, which will not be billed during this course. Once you’re done, send the email address you have used to sign up for AWS to
to have charges taken over by ABS.
Select the “US East” region
Go to EC2 in AWS Console
First, create a key pair for protected authorized access. Go to “Key Pair”
Click on “Create Key Pair” and choose a name for your key pair
And save the key pair to your local machine following your browser’s download dialog — recall where, i.e., in which directory/location, you / your browser save(s) the text file containing your key pair; you’ll need it later.
Then, go to “Security Groups” and click “Create Security Group”
The Hue Web interface runs on TCP port 8888, hence we need to create a firewall rule that allows access to this port. Please note that this is not suitable for a production environment. Name and description are of course arbitrary.
Go to EMR in AWS Console
Select “advanced options“. We need to set some configuration.
This is the “Advanced cluster configuration screen“:
Step 1: Software & Steps
Select Amazon Software Release 3.9.0
Select Hue to be installed — most probably this is nowadays done automatically with software release 3.9.0; see the tick with “Hue 3.7.1” in above screenshot.
Step 2: Hardware:
Select instance type (m2.xlarge for now) and cluster size (2 in total, 1 master, 1 core).
Step 3: General Cluster Settings:
Just click “Next”
Step 4: Security
First select the key pair you created earlier:
Then assign the security group we created earlier to the Master node:
This is how this should look like. Note that your security group ID will be different.
And finally, go for launch: click on “Create cluster”.
You will then see a screen like this:
You can check the “Hardware” tab to see instances launching. You can also check the EC2 service to see your instances starting.
Now you need to wait till you see “Waiting”. This can take a while, 15 mins are not unheard of.
While you’re waiting, calculate how much running this cluster will cost per hour
Since we opened the Firewall, we can access the Hue web interface directly. The address is the Master public DNS shown above, which will be different in your case. In this example,
You will see the Hue Web interface login screen:
Coming up with an arbitrary (simple, no special characters such as @) user name and a complex enough password is left as an exercise to the reader.
Afterwards, you will see the following screen.
We are going to use the King James version of the Bible as sample data. Download the text file from
Go to the file browser and upload the file
You will see your file uploaded if all goes well.
We are going to find the most common words in the Bible. To this end, we are going to run an Apache Pig script on the kjv.txt file using MapReduce.
Enter the following script:
a = LOAD ‘kjv.txt’;
b = FOREACH a GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word;
c = GROUP b BY word;
d = FOREACH c GENERATE COUNT(b) AS ct, group;
e = ORDER d BY ct DESC;
STORE e INTO ‘bible_wordcount’;
Then execute the script (what does it do? Can you tell?) by clicking the triangle:
The script will dutifully start executing and show the log:
Once your job has finished, you can inspect the results using the File Browser again. You will see a new folder, bible_wordcount.
Inside, there are files part-r-…, which have the word counts in it. Tadaa.
Before you shut down, you could look around a bit (if there is time)
- Check the Hue job log to see the MapReduce jobs created from your Pig script. What were their statistics?
- Check the EC2/EMR interfaces to see the various performance parameters
- If you have time, you could try to change the pig script such that only the 100 most frequent words are shown and words with a length smaller than three are omitted.
It is important to make sure no EC2 instances / EMR clusters at the end. So check the EC2 console to make sure 0 instances are running. Also check the EMR console to make sure no clusters are running.