Code & Config – sharing the smart bits

When you decide that you need bigger volume to store your data (in EC2), you do the following in your EC2 console.

Create a snapshot of your current volume. Create new volume from the snapshot. Attach the volume to your instance.
You might need to reboot the instance and mount the volume (/dev/sdf) to you instance. I usually start the mysql afterwords as my mysql data are stored on the volume so system can’t access them when rebooting. You might have better setup so you do not need to mount the dist manually after restart and start the mysql.

Update: There is fix for the solution above – Using Cron read more

Here comes the problem.
If you originally had 1gig volume and you have now created 100gig and use the snapshot of the 1gig the system will thing the dist has still 1gig.

You can resize the disk on the fly with the resize2fs command.
Parameters -p means it will print what is the command doing and -f mean force so it will force to do certain changes which needs to be done. (Google resize2fs for full description of the syntax)

sudo resize2fs -p -f /dev/sdf "SIZEOFYOURDISK"

Note: Replace “SIZEOFYOURDISK” with actual size of your new Volume.

I am including this article into the EC2 section even it does not have anything to do with EC2 services but it is one of those things you will need once your application is up and running.

You will need two services in order to make this work.

Google apps

You will need to set the Google Apps so you can access your email through their interface. I have created support@getquantify.com for my application which sits on domain name http://getquantify.com. You can sign up in here to create your google apps account.

DNS Records

The second step you will need to do is to setup the name servers so your domain name is recognized and the emails know where they should go to. I am using free service for this – http://freedns.afraid.org. Create a free account and follow the instructions bellow.

Create CNAME Record. It should look something like this – mail.getquantify.com (G) CNAME ghs.google.com.
This record will allow you or your users to access the email over domain name mail.getquantify.com. You will need to add this information in google apps inside “Service settings” ->”General” -> “Web address”.

You also need to setup couple of MX records. It will look something like this – getquantify.com (G) MX 20:alt1.aspmx.l.google.com

Google apps will provide you with all the addresses you will need for your MX records. I have got 8 of them. You will find all the necessary information inside “Serice settings” -> “How to activate email” -> “Set up email delivery” -> “Change MX records”

In this example I am using Ubuntu server with Apache Passenger.

Get an free account on http://freedns.afraid.org. You will see four name servers you can use to point your domain name to. You need to login to the service where you bought your domain name. In mine case I bought my domain name at godaddy.com, so I set the name servers over their web interface.

In the subdomain section setup A records for your domain name (http://domainname.com and http://www.domainname.com). Points both domain names into your Elastic IP Address.

If you want to run your test application on the same server set another A record for domain name http://test.domainame.com or http://staging.domainname.com and point them to the same Elastic IP Address.

In the next step you will need to configure your Apache Passenger.
There are two files where you need to add Virtual host settings.

Create a separate file for each of your domains and set name of the file to the name your domains (domainname.com and staging.domainname.com) in the following folder /etc/apache2/sites-available/ and paste this code inside. Do not forget to adjust the server IP address and paths.

<VirtualHost *:80>
    ServerName ‘your_elastic_ip_address’
    DocumentRoot ‘path_to_your_apps_public_folder’  // In my case I store my app on the elastic block in this folder /mnt/data/quantify/test/current/public
    <Directory ‘path_to_your_apps_public_folder’ >
        Allow from all
        Options -MultiViews
    </Directory>
</VirtualHost>

You will also need to add this virtual host block into the Apache configuration file which you will find in the following folder /etc/apache2

You will also need to tell the test server to use the Test rails environment setting.
You need to add “Raisl_Env = test” to apache config file.

Setting up backup policy on for EC2 servers is a three step process. You will need to backup your latest setup on EC2 instance, your files on Elastic Block and most importantly the customer – applicaton data in your database.

Creating backup of EC2 instance

You can backup your EC2 instance white easily Amazon EC2 Tools which. The tools consist of useful scripts which are easy to use, you only need to pass the required parameters. The process consist of three steps again. Number one – create a backup of the instance into the tmp directory on the server itself. Number two – upload the backup into Amazon S3. Number three – Register your own AMI based on the S3 backup.

Number 1 – Create backup of your instance

You might need to create tmp directory in /mnt. You will also need to save your amazon pk and cert keys into a folder on your server. I have used root in my sample code.

sudo ec2-bundle-vol -d /mnt/tmp -k /root/pk-yourpkcodewillcomehere.pem --cert /root/cert-yourcekcodewillcomehere.pem -u yourAWSaccounnumber

Do not forget to replace ‘pk-yourpkcodewillcomehere.pem’ with your pem kode, ‘cert-yourcekcodewillcomehere.pem’ with your cert code and ‘yourAWSaccounnumber’ with your AWS account number.

Number 2 – Move your backup to S3 server

sudo ec2-upload-bundle -b yourS3backup name -m /mnt/tmp/image.manifest.xml -a privatecode -s secretcode

Do not forget to replace ‘yourS3backup’ with your S3 backup name, ‘privatecode’ & ‘secretcode’ with your actual keys.

Number 3 – Register your AMI

Login to your Amazon Management Console. Click on AMIs link in the side bar and then click the button ‘Register NEW AMI’ you will see popup with following text – http://s3.amazonaws.com:80/. You will need to supply your bucket name and name of the xml manifest. (bucketname//image.manifest.xml).

Creating backup of Elastic Volume

Open your Amazon Management Console and click on the Volumes linke in the side bar and then click the button ‘Create Snapshot’. That is all to it. You will need to specify the name for the snapshot.

Creating Database backup

You can setup a cron job to backup the database every hour. You will need to create a script which will be called by the cron.

Lets explain what the script does. It will dump the mysql database into a sql file. It will create a variable with a date and time so every backup file has a unique name. It will compress the sql file into tgz file. It will remove the sql afterwords as it will not be needed anymore.
It will use the s3-bash script to push the backup file into your Amazon S3 bucket. It will remove the compress file from your server as it will not bee needed anymore.

You will need to download the s3-bash.0.02 tools from Google repository. As you can see in the example bellow I have stored them in the /mnt/data/scripts/ directory.

#!/bin/sh
mysqldump -uroot -p'password' --opt 'your_database_name' > /mnt/data/mysql_backup/'your_databse_name'.sql
dtb=dtb-$(date +%Y%m%d-%H%M%S).tgz
cd /mnt/data/mysql_backup
tar -zcvf $dtb 'your_databse_name'.sql
rm 'your_databse_name'.sql
/mnt/data/scripts/s3-bash.0.02/s3-put -v -k'your_public_key' -s 'path_to_your_secret_key' -T /mnt/data/mysql_backup/$dtb /'your_bucket_name'/$dtb
rm $dtb

As you can see I am storing the backup file temporarily in /mnt/data/mysql_backup/ but you can chose any folder.
Make sure your replace the values in quotes with your own terms. Please note that you will need to create a file with your secret key and not place the secret key directly as a variable. I advice you not to use vi editor when creating this file as the file needs to be exactly 40 characters and vi editor will add hiden character so you end up with 41 characters and the script will not work. I have used Coda editor to create this file and it worked well.

Once you create this file you will need to setup crontab so the file will be executed every hour. You can edit your crontab using following command

crontab -e

You will need to add the following code

00 */1 * * * /mnt/data/scripts/'nameofyourscript'

Do not forget to replace ‘nameofyourscript’ with the actual name of your script and adjust the path if you have stored the script in different location, as you can see I have stored it in the scripts folder on my elastic block /mnt/data/scripts/. The crontab syntax means that the script will be executed every hour. In general it follows the order minute hour day month…etc. Search google for crontab to get more details.

Please not that in my example the database is not going to be huge so I can afford to dump the whole database every hour but you might need better solution such is incremental backup if your database is going to be huge.

This article will guide you through the process of installation of the Amazon EC2 instance and elastic block including: Ubuntu, Mysql, Ruby on Rails, Fussion Passenger, Git, Gems

First you need to pick on of the AMI to for your instance. I have chosen ami-5059be39. Using Amazon Management Console assign the instance elastic IP Address and create elastic block.

Then instance is place for you server software such are Ubuntu and elastic block acts as your hard-rive.

You need to be able to SSH into your instance to be able to install additional software, your app and so on. You can do this by executing the following command from your machine.

ssh -i your-ssh-key.pem ubuntu@your-elastic-ip-address

Mount your elastic block into your instance running these commands while ssh connected to your instance.

yes | mkfs -t ext3 /dev/sdf
mkdir /mnt/data
mount /dev/sdf /mnt/data

Now we will need to install all the additional software which will make our life easier. This process will install apache, ruby, mysql server, and necessary libraries, subversion and git, ruby gems, Passenger Phusion.

The best way to do this is to create file called setup.sh in the folder /home/ubuntu and insert inside all this code. (you can read more details about this process on the Design by Engineers site)

# Setup Ruby on Rails, MySql, Apache + Passenger
# on Amazon Ubuntu instance (ami-5059be39 – ubuntu-intrepid)
# 10/09/09

echo Updating system...
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install build-essential
sudo apt-get install curl libcurl4-openssl-dev
sudo apt-get install zlib1g-dev libssl-dev libexpat1-dev

echo Installing tools...
sudo apt-get install apache2 apache2-threaded-dev
sudo apt-get install ruby ri rdoc ruby1.8-dev irb libreadline-ruby1.8
sudo apt-get install libruby1.8 libopenssl-ruby libopenssl-ruby1.8
sudo apt-get install mysql-server libmysqlclient15off mysql-client-5.0
sudo apt-get install mysql-common mysql-server-5.0 libmysqlclient-dev
sudo apt-get install libmysql-ruby libmysql-ruby1.8 psmisc

sudo apt-get install subversion
sudo apt-get install git-core gitweb

echo Installing ruby gems...
cd /usr/local/src
sudo wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
sudo tar -zvxf rubygems-1.3.5.tgz
cd rubygems-1.3.5/
sudo ruby setup.rb

echo Symlinking...
sudo ln -s /usr/bin/gem1.8 /usr/local/bin/gem
sudo ln -s /usr/bin/ruby1.8 /usr/local/bin/ruby
sudo ln -s /usr/bin/rdoc1.8 /usr/local/bin/rdoc
sudo ln -s /usr/bin/ri1.8 /usr/local/bin/ri
sudo ln -s /usr/bin/irb1.8 /usr/local/bin/irb

echo Installing Gems...
sudo gem install sys-proctable --no-rdoc --no-ri
sudo gem install rails --no-rdoc --no-ri
sudo gem install mysql --no-rdoc --no-ri
sudo gem install capistrano rspec rdoc --no-rdoc --no-ri
sudo gem install passenger --no-rdoc --no-ri
sudo gem install haml --no-rdoc --no-ri
sudo gem install hpricot --no-rdoc --no-ri
sudo gem install json --no-rdoc --no-ri
sudo gem install newrelic_rpm --no-rdoc --no-ri
sudo gem install sqlite3-ruby --no-rdoc --no-ri
sudo gem install will_paginate --no-rdoc --no-ri
sudo gem install authlogic --no-rdoc --no-ri
sudo gem install cached_model --no-rdoc --no-ri

echo Installing Phusion Passenger
sudo passenger-install-apache2-module

Now run this command to execute this file…

sudo bash ./setup.sh

You will be asked to create mysql password during the installation process. Write it down, you will need it for the configuration of your application. You will also need to confirm couple of steps during the installation.

We now need to configure apache so it runs Passenger for our rails application. You need to add this few lines at the end of your apache configuration file which you will find in /etc/apache/apache2.conf

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.2.11/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.11
PassengerRuby /usr/bin/ruby1.8

You will also need to add these few lines to tell Passenger where is your application sitting.
Replace “your-elastic-ip-address” with your elastic IP address
Replace “your-path” with the real path where your app will sit. You will need to create the folder structure before you set this path otherwise the Apache will complain that the path does not exist.

<VirtualHost *:80>
    ServerName your-elastic-ip-address
    DocumentRoot /mnt/data/your-path
    <Directory /mnt/data/your-path>
        Allow from all
        Options -MultiViews
    </Directory>
</VirtualHost>

The same virtual host configuration need to be added to the file /etc/apache2/sites-available/default

Restart your server…

sudo /etc/init.d/apache2 restart

At this point I have deployed my application with Capistrano. There will be a separate post how to set Capistrano to deploy on EC2 server.

In case you want to double check free space on your hard-drive you can execute this command

df -h

The most difficult part for me was to setup mysql.

You might need to check the system errors during this process so you know what is going wrong. You can see the errors by executing

sudo tail -50 /var/log/syslog

Create new folder for your msq files called mysql_data inside /mnt/data/ folder (your elastic block) then assign the right privileges to that folder for mysql user and group. Stop the server, move the original mysql files, delete the original folder and create symlink to your new directory. Restart mysql

mkdir /mnt/data/mysql_data
chown mysql:mysql  /mnt/data/mysql_data
/etc/init.d/mysql stop
mv /var/lib/mysql/* /mnt/data/mysql_data
rmdir /var/lib/mysql
ln -s /mnt/data/mysql_data mysql
/etc/init.d/mysql start

You will need to adjust configurations for mysql apparmor file by executing

vi /etc/apparmor.d/usr.sbin.mysqld

adding these few lines…
As you can see these are my paths /mtn/data/mysql_data you can put your files in different folder structure inside mnt folder, change the paths accordingly. You will find more details about this process on Article my-addr

  /mnt/data/mysql_data/ r,
  /mnt/data/mysql_data/** rwk,
  /mnt/data/mysql_data/ r,
  /mnt/data/mysql_data/** rw,

Restart apparmor

/etc/init.d/apparmor restart

This was all what was necessary to setup mysql for most of the users but not in my case, I got errors and mysql was not running. I found that these few lines has fixed my problem.

mysql_install_db --user=mysql --ldata=/new-data-location
mysqld_safe --datadir=/new-data-location --user=mysql

I hope it could help someone because I have spent long time configuring the EC2 server. There are still some tasks you will need to do after your server is up and running. I will talk about these in later posts. I mean things like backing up your instance, backing up your elastic block and setting up cron job to backup your database regularly.

1. Launch instance – choose a package of software you want to run (eg rails, mogrel, mysql). This is where all your environment sits. There are many predefined images you can launch into your instance. There are even packages which include monitoring tools, memcache and so on. Anytime you alter and install and new software you will need to create AMI (Amazon Machine Image). This will be your backup just in case something goes wrong with your instance. You will be able to launch the latest configuration form this image. If you do not do it you will lose the whole server configuration and all the stuff you have installed.

2. Launch Volume and attached it to your instance. This is the place where your data will sit. Your application files, database and so on. You need to mount the volume into the instance after you attach it.

yes | mkfs -t ext3 /dev/sdh
mkdir /mnt/data-store
mount /dev/sdh /mnt/data-store

You should be regularly creating snapshots of this volume. Again. If something goes wrong you can launch all your data from latest snapshot. The snapshots are stored on Amazon S3 servers. Probably the cheapest backup options out there.

3. Alocate your Elastic IP address. This provide you with huge flexibility of switching IP address without a need of waiting for DNS records to change.

4. Use your keypair to ssh to your instance.

ssh -i yourkey.pem root@yourinstancename.compute-1.amazonaws.com

Note: I went for reserved instance. It requires payment ahead for 1 or 3 years but it reduces the cost by 30 % for monthly payments.