AWS, Drupal and Caching: pt.2 Create the web node
Oct. 15, 2014So, where do we start on all this now we have a plan? Other than wishing I had spent time to draw a colorful 3D architectural diagram, we need to just create some accounts, and get stuck in there. Let's start making mistakes, and evolving our server.
Step 1: Create your account
head on over to http://aws.amazon.com/ and create yourself an account. Once done, head on over to your new console to see all the things at your fingertips console.aws.amazon.com .
Now, before we look around, I should raise a quick gotcha to you. AWS servers are located across the world, as I am sure the term cloud made you aware. This means any things you tweak happen in a particular country. Therefore, the first few times you start playing, look in the top-right corner of the console, and set the region dropdown to your region of choice (mine being Ireland). Once done, have a click around, and read up on some of the tools available, you may find them useful in the future. Pay particular attention to things like cloudfront and s3, which I wont be going over in this series (yet).
We are going to start with clicking on EC2 (Elastic Compute Cloud) which will give us a base system which you may recognise as is the base for many a system. It is a basic system, with an OS, which you can install stuff on to run as your web server. This may be all you need, and tbh in our case, we will put most of our stuff onto this box to reduce cost, and because I want to see how much caching can save us cost-wise as opposed to the usual performance reasons.
Step 2: Create the base
Once you are on the EC2 page (https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#), click on the big blue button marked 'launch instance'. From here I clicked on 'AWS Marketplace' and just searched for the term 'LAMP' as this seemed simple enough. This produced a few options, the first to catch my eye (due to the nice logo) was the Bitnami one, I tried this out BUT there were a few issues for me:
- Bitnami is a paid service, and unless you pay, you don't really gain anything other than another abstraction layer from your server via their GUI
- It is on Centos (and I was after Debian)
I gave the server a go, but ultimately removed it and rolled over to the Turnkey build, which was the right OS (Debian), and was structured in a way which I was more attune to.
There are a few steps to get this working from here, but they are all presented via the Amazon UI (such as choosing instance size, I went for t1.micro), once done it will ultimately tell you to 'review and launch' your server. When asked about a KeyPair, create a new one (I called mine testpair), but ensure you keep the *.pem file somewhere safe, you will need it to access the box.
Step 3: Getting into your box
To ssh into the box, you will need to have saved the key for your box to your local machine and run this using:
ssh -i <path to pem file> <ip to access>
To get the IP for your new box, go to your amazon console, find your ec2 instance and click on it. In the bottom info area, you should see your IP on the right (we wont be setting up your EIP until towards the end btw, so be aware that if/when you turn off your box at the end of each day, you will lose this IP). Note down your IP somewhere, as you will likely need it fairly often at the moment. Once you have ssh'd in, you will go through some one-off settings pages, just remember all the details you put in here, as they may be needed later.
Note: that I also skipped the hub initialisation, as that requires a turnkey paid account for the useful services.
Another note: Ooh, automatic security updates, now that is both handy and risky (risk because it could cause my site(s) to break without notice if we aren't careful. Something to be aware of.
Once done, quickly confirm you can ssh back into your box (as it will have exited you).
Step 3b: Quick alias
So long as you can, we will do one last task to make things easier for you, although this may just be for UNIX users. Edit your bashrc (or basprofile on a mac) file (sudo vim ~/.bashrc) and add something like this into it:
alias testaws="ssh -i /home/dan/working_files/testpair.pem root@<your ip address>"
Once done, save the file, and refresh bash (source ~/.bashrc). Now you should be able to just type in testaws on your terminal, and voila, you are into your new box
Step 4: Test basicapache
Shall we see if this is working? Edit your local machines /etc/hosts file and add a new line in there, the beginning of this line is the IP of your new server, and the end is the Domain you want to redirect to this IP (eg. 12.345.678.90 test.yoursite.com). Now go back into your web browser, put this URL in, and you should see a new website, with some nice Turnkey icons, implying this is working. Note the webmin, it gives you some nice UIs for a lot of things, we won't be using them for these articles, but they may provide shortcuts for you, so after this is done, consider hooking up the webmin to a secure address for you to use.
So now what, are we done? Well, actually, we are very close to having a base site, but the next few steps may take a while to get working properly. But let's see what we can do.
Step 5: Clone/checkout your code
I do hope you are using git. Assuming you are, ssh into your shiny new box, and go to your www directory (cd /var/www/) you will see the webmin html in there, so quickly move that to a new directory (cd ../; mkdir webmin; mv www/* webmin/; mv webmin www/; cd www; ls -la) . Now git clone your application into a new directory. (Okay fine, if you are using svn, then first runapt-get install subversion).
Once all the code is checked out, you will likely want to set the owner of your whole site to www-data
sudo chown www-data:www-data -R <your sites directory>
Step 6: Apache conf
Now the code is there, we need to point apache at it. The nice tidy way to do this is to get to your available sites directory (cd /etc/apach2e/sites-available) and duplicate one of the conf files there, renaming it to use the domain for your new site. I did not like the defaults though, so here is an example of the one I am using on this test:
<VirtualHost *> ServerName retrobadger.net ServerAlias testaws.retrobadger.net DocumentRoot /var/www/retrobadger.net/public RewriteEngine On <Directory /var/www/retrobadger.net/public> Options FollowSymLinks AllowOverride All Order allow,deny Allow from all </Directory> LogLevel info ErrorLog /var/log/apache2/retrobadger.net-error.log CustomLog /var/log/apache2/retrobadger.net-access.log combined RewriteEngine On RewriteLog /var/log/apache2/retrobadger.net-rewrite.log # level 0 => Do not log rewrite RewriteLogLevel 0</VirtualHost>
Then just edit this file in your chosen editor (I am a VIM man myself) and you should be away. Once done, save the file, and now go to the folder with your enabled sites (cd /etc/apach2/sites-enabled) from here we want to create a symlink to that new conf file (ln -s /etc/apaches/sites-available/yoursite.com.conf yoursite.com.conf). Whilst there, delete the symlink to the default.conf file, we don't need it, and it will cause conflicts when we restart otherwise.
Once this is done, restart apache, and hope you don't get any big scary errors which cause the server to not start again (sudo service apache2 restart). If you do get an error such as the rewrite one I got, then all you need to do is go into /etc/apache/mods-enabled and symlink the rewrite module from mods-available in the same way as we did with the conf file earlier. If you are running some form of CMS (eg. Drupal) you will hopefully see your site, but it will load the install page, or tell you it cannot connect to the database. Funnily enough, this is our next task, so let's get to it.
Step 7:MySQL
MySQL can be a little fiddly, especially if you are trying to do it using the CLI, but let's give it a go.
Step 7a: Create MySQL users
First you will need to login to you mysql application (mysql -u root -p), once you are in, choose a name for this DB user, I usually pick a convention which I can extend in the future, such as below. We need to create 2 users IF you want to access the DB remotely aswell as from the local server. Otherwise, localhost on it's own should suffice.
CREATE USER 'site-sitename-prod'@'%' IDENTIFIED BY '<PASSWORD>';CREATE USER 'site-sitename-prod'@'localhost' IDENTIFIED BY '<PASSWORD>';
Note: If you want to quickly check the 2 new accounts were created, just try this command out:
SELECT User FROM mysql.user;
Step 7b: Migrate DB from old server
Chances are, your site is not a new one. Maybe you (like me) are moving hosts, and so need to migrate your database across to the AWS stack. First, SSH into your current server, and run the below command (remember to replace username) and get the sql dump. Once it has been created, FTP the file to your local machine (or, if you are feeling clever, WGET it over SSH).
mysqldump -u <username> -p -v <olddatabase> > <olddbdump>.sql
Note: You may need to tinker with the above depending on your users permissions. I had to add--lock-tables=false due to the strict restrictions I use.
Then, once the file is on your new AWS server, using whatever method you prefer, we can import it. The commands below will first create the new database, then import the DB export into it
mysqladmin -u <username> -p create <db name>mysql -u <username> -p <db name> < <filename>.sql
Step 7c: Add user to the new DB
Once the user has been created and database populated, we then need to give the user access to the database with specific priviledges. You will need to do this for each of the users created above, and have logged in to mysql CLI first (mysql -u root -p)
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON <db name>.* TO '<username>'@'localhost' IDENTIFIED BY '<password>';GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON <db name>.* TO '<username>'@'%' IDENTIFIED BY '<password>';
Now refresh your site, you should see Drupal connect if your settings.php file is there and ready to go(if you get a WSOD look in the apache logs at /var/log/apache2/xxxxxxxx-error.log as it is likely a memory issue. If so, just edit the memory_limit variable in /etc/php5/apache2/php.ini to something higher than the error).
Note: At this stage I also received errors such as 'Call to undefined function curl_init', which basically means, install phpcurl you fool. So run this(apt-get install php5-curl) then restart apach again.
Step 8: CREATE NEW SSH USERS
Okay, I kind of want the sudo user to be available on this server, and now would be a good time to do it, so install this(apt-get install sudo). I was getting occassional memory issues when trying to do this, to get around this just stop apache and mysql (service xxxxxx stop), install the file, then start the services again (service xxxxx start). You should now have sudo access, which will be great for when you use a non-root user to administer things (probably best to get into the habit of this sooner rather than later). Final configuration of this will have to be left until another time though. Now it is time to create your new user, with their own home directory
useradd -m <username>passwd <username>usermod -s /bin/bash <username>adduser <username> sudo
Ensure ssh is running nicely on this machine by running through ssh-keygen for the root user.
Now you will need to logout as your root user, I am going to assume your local machine has SSH keys setup already, if not, just run 'ssh-keygen' and click enter on all the answers to quickly do this. Once you have this copy the contents of the file in ~/.ssh/id_rsa.pub to a text editor, as we will want this in a moment. Now login back into the server, but this time as your new user. Note: When you login as your new user, you may not have bash straight away, a quick fix here is to just type bash and hit enter.
Once in there we will copy the content of that id_rsa.pub file into the file ~/.ssh/authorized_keys (if the folder doesn't exist yet run through ssh-keygen for your user, and create the file). If you created these, you will likely need to alter their permissions.
chown -R <username>:<username> .ssh
You should now be able to logout of your server, and log back in without needing to put a password in at all. From now on, I would recommend just using this user account for your work unless you urgently need the root one for something.
Step 9: Final tidy up (eg. get Drush in there)
I then also installed Drush, this is Drupal specific, so wont go into that here too much, there are also many tutorials on this which should cover it and the versions you may need (eg. https://www.drupal.org/node/2132447). Once installed, it may not work right straight away, there is a drushrc.php file there which the initial setup may have added a specific default site too. Just edit the file and remove this line (unless you know your way around drushrc.php and want to make other changes that is).
That is pretty much it. Now you just need to update your application (eg. in Drupal, update settings.php file if you haven't already), then retest. It should all connect nicely, with your site loading. You can now check any other application specific config such as file permissions to ensure the site runs. But now, you have a working server, and a website sitting on it. You could end it here with EIP and a DNS redirect, but then you aren't really much better off than you were before. Let's add some caching sauce to this server, and git it running like a T1000 straight out of Cyberdyne.