The Loop

I’ve recently started using Amazon EC2 and put quite simply, it could well be the best computing platform the world has ever seen! Its flexible, scalable, and very competitively priced which makes it an attractive proposition for users that are currently on fixed priced virtual machine hosting.

The real differentiator with EC2 is that you are effectively building a linux machine from scratch that can be flug around there cloud to any location. So, that means you could make an image of your running machine using there AMI Tools and have your box running in East Coast America. Pretty straight forward so far. You then get wind (pun intended) that a massive hurricane is about to hit the East Coast, and you’d feel safer if your servers were far far away, right? No problem, with a simple one line command you can re-deploy the image to the West Coast cloud and everything would just pick up where it left off! Amazing!

On top of all that clever trickery, you get pretty decent control over firewall ACL’s and the choice of box configuration is pretty decent – the ‘small’ machine still has 1.7GB of RAM, which is a boat load more than most of VM’s on the market. You can even get 7.5GB and 15GB variants.

Amazon, you have out-done yourselves. Congratulations!

Just a short post for people who might not be sure how to get Wireshark to read the interfaces from your mac…

I had to run the applications as root:wheel to let the X11 interface read the network interface devices. To do the same, just run:

sudo /Applications/Wireshark.app/Contents/MacOS/Wireshark

Then, when prompted, just enter your password (provided you are a machine administrator that is). Wireshark X11 should then boot up and work without problem.

If your on 10.4 you will need to install X11 beforehand

I’ve been doing some work with NGINX of late and anyone familiar with CakePHP will know that it ships out of the box with Apache .htaccess files to make sure that the URL’s are devoid of there query string.

Anyway, enough talk, if you want to host cakephp on NGINX, you’ll need to use a vhost like so:

    server { 
        listen       80;
        server_name  somedomain.com;
        access_log  /var/www/logs/somedomain.access.log  main;
        error_log   /var/www/logs/somedomain.error.log info;
        rewrite_log on;

        # rewrite rules for cakephp
        location / {
          root   /var/www/sites/somedomain.com/current;
          index  index.php index.html;

          # If the file exists as a static file serve it 
          # directly without running all
          # the other rewite tests on it
          if (-f $request_filename) { 
            break; 
          }
          if (!-f $request_filename) {
            rewrite ^/(.+)$ /index.php?url=$1 last;
            break;
          }
        }

        location ~ \.php$ {
          fastcgi_pass 127.0.0.1:9000;
          fastcgi_index index.php;
          fastcgi_param SCRIPT_FILENAME \
          /var/www/sites/somedomain.com/current$fastcgi_script_name;
          include fastcgi_params;
        }
    }

Strong title I know… It feels like thats how Java is regarded these days – Ruby and Python are becoming the golden boys of programming choice and poor ol’ java is being left out in the cold because people only remember it as having crap costing/license models where you [the developer] needed to pay for everything.

Well, I have good news for you my friends, Java should no longer be a dirty word! The inertia that Ruby on Rails had in the Java community did cause quite a lot of controversy its true, but its almost like after the initial xenophobia had worn off they sat up and thought “you know what, these guys might actually be onto something here”. I recently got into using Maven 2, and god, my good god, its a million times better than Ant, which just sucked so badly I cant explain! For anyone who is familiar with Rake, Maven is like Rake, but with all the package management stuff built right in – it does still have a certain level of entry required, but it just simplifies the process of created projects and managing dependencies in a very streamlined way.

The language itself has evolved quite a lot too – the new annotations are used heavily be JEE apps and frameworks – take CXF as an example; it will eventually replace XFire and Axis2 as the premier SOAP/WebService framework for Java, its a very very powerful tool and pretty easy to use (relatively speaking)

Anyway, I digress… my point is that if, like me, you left Java because the last time you worked with it you were slogging it out with 1.4 and Ant hell, then perhaps its time you took another look? Java has had significant investment as a platform, and there is a hell of a lot of good code out there for doing pretty much anything you can think of. In all honesty, one of the main reasons I have come back to JEE and the JVM in general is deployment – the JVM is very robust, and the containers built of top of it like Resin and Glassfish are being used in very high-concurrency environments and the deployment method via JAR or WAR is just so much more robust than on a platform like Ruby – and thats coming from having spent nearly the past 3 years solidly coding Ruby and doing all manner of deployments!

Dont write Java off – its an amazing platform, so what if its not as easy to get started as with other languages; think long-term, its a language that will grow with you rather than one you might someday reach the ceiling of – or just find incredibly annoying when the application dispatchers crash for a past time ;-)

Over and out

With new cross-language mash ups appearing all the time with things like JRuby and Jython it makes me wonder where all this is going? Will we have J2EE apps deployed alongside dynamic languages all within the same container, all capable of leveraging the awesome JVM runtime? If so, then wow, thats an awesome proposition. Pretty much every language you can think of is now deployable within a java container – its even possible to deploy .net applications via mono!

Perhaps people will start to standardize on the JVM as a platform – all this constant fighting over runtimes, platforms and environments makes work for the developer and architect extremely difficult; we want to keep up with the latest developments, but it just makes it so difficult to do that with all this chopping and changing.

Anyway, things I think are going to be seriously hot this year are:

Resin Application Server

Lift Framework and Scala

JRuby

Hessian Binary Remoting

We’ll see how things play out over the next few months, but whatever happens, its going to be exciting to see how all these technologies interact with each other – which fizzle out and which go on to achieve wonderful things.

Further to my article about God, and using it for watching Merb it seemed like a good idea that I also post my capistrano recipe so people can get the complete picture :)

# MIT License
# 
# Copyright (c) 2008, Tim Perrett
# 
# Permission is hereby granted, free of charge, to any person 
# obtaining a copy of this software and associated 
# documentation files (the "Software"), to deal in the Software
# without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, 
# and/or sell copies of the Software, and to permit persons 
# to whom the Software is furnished to do so, subject to the 
# following conditions:
# 
# The above copyright notice and this permission notice shall 
# be included in all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF 
# ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED 
# TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT 
# SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR 
# ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 
# OR OTHER DEALINGS IN THE SOFTWARE.

set :application, "domain.com" 

set :repository, "svn://yourrepo.com/path/to/repo" 

# Set your SVN and SSH User
set :user, "sshuser" 
set :svn_user, "svnuser" 
#Set the full path to your application on the server
set :deploy_to, "/path/to/your/#{application}" 

# define your servers
role :app, "domain.com" 
role :web, "domain.com" 
role :db, "domain.com", :primary => true

desc "Link in the production extras" 
task :after_update_code do
  run "ln -nfs #{shared_path}/log #{release_path}/log" 
end

desc "Merb it up with" 
deploy.task :restart do
  # you need to add restart tasks for each port you plan 
  # to run merb on. Whilst this is a little long winded, 
  # it will ensure uptime compared to the sledge hammer
  # than is #{current_path}/script/stop_merb
  run "(cd #{current_path}; merb -k 4006); sleep 1; \
  #{current_path}/script/merb -u timperrett -G timperrett \
  -M #{current_path}/config/merb.yml -p 4006 \
  -e production -d" 
  run "(cd #{current_path}; merb -k 4007); sleep 1; \
  #{current_path}/script/merb -u timperrett -G timperrett \
  -M #{current_path}/config/merb.yml -p 4007 \
  -e production -d" 
end

Any problems, let me know :)

Before you start reading, this post only applies for Merb 0.5.3, any newer version is totally untested

When you take Merb out into the wild, it does, unfortunatly, suffer a lot of the same problems as the mongrel handler than runs Rails.

There is however a saviour out there – God – To clarify, im not talking about the man upstairs; rather the process monitoring tool which rocks at restarting bloated mongrel processes on *nix based OS.

The Merb handler does not have any way of restarting a running cluster, so you physically have to stop, then start a new merb process. This is somewhat out of sync with how the god process handling and restarting works in that you define start, stop and critically, restart paramaters. To get around this we have to use a hacky sleep then start method – its not ideal, but hey, it works :)

Rather than letting merb handle the process forking, what were going to do is let God handle the writing of Pids and managing of the process. Ok, less of all this talk and lets take a look at some configuration code for the God configuration.

# MIT License
# 
# Copyright (c) 2008, Tim Perrett
# 
# Permission is hereby granted, free of charge, to any person 
# obtaining a copy of this software and associated 
# documentation files (the "Software"), to deal in the Software
# without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, 
# and/or sell copies of the Software, and to permit persons 
# to whom the Software is furnished to do so, subject to the 
# following conditions:
# 
# The above copyright notice and this permission notice shall 
# be included in all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF 
# ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED 
# TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT 
# SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR 
# ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 
# OR OTHER DEALINGS IN THE SOFTWARE.

SITE_LOCATION = "/var/www/sites/domain.com" 
MERB_ROOT = "#{SITE_LOCATION}/current" 
MERB_CONFIG = "#{SITE_LOCATION}/shared/config/merb.yml" 
MERB_ENVIROMENT = 'production'
PROCESS_USER = 'timperrett'
PROCESS_GROUP = 'timperrett'

%w{4006 4007}.each do |port|  
  God.watch do |w|

    pid_path = File.join(MERB_ROOT, 'log',"merb.#{port}.pid")

    w.name = "merb-#{port}" 
    w.interval = 30.seconds # default
    w.start = "#{MERB_ROOT}/script/merb -u #{PROCESS_USER} \
    -G #{PROCESS_GROUP} -M #{MERB_CONFIG} -p #{port} \
    -e #{MERB_ENVIROMENT} -d" 
    w.log = "/home/timperrett/godmerb.log" 
    w.stop = "(cd #{MERB_ROOT}; merb -k #{port})" 
    w.restart = "(cd #{MERB_ROOT}; merb -k #{port}); sleep 1; \
    #{MERB_ROOT}/script/merb -u #{PROCESS_USER} -G \
    #{PROCESS_GROUP} -M #{MERB_CONFIG} -p #{port} \
    -e #{MERB_ENVIROMENT} -d" 
    w.start_grace = 5.seconds
    w.restart_grace = 20.seconds
    w.pid_file = File.join(MERB_ROOT, "log/merb.#{port}.pid")
    # w.group = "merbs" 
    w.behavior(:clean_pid_file)

    w.start_if do |start|
      start.condition(:process_running) do |c|
        c.interval = 10.seconds
        c.running = false
      end
    end

    w.restart_if do |restart|
      restart.condition(:memory_usage) do |c|
        c.above = 51.megabytes
        c.times = [3, 5] # 3 out of 5 intervals
      end

      restart.condition(:cpu_usage) do |c|
        c.above = 50.percent
        c.times = 5
      end
    end

    w.lifecycle do |on|
      on.condition(:flapping) do |c|
        c.to_state = [:start, :restart]
        c.times = 5
        c.within = 5.minute
        c.transition = :unmonitored
        c.retry_in = 5.minutes
        c.retry_times = 5
        c.retry_within = 2.hours
      end
    end

  end
end

Im not sure that this is ideal, but it certainly seems to work for me and importantly keeps the site running without problems – which is a dang sight better than them becoming unresponsive!

I hope this might help someone, somewhere….

Following on from setting up Django on your local system for dev purposes, I thought id just do an article about how to set up apache to use mod_python for when you want to roll your Django app out to the big bad world.

Before you start, you will NEED to have completed the other post about installing Django, as you need to have the mac python installer in and configured properly for this to work.

Step 1

First of all we’ve got to get Apache 2.2 working, as Apple only ship 1.3 with OSX (I wonder if this will change in Lepord? Lets hope so!). You can download the source here.

Extract the archive and cd into the new directory. We need to configure it with some options so that we have proxying (which you might want for rails etc) and just some other candy you might find usefull at a later date.

Set the CFLAGS:

export CFLAGS=”-arch i386″

Run this configure command:

./configure \
  --prefix=/Library/Apache-2.2.6 \
  -–enable-so \
  --enable-mods-shared=all \
  --with-mpm=prefork \
  --enable-dav \
  --enable-cache \
  --enable-proxy \
  --enable-shared \
  --disable-static \
  --disable-unique-id \
  --disableipv6 \
  --enable-logio \
  --enable-deflate \
  --with-ldap \
  --with-ldap-include=/usr/include \
  --with-ldap-lib=/usr/lib 
  --with-included-apr 
  --enable-ldap \
  --enable-auth-ldap \
  --enable-cgi 
  --enable-cgid \
  --enable-suexec

Let it do its business then run the usual:

make

then

make install (NB: we dont use sudo here as that way apache is compiled with permisions that let you edit the conf files and so on without needing to be an admin which is just ‘nicer’ for local dev)

Step 2

Configure your httpd.conf file as you see fit. I only really felt the need to change the following:

Change user and group to your user name (or any other one you want) Change ServerAdmin email to an appropriate one Uncomment “Include conf/extra/httpd-mpm.conf” Uncomment “Include conf/extra/httpd-default.conf” Add “NameVirtualHost *” for virtual hosting

My personal preference is then to have an “applications” folder in the apache conf dir, for which I then add a directive to httpd.conf to load in per-application virtual hosting configurations. It just keeps it nice and clean that way.

mkdir /Library/Apache-2.2.6/conf/applications

Include conf/applications in httpd.conf (in your httpd.conf)

Step 3

If you havent already, grab mod_python from here. Extract, it and cd into the directory.

You’ll then need to configure it with the following command:

./configure -–with-apxs=/Library/Apache-2.2.6/bin/apxs \
  -–with-python=/Library/Frameworks/Python.framework/Versions/Current/bin/python

Then do make and sudo make install

Provided you got no errors upon compilation, all should be well. You just need to add another line to your httpd conf:

LoadModule python_module modules/mod_python.so

Step 4

Poor yourself a nice cuppa!

No doubt you’ve heard about the new python web framework thats making some waves at the moment - Django.

As with most *nix developers, im running OSX (10.4), and wanted to get a working Django dev environment function to have a play around with it and found that a needed to mess around with a few things to get it working, so I thought id post them up here in case anyone else found them usefull :)

Step 1

Download the python-mac installer from here and run it. That will give you a working version of the latest python bin.

Step 2

Checkout the latest version of Django source like so

cd /usr/local

svn co http://code.djangoproject.com/svn/django/trunk/ django

Step 3

If you used the python installer for mac, then the below will work, otherwise, you’ll need to just tweak the path’s a bit:

ln -s /usr/local/django/django /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django

Step 4

Add the django source to your path:

PATH=/usr/local/django/django/bin:$PATH

Then that should be it :) However, if your a touch on the forgetfull side when using the terminal, it can be annoying having to keep putting the .py on the end of the django commands, so we’ll just quickly symlink them so if we forget to add .py, it wont slap us with a kipper

cd /usr/local/django/django

ln -s django-admin.py django-admin

Then your all done! Enjoy Django :)

Pound is the unix reverse proxy software popularized by Rails deployment over the past 8-12 months. Pound is a unix tool, and was never meant to run under windows - it is however possible to make it run under windows if you package it up with some key DLL files from cygwin :)

I must stress here that this is not necessarily the best thing to be doing, this post is more about how to do it if you really want too rather than a "do this, I really recommend it" type post. There are also some issues with pound and its behavior with X-FORWARDED_FOR headers, but i wont bang on about that here; refer to google for more info. Anyway... back to the post.....

Take the latest pound distro from here then compile it from within Cygwin like so

./configure --without-ssl --disable-log --disable-dynscale

then

make all

Then you should be left with a working binary of pound within your build directory. In order to then make this work in a standalone fashion (I use the term standalone loosely as its still using the cygwin environment) you need to package several DLL files from cygwin:

cygssl-0.9.8.dll (required)

cygcrypto-0.9.8.dll (required)

cygwin1.dll (required)

pound.exe (required, should be fairly obvious!)

cygrunsrv.exe (optional, include only if you want to install pound as a service)

with these files you can then run pound, and even install it as a service on the host machine. If you do want to install the service, run the following command:

cygrunsrv --install Pound --path X:\applications\pound-2.3.2\bin\pound.exe --args "-f X:\applications\pound-2.3.2\config\pound.cfg" --stdout X:\applications\pound-2.3.2\log\pound.log --stderr X:\applications\pound-2.3.2\log\pound.log

Where "X:\applications\pound-2.3.2" is the real path to where ever it is on the file system you chose to install.

If you don't want to compile your own, then you can download my version of pound 2.3.2 for windows here. It goes without saying I take no responsibility for any problems you might run into by using my compiled version; it worked for me thats all I can say!