Paul Ingles

The project I’m currently working on uses both the Asset Packager and Distributed Assets to ensure we have only a few external assets, and that we can load assets across more than one host - all so that the pages for our site load nice and quick.

Unfortunately, wiring in the Asset Packager plugin caused the Distributed Assets plugin to break, and I spent an hour or two tracking it down yesterday. The cause? Asset Packager redefines the compute_public_path method.

# rewrite compute_public_path to allow us to not include the query string timestamp
    # used by ActionView::Helpers::AssetTagHelper
    def compute_public_path(source, dir, ext=nil, add_asset_id=true)
      source = source.dup
      source << ".#{ext}" if File.extname(source).blank? && ext
      unless source =~ %r{^[-a-z]+://}
        source = "/#{dir}/#{source}" unless source[0] == ?/
        asset_id = rails_asset_id(source)
        source << '?' + asset_id if defined?(RAILS_ROOT) and add_asset_id and not asset_id.blank?
        source = "#{ActionController::Base.asset_host}#{@controller.request.relative_url_root}#{source}"
      end
      source
    end

Distributed Assets works by chaining compute_public_path - decorating the calculated path, adding the asset host prefix onto the url. But, Asset Packager works by defining the method into ActionView::Base. So, when DistributedAssets::AssetTagHelper is included with ActionView::Helpers::AssetTagHelper, it chains a (now) hidden method.

But, the only places that use the new compute_public_path code inside the Asset Packager Helper (which just avoids using the query string timestamp) is within Asset Packager itself.

So, I tweaked the implementation of AssetPackageHelper to

def compute_public_path_for_packager(source, dir, ext, add_asset_id=true)
  path = compute_public_path(source, dir, ext)
  return path if add_asset_id
  path.gsub(/\?\d+$/, '')
end

def javascript_path(source)
  compute_public_path_for_packager(source, 'javascripts', 'js', false)       
end

Beware the monkey patch.

I’d read a while back that Apple were going to include Sun’s DTrace tool in the now newly released Leopard - underpinning Instruments (formerly known as Xray).

What I hadn’t noticed was that the build of Ruby 1.8.6 included in Leopard (patchlevel 36 with some extras) also includes Ruby probes from the lovely folks at Joyent. Apple have ported everything. So, out of the box, Ruby works with DTrace! Sweet!

From Sun’s page:

DTrace is a comprehensive dynamic tracing facility that is built into Solaris and can be used by administrators and developers to examine the behavior of both user programs and of the operating system itself. With DTrace you can explore your system to understand how it works, track down performance problems across many layers of software, or locate the cause of aberrant behavior.

Joyent have got some sample scripts in their Subversion repository, and, what looks like some nice extensions for Rails (DTrace probes can be fired from within Ruby). Blimey!

For those (running Leopard) and want to just see something, try this out. It’s a script that traces the number of times a method is called. Once downloaded, all you need to do is

$ chmod +x functime.d
$ sudo ./functime.d -p <pid>

Replacing <pid> with the process id for your target Ruby process. Once you stop tracing (or your target process ends you’ll see some nice text formatted output. Including something like the following (from my current project):

Hash                     key?                     25123         15       384174
Module                   each_object                  1     390824       390824
Module                   included_in_classes          1     391025       391025
Module                   reloadable_classes_with      1     391203       391203
Module                   reloadable_classes           1     391835       391835
ActionController::Routin load_routes!                 1     394864       394864
ActionController::Routin reload                       1     398364       398364
Class                    reset_application!           1     398947       398947
Class                    reset_after_dispatch         1     400474       400474
ActionController::Routin connect                     39      13192       514506
Array                    include?                  3353        166       556605
ActionController::Routin build                       47      13116       616459
ActionController::Routin add_route                   47      13329       626488
Array                    each                       874        780       682423
Object                   load                         2     376432       752864
Module                   silence                      2     391586       783172
Kernel                   gem_original_require        28      43460      1216884
Array                    select                     316       4106      1297580
Array                    collect                    351       4124      1447524
Module                   local_constants             91      17120      1557986
Module                   new_constants_in            25      93885      2347139
Object                   require                     50      58495      2924783

The columns (from left to right) are: Class, Method, Count (number of times method was called) and then the average and total micro seconds. The d script was one of the Joyent examples. Arstechnica also have a very cool example showing the call stack from a system call.

What’s more amazing is this doesn’t require any recompilation, special flags, running in special environments. It’s available from the off!

If all that’s whetted your appetite (and I’m not sure how it couldn’t), head on over to the DTrace how to for a little more explanation of how it all works.

Time to get stuck in and see whether anything emerges about my current project. Very cool stuff!

In preparation for a Ruby project I wanted to play around with Rails’ page caching a little over the weekend. Naturally, I wanted to find a way to ensure that my controller was writing files correctly so that Apache (or any other HTTP server) can then serve the static content directly (without hitting my Rails processes). Since I’m also digging RSpec at the moment, I wanted to find a way of expressing it nicely in my specifications. This is what I ended up with.

Here’s the relevant controller’s specification:

describe UsersController, "caching" do
  include CachingExampleHelper

  it "should cache new users page" do
    requesting {get :new}.should be_cached
  end

  it "should not cache the activation page" do
    requesting {get :activate, :activation_code => 'blah'}.should_not be_cached
  end
end

The key part is the requesting line. That’s my extension that encapsulates the request that should be cached, so that we can both make the request and check that it ended up with a file being written to the filesystem.

module CachingExampleHelper
  ActionController::Base.public_class_method :page_cache_path
  ActionController::Base.perform_caching = true

  def requesting(&request)
    url = request.call.request.path
    ActionController::Base.expire_page(url)

    request.call
  end

  module ResponseHelper
    def cached?
      File.exists? ActionController::Base.page_cache_path(request.path)
    end
  end

  ActionController::TestResponse.send(:include, ResponseHelper)
end

Note that we have to expose the pagecachepath method so we can calculate the location for where the cached files will get written, and then also explicitly enable caching. When we call requesting giving a block to the request we want to make (such as get :index to invoke the index action), we capture the block as a Proc, invoke it and then read the request’s path from the generated TestResponse object. This lets us then first expire the page (in case it was cached previously), and then make the request. Finally, we then check whether the cached file exists by mixing in a cached? method to the Response.

Unfortunately, the code above will make 2 requests per assertion. But, I’d rather have it read nicely for the time being. Any suggestions on how you could tidy this up further would be well received!

I’m not sure I’d want to run these tests all the time, maybe something a little lighter to ensure that we tell Rails to caches_page for relevant actions in our controllers, but as a definite re-assurance it seems ok.

Inside my tests I try and keep everything I need inside the test, rather than depending on too many fixtures. That way I ensure they’re as readable as possible, and more importantly, obvious.

But, take the following example:

def test_total_is_due_now_if_return_is_less_than_14_days
  booking = Booking.new(
    :customer => customers(:valerie),
    :collect_at => 1.day.from_now.to_date,
    :return_at => 13.days.from_now.to_date,
    :boxes => 2,
    :collection_address => addresses(:one))
  assert_equal Date.today.to_s, booking.final_payment_due_date.to_s
end

def test_payment_due_14_days_from_now
  booking = Booking.new(
    :customer => customers(:valerie),
    :collect_at => 1.day.from_now.to_date,
    :return_at => 20.days.from_now.to_date,
    :boxes => 2,
    :collection_address => addresses(:one))

  assert_equal 6.days.from_now.to_date.to_s, booking.final_payment_due_date.to_s
end

Well, everything’s in the test, but there’s a fair bit of duplication. What would be nicer, is to take the prototype booking above and just override certain values.

Instead we might write something like

def test_total_is_due_now_if_return_is_less_than_14_days
  booking = new_booking_with(:return_at => 13.days.from_now.to_date)
  assert_equal Date.today.to_s, booking.final_payment_due_date.to_s
end

def test_payment_due_14_days_from_now
  booking = new_booking_with(:return_at => 20.days.from_now.to_date)
  assert_equal 6.days.from_now.to_date.to_s, booking.final_payment_due_date.to_s
end

private
def new_booking_with(args)
  Booking.new(args.reverse_merge({
    :customer => customers(:valerie),
    :collect_at => 1.day.from_now.to_date,
    :return_at => 15.days.from_now.to_date,
    :boxes => 2,
    :collection_address => addresses(:one)
  }))
end

Much nicer (to me anyway), particularly once you start adding more and more tests (and especially validation related ones).

I’m working on a little pet Rails project and wanted to use Mocha to isolate my controller tests a little from what is usually encouraged, especially in light of what happened last time. I figured I’d give Mocha a go.

I wanted to verify that my service was called with objects that look equal to what I was expecting. Ordinarily, it will compare instances - are they the same object. I didn’t want that, but rather wanted to test that the values of my (ActiveRecord) objects were the equal. Easy. Mocha allows with to be called with a block that is evaluated to compare the objects.

My tests look something like this

def test_should_ask_service_to_calculate_new_prices_for_booking
  address = Address.new(:postcode => "W1 1QE")
  booking = create_booking

  OrderBookingService.expects(:calculate_price).with(booking, address) do |b, a|
    b.attributes == booking.attributes && a.attributes == address.attributes
  end

  post :calculate_price, :booking => booking, :collection_address => address
end

It’s something I’ve had to do many times in the past with JMock and NMock - I want to test equality for an aspect of some other object but it’s always meant writing rather a lot of code. Neat.

Thank’s to Nathaniel Brown for pointing this out.

A few days ago Flickr seemed to change their URL scheme for images served inside their RSS feed. The Mephisto plugin I wrote included the Flickr aggregation library from Typo. This does some regular expression parsing to determine the URL for other image sizes, but, because Flickr had changed their URL scheme, it broke.

Fortunately, the fix is pretty straightforward. Of course, first the test to show the new behaviour of Flickr:

1
2
3
4
5
6
7
8

def test_should_generate_correct_address_for_each_image_size
  pic = FlickrAggregation::Picture.new(
    :title => 'test',
    :description => 'http://farm1.static.flickr.com/my_image_m.jpg'
  )

  assert_equal 'http://farm1.static.flickr.com/my_image_m.jpg', pic.image
  assert_equal 'http://farm1.static.flickr.com/my_image_d.jpg', pic.medium

Running the test shows that we get an error - the regular expression fails to pick up the URL correctly:

NoMethodError: You have a nil object when you didn’t expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.first

So, we change our aggregation library to fix it as:

1
2
3

def image
  description.scan( /(http:\/\/.*(static|photos).*?\.jpg)/ ).first.first
end

Run the test again and we’re green.

I’ve committed the changes into the Subversion repository. So feel free to get the update!

Incidentally, I wonder whether this is related to Ezra Zygmuntowicz’s post about speeding up page loads by serving images from cnames. Maybe it’s just coincidence.

One of the first things I noted when I posted (some would say borderline complaining) about my first encounters with Liquid - the templating engine used in Mephisto and a few other apps - was the way it didn’t seem to abstract how parameters were set on the block.

Well, turns out it’s not quite so nasty as I first expected - you indeed don’t have to write all the regular expressions yourself. Inside liquid.rb a number of constants are defined, a lot of these are to do with capturing aspects of the tag.

One such regular expression is for TagAttributes, so we can write a test that:

1
2
3
4
5
6
7
8

def test_should_use_parameterised_url_for_feed
  template = Liquid::Template.parse(
    "{% flickrphotostream feed: http://blah/test.xml %}
    {% endflickrphotostream %}")
  first_block = template.root.nodelist.first
    
  assert_equal 'http://blah/test.xml', first_block.feed_url
end

first_block is the first block that we encounter in our template - the Flickr photostream one. Inside our class, we can then build up an attributes hash from these items using the TagAttributes regular expression as follows:

1
2
3

markup.scan(Liquid::TagAttributes) do |key, value|
  @attributes[key.to_sym] = value
end

Hey presto, we’re done.

Now, I’m not so sure I like this too much, it doesn’t feel very Rails-like. Too much dealing with regular expressions for my liking.

So, I’m going to continue refactoring my solution (please don’t read too much into the tests or code I’ve written so far, it’s very much a first stab over the course of a little hacking). As part of that, I’d definitely like to see if I can Rails-ActiveRecord-it-up a tad, a la:

1
2

class FlickrPhotoStream < Liquid::Block
  attr :feed_url, :count

Maybe I’ll give that a go tomorrow.

I’ve been looking at trying to write some Liquid tags for use as Mephisto plugins. I still have my doubts as to how productive it is to extend/work with Liquid, but at least I’m making some small headway.

I’d gotten as far as having some code working (there were no errors in the running of the test) but I needed to test more deeply: that I could actually render something useful.

To me, tests are essential as a means of proving I’m making progress, and actually writing code that works. This is even more useful in a relatively new environment (read: Mephisto and Liquid).

Thankfully, Rails already includes some great stuff for testing HTML templating (and our controllers), especially now with the recent addition of assert_select to core. By default, it works by reading the respone inside your functional and integration tests via Rails’ @response object. However, it will also work with an HTML::Node so to test our Liquid block renders correctly we just need to create an HTML document from it!

Our test looks something like this

1
2
3
4
5
6
7
8
9
10

    def test_should_see_one_image_in_template_results
      template = Liquid::Template.parse(
        "{% flickrphotostream %}
          <img src='' alt='{{pic.title}}' />
        {% endflickrphotostream %}"
        )
      root_node = HTML::Document.new(template.render).root

      assert_select root_node, 'img[alt=my title]'
    end

We assert that we find an image tag that with some alt text set to our picture’s title. All nice and simple.

I’ll work on getting the rest of the code written, now I have a way to test it! All that’s left is to beautify it up a little - that’s too much hook-up code I’ll need in each test for my liking :)

Well I’m not so sure that Lighty, and FCGI running Rails and PHP isn’t such a great mix after all. After approximately a day all goes a little crazy and everything becomes largely unresponsive. So I’m taking drastic measures, and trying yet another web server.

This time, I’m giving Litespeed a go. It’s a commercial piece of software, but, they do offer a standard edition that’s free. So far it seems to run pretty good, importantly it seems to do a good job of serving PHP (which always seems to have been on the problem on my server anyway – I had some PHP apps for stats tracking and webmail).

After a while I’d have `defunct` php processes kicking around, and fairly soon after my Rails processes would go crazy. Then I was in a world of pain, which most certainly did not rock!

I’ll keep an eye on things. So far it seems pretty stable, and was very easy to setup (following their own guide). I just need to get a few more things configured and I’m all ported.

I’d definitely be tempted to switch back to Lighty, but I think I’ll have to wait until 1.5.0 and `mod_proxy_core`. Also, have any people on the lazyweb had problems with Mongrel 0.13? I tried Lighty -> Pound -> Mongrel and that also seemed to die as often as Lighty -> Mongrel?

As alluded to in my last post, I’ve been looking at enhancing my Typo plugin by adding some kind of relevance to the results.

Since I’m considering articles that share some tag to be considered related, I’m also going to consider articles that share more than one tag more relevant. This is a post that covers my adding that support, as well as some much needed Ruby-ification (is that a word? :p) and refactoring of the code following comments from Piers Cawley on the code in my last post.

Apologies if you’ve been trying to access one of the posts here over the last few days and received any one of the various Internal Server Error, or Service Unavailable error messages.

It seems that Lighty (`mod_proxy` – I also tried running the latest 1.5.x code from the Subversion trunk to use `mod_proxy_core` but that killed Mongrel everytime), Mongrel and Rails still aren’t playing too well together. Without being able to regularly monitor the server, I’ve not been able to guarantee any kind of uptime.

I’ve reverted back to good ol’ FCGI. So far it seems been pretty good, and I don’t seem to remember having to restart much at all before I went down the Mongrel route.

I can now (finally) get back to adding a ranking (to indicate potential relevance) to the Related Articles plugin I developed for Typo.

I’m not so sure how relevant this post is now that Piers has started removing the component-ness of Typo’s sidebars (kudos for changing it all so quick). But, it was again an interesting exercise, and hopefully people will take something from it.

Tonight I decided to follow on from my previous post and take a look at packing it all up into a Rails plugin. Well, move the `related_articles` method into a plugin, and have a zip file with the component until I figure out how I can put it all together. Piers’ changes sound like a really cool way of doing things, can’t wait to look at the code later.

Also, please note that I’ve tested this on my machine and it works, but of course that’s no guarantee I haven’t done something stupid :) So I’d love it if people did download it if they could let me know how it all goes.

UPDATE: Sorry, I pulled the trigger too quickly on this and included some old code that I’d updated. I’m in the process of getting the right code up on Trac, all should be well now though.

I wanted to see how easy it would be to make an extension to the popular Rails application, Typo. Primarily, because I needed some extra functionality myself. Secondly, I thought it would be a nice exercise to write something test-first for an existing application I’ve not developed. This (rather large post) is the result.

The post covers most of my process and thoughts as I was working on the code (I started hacking away whilst on an early morning train journey for work).

Getting the Code

At the moment, I’m hosting the code at a Trac install I’ve put up for the (eventual) component. So far it’s just the Subversion patch file, and some notes for installation.

I’m working on changing it so it can be installed without changes to the Typo code, and packaging it properly but I’m learning so please be patient :) If people have suggestions, feel free to add comments!

Anyway, onto the main event…

So it looks like there’s a security problem with a recent-ish version of Rails (well, anything older than the Edge as of a few weeks ago seem to be at risk). The hole has been described as

This is not like “sure, I should be flossing my teeth”. This is “yes, I will wear my helmet as I try to go 100mph on a motorcycle through downtown in rush hour”. It’s not a suggestion, it’s a prescription.

Fortunately, I’m running an up-to-date version of Typo which quite happily (at least currently anyway appears to) run against the latest Edge Rails (revision 4745).

How to Update Typo Rails

As of a few versions ago, Typo used to use the Rails gem. However, during the Rails 1.1 release a few shared hosts automatically installed the system-wide gem. This seemed to break a few applications, as sites ran code that wasn’t compatible with some breaking changes in Rails 1.1.

The solution was to freeze the version of Rails to 1.0 in the `vendor/rails` directory. Going forward, Typo was brought up-to-date against 1.1, and the repository was also changed to include an svn:externals link to the Rails trunk. The result is that all that is needed to do the update was

$ cd vendor
$ svn up 
...
Updated external resource to revision 4745.

Ahhh, I can breathe easy! Back to the Typo Sidebar hackery (an article will be coming soon)...

UPDATE: I also took the opportunity to upgrade Typo itself, after which, I ran the usual `rake migrate`. Sorry, made a mistake in my original post.

Well, after a few problems with getting Gems working yesterday I appear to have made some progress.

As Dave mentioned, it indeed turned out to be a memory problem. Stopping all my sites and MySQL seemed to do the trick, allowing me to pull down Mongrel and Mongrel Cluster.

Now all I needed to do was to configure everything so Lighttpd would proxy everything save for the Mint stats requests to Mongrel.

Here’s how it all went.