During the most recent ThoughtWorks away-day (a chance for the office to get together, catch-up, drink etc.), George and I presented on a number of Ruby and Rails lessons we learned from our (now previous) project. One of the most interesting sections (to us anyway) was on declarative programming, specifically, refactoring to a declarative design.

I guess much like DSLs, its easier to feel when you’re achieving something declarative as opposed to necessarily defining what makes it. But, I’ll try my clumsy best to define something.

Almost every language I’ve turned my hand to (save for Erlang) are imperative languages - where programs are written as sequences of operations, with changes of state. You determine the what and the how of the system - what to do and how to do it.

Declarative programming is an alternative paradigm, whereby code is expressed as what’s. How the system executes is someone else’s responsibility.

So, for the purposes of this discussion let’s consider that application code can be split into two groups

1. Logic- the rules, the guts of things- that what’s.
2. Control- statements about execution flow.

Interestingly, one of the principles listed in Kent Beck’s most recent book (Implementation Patterns) includes “Declarative Expression” - that you should be able to read what your code is doing, without having to understand the wider execution context.

Declarative languages are all around us, with most developers I’m guessing using them almost daily.

Think of SQL, when you write a statement such like SELECT [Name], [Age] FROM [Person] WHERE [Age] > 15 you’re making a statement about what you’d like, not how you get there- that’s up for the database engine to figure out. And a good thing to! Have you ever taken a look at an execution plan for modestly complex queries?

Closter to home - think of .NET and Java Annotations- where you can decorate constructs with additional behaviours. They look and feel like core extensions to the language, but are programmable and can be used to adapt the runtime behaviour of the system.

Before working for ThoughtWorks, I worked on a system where we used attributes to allow us to add validations to properties, allowing us to re-use code and extend easily.

[LengthMustBeAtLeast(6)]
public property string FirstName
{
  get { ... }
  set { ... }
}

Everything was nicely decoupled, read well, and reduced the amount of clutter in our code. More importantly, our validation code was not spread throughout every setter of every property. We could isolate responsibilities making code easier to digest and understand, and test!

Onto Ruby. A substantial part of our project involved reading lots of CSV files from different sources to update our deal information. The answer was a kind of anticorruption layer (borrowing heavily from domain-driven design) for each different feed.

Quickly, we ended up with a few concrete classes and a base class that co-ordinated effort. Dependencies were shared both ways, imagine a number of template methods that are called in sequence to accomplish their work. Over time it grew complex, and with Ruby it’s a little tougher (than with languages like Java or .NET) to navigate and browse around the code without strong IDE support.

It was starting to get a little too complex and we felt we needed to change things, so we did (bolstered somewhat by having Jay with us).

Onto the code.

So imagine we have a class representing a Feed of information (read from a CSV file), and we want to be able to ask that Feed to provide us with a number of Deals. Internally, it will iterate over the items in the Feed, creating a Deal for each one (if possible).

Our first solution looked a little like this, firstly in the ‘abstract’ base class:

def create_deal
  Deal.create(:network_name => network_name)
end

and in the feed’s concrete implementation:

FIELD_INDICES = {:name => 1, :network_name => 2}

def network_name
  read_cell(FIELD_INDICES[:network_name])
end

Our main feed class asks our implementation class for the network_name - a template method. Not bad, now build that up to tens of attributes, a bit longer. Since we’ve defined column indices in a constant, we also now have to navigate up and down lots to determine where we’re reading from. Add in a few other tables with look-ups for other bits that we need to do the mapping and it can get a little complex pretty quick.

Our code is not only made up of the stuff determining what it is to translate between a CSV representation of our Deals to an object model one, but also all of the code necessary to find out which CSV column we’re in, how we map that column etc. They were essentially just reading values from cells, no translation needed. Most of the code we had was infrastructural, and the logic (the what of our application) was hidden amongst the noise.

This complexity, combined with the split of flow between abstract and concrete classes, made it difficult to follow and understand. Our goal was to try and reduce each concrete implementation to a single page on our screens.

These little ‘mapping’ translation methods were our first target - reduce the amount of code for each of these to one line that described the mapping, rather than how we get it all.

Firstly, we introduced a convention - every attribute of a Deal could be retrieved by calling a deal_attribute_my_attribute style method. So, we renamed all our methods, ran the tests, and then started to make steps towards having each deal_attribute_blah method defined dynamically.

We went from:

def network
  ...
end

to

def deal_attribute_network
  ...
end

to … nothing.

Well, not quite. Instead, what we wanted was to have a method constructed by adding a class method to a module that we could mix-in. Then, we could just define the mapping and Ruby would wire up the rest. We settled on the following syntax

deal_attribute :name => 'NAME'

Neat. A little classeval and instanceeval magic later we were able to push our infrastructural code out of our concrete feed class and into the co-ordinator.

Our code is now more declarative. We’re stating what we need to do our work, rather than worrying about how we get at it. Not only that, for the common case (where we may be just moving values from one place to another) there’s no need to do anything more than describe that relationship. Declarative programming makes it much easier to express important relationships. It’s now much easier to see the relationship between the name attribute of a Deal and the NAME column for this CSV feed.

Notice also how we’re pushing our dependencies up to our caller- the coupling is now one-way. We state what we need from our caller (the main deal feed class) - our caller is able to then pass the information on. We’re just answering questions, rather than answering questions and asking questions (of our caller).

The syntax also lends itself to explaining a dependency, that an attribute of our deal is read from ‘NAME’ (for example).

That’s great, next step was to tidy up some of the slightly more complex examples where we do some additional translation. For example, where we take the name of something and we need to pull back an object from the database instead. Let’s say we keep track of the Phone that a Deal is for.

So, from

def deal_attribute_phone
  Phone.find(deal_attribute_brand, deal_attribute_model)
end

to

deal_attribute(:phone => ['MAKE', 'MODELNAME']) do |brand, model|
  Phone.find(brand, model)
end

We’ve extended the syntax to reveal that this translation needs values from both the ‘MAKE’ and ‘MODELNAME’ columns. From our perspective, we’re pushing responsibility up and keeping our code focused on what we need to map this attribute.

This is a little more complex to achieve since we’re also passing arguments across (and our deal_attribute translator methods also sometimes need to access instance variables) so we need to use instance_exec instead of the standard instance_eval.

The end result was feed classes that looked as follows

class MySpecialFeed
  deal_attribute :name => 'NAME'
  deal_attribute(:description => 'DESC') {|desc| cleanse_description(desc) }
  deal_attribute(:phone => ['MAKE', 'MODELNAME']) do |brand, model|
    Phone.find(brand, model)
  end
  ...
end

In total, it took us probably just over a day to refactor the code for all of our classes. Most of which we managed to get down to some 40 or 50 lines total. We didn’t refactor all the code, so there’s still potential for exploiting it further but it was definitely a very exciting thing to see happen.

From across the desk, George asks “can you copy classes in Ruby?”. We talk about it quickly and reason that since everything’s an Object (even classes), you probably can. Since the constant isn’t changed or duplicated (you’re essentially assigning a new one) then it ought to be possible.

Turns out it is!

class First
  def initialize
    @value = 99
  end

  def say_value
    @value
  end
end
First.new.say_value # => 99

Second = First.clone
Second.class_eval do
  define_method :say_value do
    @value + 100
  end
end
Second.new.say_value # => 199

Neat.

I’m not sure quite why you would want to clone a class to take advantage of re-use - rather than extract to a module (and share the implementation that way) or, if there’s a strong relationship that doesn’t violate the LSP etc. then look for some kind of inheritance-based design.

But, I guess you could work some kind of cool ultra-dynamic super-meta system from it. Perhaps someone with way more of a Ruby-thinking brain than me could offer some thoughts?

The project I’m currently working on uses both the Asset Packager and Distributed Assets to ensure we have only a few external assets, and that we can load assets across more than one host - all so that the pages for our site load nice and quick.

Unfortunately, wiring in the Asset Packager plugin caused the Distributed Assets plugin to break, and I spent an hour or two tracking it down yesterday. The cause? Asset Packager redefines the compute_public_path method.

# rewrite compute_public_path to allow us to not include the query string timestamp
    # used by ActionView::Helpers::AssetTagHelper
    def compute_public_path(source, dir, ext=nil, add_asset_id=true)
      source = source.dup
      source << ".#{ext}" if File.extname(source).blank? && ext
      unless source =~ %r{^[-a-z]+://}
        source = "/#{dir}/#{source}" unless source[0] == ?/
        asset_id = rails_asset_id(source)
        source << '?' + asset_id if defined?(RAILS_ROOT) and add_asset_id and not asset_id.blank?
        source = "#{ActionController::Base.asset_host}#{@controller.request.relative_url_root}#{source}"
      end
      source
    end

Distributed Assets works by chaining compute_public_path - decorating the calculated path, adding the asset host prefix onto the url. But, Asset Packager works by defining the method into ActionView::Base. So, when DistributedAssets::AssetTagHelper is included with ActionView::Helpers::AssetTagHelper, it chains a (now) hidden method.

But, the only places that use the new compute_public_path code inside the Asset Packager Helper (which just avoids using the query string timestamp) is within Asset Packager itself.

So, I tweaked the implementation of AssetPackageHelper to

def compute_public_path_for_packager(source, dir, ext, add_asset_id=true)
  path = compute_public_path(source, dir, ext)
  return path if add_asset_id
  path.gsub(/\?\d+$/, '')
end

def javascript_path(source)
  compute_public_path_for_packager(source, 'javascripts', 'js', false)       
end

Beware the monkey patch.

Rails makes heavy use of a declarative style around it’s codebase- for example the has_one and belongs_to declarations inside ActiveRecord (amongst others). These are just class methods defined on modules, letting Rails wire up relationships, but they read like fully-fledged statements within a mini-language:

class Student < ActiveRecord::Base
  has_many :tutorials

We can take advantage of the same in our code- letting us write in a declarative-style (hopefully revealing stronger intent) and reduce the amount of code we write (by using declarations to meta-program for us). I posted a little while ago that we’d used such an approach for marking attributes as immutable.

Anyway, back to the title of the post- #instance_exec, it’s a method defined in Ruby Facets. Like it’s documentation says, it’s equivalent to instance_eval but also lets you pass parameters- roll on the meta magic.

Let’s say we’re writing a system to calculate the monthly salary payment for an employee. We want to be able to say what the payment is rather than how it’s calculated.

class FullTimeEmployee
  include Employee

  bill_at {|hours| 50.pounds_sterling * hours}

In the snippet above, we’re making a declaration - defining the relationship between an hourly rate and the number of hours the employee works. We can implement bill_at in Employee as follows:

module Employee
  module ClassMethods
    def bill_at &block
      define_method(:calculate_bill) do |hours|
        instance_exec hours, &block
      end
  ...

Our assertion could be:

assert_equal 500.pounds_sterling, employee.calculate_bill(5.hours)

I’d read a while back that Apple were going to include Sun’s DTrace tool in the now newly released Leopard - underpinning Instruments (formerly known as Xray).

What I hadn’t noticed was that the build of Ruby 1.8.6 included in Leopard (patchlevel 36 with some extras) also includes Ruby probes from the lovely folks at Joyent. Apple have ported everything. So, out of the box, Ruby works with DTrace! Sweet!

From Sun’s page:

DTrace is a comprehensive dynamic tracing facility that is built into Solaris and can be used by administrators and developers to examine the behavior of both user programs and of the operating system itself. With DTrace you can explore your system to understand how it works, track down performance problems across many layers of software, or locate the cause of aberrant behavior.

Joyent have got some sample scripts in their Subversion repository, and, what looks like some nice extensions for Rails (DTrace probes can be fired from within Ruby). Blimey!

For those (running Leopard) and want to just see something, try this out. It’s a script that traces the number of times a method is called. Once downloaded, all you need to do is

$ chmod +x functime.d
$ sudo ./functime.d -p <pid>

Replacing <pid> with the process id for your target Ruby process. Once you stop tracing (or your target process ends you’ll see some nice text formatted output. Including something like the following (from my current project):

Hash                     key?                     25123         15       384174
Module                   each_object                  1     390824       390824
Module                   included_in_classes          1     391025       391025
Module                   reloadable_classes_with      1     391203       391203
Module                   reloadable_classes           1     391835       391835
ActionController::Routin load_routes!                 1     394864       394864
ActionController::Routin reload                       1     398364       398364
Class                    reset_application!           1     398947       398947
Class                    reset_after_dispatch         1     400474       400474
ActionController::Routin connect                     39      13192       514506
Array                    include?                  3353        166       556605
ActionController::Routin build                       47      13116       616459
ActionController::Routin add_route                   47      13329       626488
Array                    each                       874        780       682423
Object                   load                         2     376432       752864
Module                   silence                      2     391586       783172
Kernel                   gem_original_require        28      43460      1216884
Array                    select                     316       4106      1297580
Array                    collect                    351       4124      1447524
Module                   local_constants             91      17120      1557986
Module                   new_constants_in            25      93885      2347139
Object                   require                     50      58495      2924783

The columns (from left to right) are: Class, Method, Count (number of times method was called) and then the average and total micro seconds. The d script was one of the Joyent examples. Arstechnica also have a very cool example showing the call stack from a system call.

What’s more amazing is this doesn’t require any recompilation, special flags, running in special environments. It’s available from the off!

If all that’s whetted your appetite (and I’m not sure how it couldn’t), head on over to the DTrace how to for a little more explanation of how it all works.

Time to get stuck in and see whether anything emerges about my current project. Very cool stuff!

Whilst working on a bit of code the other day we found this little nugget in the RDoc for Object#instance_variable_set:

Sets the instance variable names by symbol to object, thereby frustrating the efforts of the class‘s author to attempt to provide proper encapsulation.

Brilliant!

In preparation for a Ruby project I wanted to play around with Rails’ page caching a little over the weekend. Naturally, I wanted to find a way to ensure that my controller was writing files correctly so that Apache (or any other HTTP server) can then serve the static content directly (without hitting my Rails processes). Since I’m also digging RSpec at the moment, I wanted to find a way of expressing it nicely in my specifications. This is what I ended up with.

Here’s the relevant controller’s specification:

describe UsersController, "caching" do
  include CachingExampleHelper

  it "should cache new users page" do
    requesting {get :new}.should be_cached
  end

  it "should not cache the activation page" do
    requesting {get :activate, :activation_code => 'blah'}.should_not be_cached
  end
end

The key part is the requesting line. That’s my extension that encapsulates the request that should be cached, so that we can both make the request and check that it ended up with a file being written to the filesystem.

module CachingExampleHelper
  ActionController::Base.public_class_method :page_cache_path
  ActionController::Base.perform_caching = true

  def requesting(&request)
    url = request.call.request.path
    ActionController::Base.expire_page(url)

    request.call
  end

  module ResponseHelper
    def cached?
      File.exists? ActionController::Base.page_cache_path(request.path)
    end
  end

  ActionController::TestResponse.send(:include, ResponseHelper)
end

Note that we have to expose the pagecachepath method so we can calculate the location for where the cached files will get written, and then also explicitly enable caching. When we call requesting giving a block to the request we want to make (such as get :index to invoke the index action), we capture the block as a Proc, invoke it and then read the request’s path from the generated TestResponse object. This lets us then first expire the page (in case it was cached previously), and then make the request. Finally, we then check whether the cached file exists by mixing in a cached? method to the Response.

Unfortunately, the code above will make 2 requests per assertion. But, I’d rather have it read nicely for the time being. Any suggestions on how you could tidy this up further would be well received!

I’m not sure I’d want to run these tests all the time, maybe something a little lighter to ensure that we tell Rails to caches_page for relevant actions in our controllers, but as a definite re-assurance it seems ok.

I’m working on a little pet Rails project and wanted to use Mocha to isolate my controller tests a little from what is usually encouraged, especially in light of what happened last time. I figured I’d give Mocha a go.

I wanted to verify that my service was called with objects that look equal to what I was expecting. Ordinarily, it will compare instances - are they the same object. I didn’t want that, but rather wanted to test that the values of my (ActiveRecord) objects were the equal. Easy. Mocha allows with to be called with a block that is evaluated to compare the objects.

My tests look something like this

def test_should_ask_service_to_calculate_new_prices_for_booking
  address = Address.new(:postcode => "W1 1QE")
  booking = create_booking

  OrderBookingService.expects(:calculate_price).with(booking, address) do |b, a|
    b.attributes == booking.attributes && a.attributes == address.attributes
  end

  post :calculate_price, :booking => booking, :collection_address => address
end

It’s something I’ve had to do many times in the past with JMock and NMock - I want to test equality for an aspect of some other object but it’s always meant writing rather a lot of code. Neat.

As alluded to in my last post, I’ve been looking at enhancing my Typo plugin by adding some kind of relevance to the results.

Since I’m considering articles that share some tag to be considered related, I’m also going to consider articles that share more than one tag more relevant. This is a post that covers my adding that support, as well as some much needed Ruby-ification (is that a word? :p) and refactoring of the code following comments from Piers Cawley on the code in my last post.

Every now and then I get an urge to go back over some test code to figure out what I like and I don’t like, and try and find ways to make the tests more useful as a means of communication.

I place a lot of importance, as a developer, on being able to think about writing code through tests – statements of things that I want my code to do. And, when going to work with code that I may not have written (or that I may have written a long while ago) I look to the current tests to give me an idea about how to work with it.

With Ruby, I’ve found that because I’m still very much learning the language (I’ve just received a little baiting here in the office because of it :)) I find myself more inclined to explore the language through refactoring, taking examples in Rails and other Ruby apps for inspiration.

So this evening I did just that, and spent some time going back over some code I wrote earlier as part of my Related Articles Plugin.

I thought I’d finally take a look at Mongrel and see whether it was worth changing my configuration from Lighttpd/FCGI. So far it’s been pretty stable (with only the odd restart needed every few month or so), but Mongrel’s had such a good response I couldn’t resist.

However, it looks like Ruby Gems is failing to work on my VPS. Below is what I run and the output:

sudo gem install daemons
   Password:
   Bulk updating Gem source index for: http://gems.rubyforge.org
   Killed

I was originally on an old version of Gems, so I downloaded the latest release (0.9.0), but same result.

If I run the same above on my PowerBook G4

pablo:~/work/mephisto/trunk pingles$ sudo gem install daemons
Password:
Attempting local installation of 'daemons'
Local gem file not found: daemons*.gem
Attempting remote installation of 'daemons'
Updating Gem source index for: http://gems.rubyforge.org
Successfully installed daemons-0.4.4
Installing RDoc documentation for daemons-0.4.4...

D’oh. Not too sure what’s causing it, and a quick poke around Google and Google Groups didn’t yield any information about log files etc. Does anyone have any suggestions?

of some domain driven design at work. Although I posted somewhat about it a few days ago, I couldn’t hold back from some DDD loving.

The example he provides is largely because of a code smell that will result in your model as a result of certain association modeling: the tangled mess of code that will appear if the concept wasn’t introduced as a fully fledged member of your model.

I won’t repost the example, but it’s essentially concerned with news subscriptions via some kind of feed. The key to his example was the identification and introduction of a `Subscription`. Although it would have existed implicitly in an association between a user and their news categories, the explicitness has immense value. Particularly in readability through it’s encapsulation – you’ve gotta love OO :)

I’ve noticed a little instability with my setup recently, at present I have Lighty configured to manage the FastCGI processes itself.

The relevant part of my config looks as follows:

fastcgi.server= (".fcgi" =>
("oobaloo.co.uk" =>
  (
    "socket" => "/tmp/oobaloo-lighttpd-fcgi.socket",
    "bin-path" =>
  "/var/www/servers/www.oobaloo.co.uk/current/public/dispatch.fcgi",
    "bin-environment" => ("RAILS_ENV" => "production" ),
    "max-load-per-proc" => 4,
    "min-procs" => 1,
    "max-procs" => 3,
    "idle-timeout" => 90
  )
)

However, I’ve read people suggest that it’s better (when running more than one rails application) to instead use static processes, and there’s a page on the Rails wiki describing this setup.

So, I changed my Lighty config to the following:

fastcgi.server = (".fcgi" =>
  ("localhost-7000" =>
    ("host" => "127.0.0.1", "port" => 7000,
  "bin-environment" => ("RAILS_ENV" => "production"))),
  ("localhost-7002" =>
    ("host" => "127.0.0.1", "port" => 7002,
  "bin-environment" => ("RAILS_ENV" => "production"))),
  ("localhost-7001" =>
    ("host" => "127.0.0.1", "port" => 7001,
  "bin-environment" => ("RAILS_ENV" => "production")))
)

and instead started the processes manually using the `spawner` with the following script:

!/bin/sh

/var/www/servers/\ www.oobaloo.co.uk/current/script/process/spinner \ -c ’/var/www/servers/\ www.oobaloo.co.uk/current/script/process/spawner -p 7000 -i 3’ \ -d

However, I noticed that over time new processes were being created to handle `dispatch.fcgi` for the blog. This got to the point with about 6 or 7 processes that would choke the VPS and cause it to become relatively unusable.

Is this normal for other processes to be created? Under what conditions would additional processes be spawned?

Anyway, for the time being I’m back on Lighty managed processes—at least the number of processes remain static.

Last week I noticed that Rails had officially had it’s 1.1 release, and since then I’ve used it for a few things, but my production server is still using the Rails 1.0 gem. Over the weekend I also noticed that Typo was being cleaned up a little ready for it’s 4.0.0 release that would also be compatible with 1.1, but that the trunk contained essentially everything from the upcoming release. So I decided to bite the bullet and give it a go! I couldn’t resist Piers’ advice.

However, since my current Ruby install was 1.8.3, I needed to upgrade a few things: Typo, Rails and Ruby itself!

Here’s what I did.

As the final iteration of the first release winds down for Sureboss, one thing I’ve been trying to do recently is to test that when some action is performed on a controller, there is some interaction with other participating classes.

In my case, it was testing that when an administrator of the shop clicks on a link to mark an order as posted an email is automatically sent to the customer to let them know a package is on it’s way.

Fortunately, when creating a Rails mailer, Rails automatically writes some unit tests for you that test your email is templated correctly and looks as it should. For those non-familiar with Rails or ActionMailer and are interested in how this is done, this is roughly how it works.

Running the `generate mailer Notification posted_receipt` script will put together a `Notification` class that contains one type of email—`posted_receipt`. It also generates a fixture for you in the `test/fixtures/Notification` directory. This is just a plain old text file that contains the expected output for your email’s unit test. Which, immediately after running the generator, will look as follows:

def test_receipt
  @expected.subject = 'Sureboss Receipt'
  @expected.body    = read_fixture('receipt')
  @expected.date    = Time.now
  @expected.from    = "info@sureboss.com" 
  @expected.to      = @order.customer.email_address
end

assert_equal @expected.encoded,
  StoreMailer.create_receipt(
    @order,
    @expected.date
  ).encoded

Rails’ ActionMailer automatically wires up `create_` and `deliver_` prefixes for your method, allowing you to both create a `TMail` object that represents your email in the first instance, and for attempting delivery of your email in the second. The purpose of the unit test is merely that when asked to send an email about an order, it renders the content correctly, and recipient/senders are also correct. This works through calling the `create_` method, and checking the encoded content of the message is as it should be, and that the recipient/sender all look correct.

However, this only covers the low-level operation of the classs. We need to wire it in to our controller, but how can we test that this happens while avoiding the need for actually trying to connect to an SMTP server? Read on to see the way I did it.