DBIx::Class join, prefetch

Over the last 15 years I've joined two database tables into a one-to-many relationship about a billion times. But I've never had any software let me loop the left table and right tables separately before. Turns out this magic has been in DBIx::Class for several years.

Runnable demo.

When table 'foo' is in a one-to-many relationship with table 'bar', you can:

my $foo_rs = $schema->resultset('Foo')->search(
   {},
   {
      join     => 'bars',
      prefetch => 'bars',
      order_by => 'me.id',
   }
);

while (my $foo = $foo_rs->next) {
   printf "%s\n", $foo->id;
   if (my $bar_rs = $foo->bars) {
      while (my $bar = $bar_rs->next) {
         printf "   %s\n", $bar->desc;
      }
   }
}

Output:

1
2
   aaa
   bbb
   ccc
3

Still one SQL statement against the database (fast!). Much prettier than ugly manual de-duping of redundant columns.

Catalyst and Quality Assurance bliss

I've fallen in love with website testing. Again. Catalyst and WWW::Mechanize are awesome. The scenario below takes several monotonous and error prone minutes to QA as a human, or several seconds as a series of automated tests. New feature + test coverage == Joy!!

$work[0] test count is back over 3,000 again and Hudson is keeping an eye on everything for us every 30 minutes. Very cool.

$ perl -Ilib t/avail_book_SG-autoenroll_cancel.t 
[debug] Statistics enabled
1..21
ok 1 - GET http://localhost.omnihotels.com/
ok 2 - Content-type:
ok 3 - title>
ok 4 - Select a City
ok 5 - No errors
ok 6 - !- BEGIN: om/rr/RR1_2.tt -->
ok 7 - !- BEGIN: om/rr/RR1_3.tt -->
ok 8 - !- BEGIN: om/rr/RR1_4.tt -->
ok 9 - !- BEGIN: om/rr/RR1_5.tt -->
ok 10 - Res ID(s) are 40002401426
ok 11 - SG number 62379458 created by sg_auto_enroll checkbox
ok 12 - click Activate Account Password & Preferences
ok 13 - !- BEGIN: om/sg/SI4.tt -->
ok 14 - submit SI4 (form fields should have been defaulted for us)
ok 15 - !- BEGIN: om/sg/SI3.tt -->
ok 16 - submit SI3
ok 17 - !- BEGIN: om/sg/SG6.tt -->
ok 18 - GET /?Phoenix_state=clear&pagedst=RR2
ok 19 - !- BEGIN: om/rr/RR1_5.tt --> Retrieved 40002401426 for cancellation
ok 20 - Click 'Cancel Reservation'
ok 21 - Reservation 40002401426 cancelled
------------------------------------------------
$ cat t/avail_book_SG-autoenroll_cancel.t 
use Test::WWW::Mechanize::Catalyst 'Phoenix';

use strict;
use Test::More tests => 21;
use FindBin qw($Bin);
use lib "$Bin/lib";
use phoenix_test;

my %SI3_args = (
   loginName => ('jhannah' . time),
   password  => 't123',
   EMAIL     => 'jhannah@omnihotels.com',
   Question  => 'CITY',
   Answer    => 'Shmomaha',
);

my $mech = Test::WWW::Mechanize::Catalyst->new;
my $pt = phoenix_test->new(mech => $mech);

my $cro_resno = $pt->book(sg_auto_enroll => 'on');
my $msg = qr/Your Select Guest number is (\d+)/;
my ($sgnum) = ($mech->content =~ $msg);
ok($sgnum,                                    
   "SG number $sgnum created by sg_auto_enroll checkbox");

$msg = 'Activate Account Password & Preferences';
$mech->follow_link_ok({text => $msg},
   "click $msg");

my $msg = '!- BEGIN: om/sg/SI4.tt -->';
$mech->content_contains($msg,                 $msg);
$mech->submit_form_ok({},                     
   "submit SI4 (form fields should have been defaulted for us)");

my $msg = '!- BEGIN: om/sg/SI3.tt -->';
$mech->content_contains($msg,                 $msg);
$mech->submit_form_ok({
   with_fields => \%SI3_args,
},                                            "submit SI3");

my $msg = '!- BEGIN: om/sg/SG6.tt -->';
$mech->content_contains($msg,                 $msg);

$pt->cancel($cro_resno);

Portland's marketplace of ideas... in Omaha

In Portland tomorrow night:

Robert Buels presents "The Amazing Miracle of DBIx::Class"

Rob will give an introduction to and overview of DBIx::Class. It's an object-relational mapping framework, much better than the old Class::DBI, and it will make your life easier if you are currently writing a lot of SQL in your Perl.

In Omaha tonight:

Randal "merlyn" Schwartz (who lives in Portland) presents "Forget the ORM! Persistent Data with non-traditional databases."

What an Object-Relational Mapper is, why it sucks, and a couple dozen alternatives (most open source, many with Perl bindings).

Gratuitous Moosey ego stroking

The Godfather of Moose gave me mad shout-outs on his blog for some minor tweaking I did to the Moose website.

Later I expressed some of my recent concern about academia, and he offered this sage advice:

11:37 <@stevan> jhannah: go into their lab, climb up on one of the work benches and yell "ANY OF YOU NANCY BITCHES STEAL MY CODE AND I WILL CUT YOU INTO LITTLE PIECES"
11:37 <@stevan> then kick over a few bunson burners and walk out
11:37 <@stevan> they wont bother you in the lunch room anymore
11:38 <@perigrin> stevan did something like this in Art School
11:38 <@perigrin> 'cept s/bunson burners/cerulean blue/
11:38 <@stevan> I offcentered their pots on the wheel

Gource: Programmers as action heroes

Check out the history of Perl development 1988-present. As you can see, when you speed it up enough and use lots of colors and movement suddenly programming is exciting to watch! (For a few minutes.) :)

They created that with Gource (watch other, even prettier videos).

At $work[0] we have CVS + SVN history back to 2000. It'd be neat to throw Gource against that and see what I (and my coworkers) have been doing with our lives for the last 10 years.

Catalyst::Plugin::AutoCRUD is a sexy beast

Wow. It took me about 30 seconds to add AutoCRUD to my existing Catalyst application. And now I've got a very shiny web 2.0'y CRUD (Create, Read, Update, Delete) application for all my databases. Double clicking pulls up the edit window.

Yes, this wheel has been invented scores of times over the last 10 years. But boy is this one shiny.

Firefly shot me 51 days into the future

I feel pretty good about my project this weekend. I used the University of Nebraska Firefly Cluster for a real project for the first time, completing 51.6 DAYS worth of data crunching while I slept between 3am and 9:30am this morning. The code I ran is my fork + branch of a bunch of Perl from Harvard against the latest human genome from UCSC. Masochists can read all the gory details (username: guest password: guest) and suggest improvements.

Computers are pretty cool sometimes. :)

ack is awesome!

Wow! People have been telling me this for years, but I was getting along OK with my old school Unix craziness:

$ find ./ | grep -v svn | xargs grep '\-\>Promotion'

But today I couldn't get that working. Something with xargs escaping of -> just refused to fly.

So I installed ack and ran it:

# cpan App::Ack
$ ack '\-\>Promotion'

And it found what I was looking for with zero fuss!

Control/SG.pm
335:      $self->Promotion($promo_hash);

View/Web/Phoenix/lib/Phoenix/Controller/SG.pm
2844:      $c->log->debug("you have these PromotionCode's already: " . (join ", ", keys %{$sg->Promotion}));
2846:      unless ($sg->Promotion->{$preauth}) {

Holy buckets! That is AWESOME! Thanks Andy!

Pipeline evolution - again!

It evolved again! This too, shall pass. :)

Pipeline evolution

Over the last couple of weeks I've been writing and changing a bunch of software trying to answer some research questions. I find the visual evolution of this pipeline interesting. Maybe you will too.

Sneak peak of the first chart:

$76M? Thanks Catalyst!

Woot! Catalyst, our favorite Perl web development framework, served us well in 2009:

Over $76 million in revenue was booked through omnihotels.com, a 13.5% increase over the previous year; 460,902 room nights were booked during 2009, representing a 28% increase.

...so complex that you need a large IDE to comprehend it

This is an excerpt from an Omaha.pm thread today.

We're discussing Eliminating Inheritance Via Smalltalk-Style Traits by Curtis "Ovid" Poe.

In his paper, Curtis wrote:
Anyone forced to use "vi" (not even "vim") while trying to create an emergency patch of broken code over a slow telnet connection at 2:30 in the morning is going to get very irritated if your codebase is so complex that you need a large IDE to comprehend it.

Trey wrote:
That is kind of silly. If you're shop is using IDE and developing under that sort of framework, your support people should be familiar with the code and the methods used to program it/maintain it.

Agreed. And have the capacity to perform emergency maintenance remotely via whatever means are necessary to do so.

I don't think he's arguing that slow Internet connections are a good idea. I think he's saying they happen.

Trey wrote:
That's sort of like saying that somebody is going to be irritated because they can't stoke up a hydroelectric plant with bellows...So what??

I love your analogy. :) So what? So your business is losing money in the scenario he describes.

I see this an indictment of relying too heavily on fancy developer tools to gloss over crappy code. If my code sucks in text form, but when I right-click on it a million lines of code in my IDE jumps to my rescue and explains it to me in pretty colors, then I'm helpless without that IDE. That sounds like a bad idea.

In the stack of technology ignorance and dependency I've planted my flag here:

- I don't really know or care about sub-atomic particles.
- I don't really know or care how electrons flow through conductors.
- I don't really know or care about chip design.
- I don't really know or care how hard drives, memory, and monitors actually work.
- I don't really know or care how the Linux kernel works.
- I don't really know or care how perl is written (in C).
- I don't really know or care how most of perl's core packages work (in Perl/XS/C).
- I don't really know deeply or care how most of the CPAN modules I use work (in Perl/XS/C).
- I know EXACTLY how the text of the Perl I wrote works. I do recursive grep's through text files to find out anything I need to know.
- Syntax highlighting is a crutch for the weak.
- Method auto-discovery is a crutch for the weak.
- Per-keystroke debugging is a crutch for the weak.
- Re-runnable images of crashes are a crutch for the weak.
- All that other fancy crap in your IDE is fancy crap I don't need. For the weak. :)

Objectively, my gut-opinions about exactly where that line is are silly. Aren't they?

If you take me and my Apple laptop running Terminal.app to ssh into a server and run vi, and next to me you sit a .NET guy running the latest Visual Studio on a mapped drive; is the total complexity of my dependency stack much smaller than his? (OS X is how many millions of lines of code?)

I don't know. I've now spent an hour arguing with myself about it. :)

In any event, I don't thing it's the use or avoidance of an IDE that makes good or poor code. You can create beautiful or ugly code with the simplest or fanciest of tools.

I do think he has a point. If your "...codebase is so complex that you need a large IDE to comprehend it." Then you're doing something wrong. Use an IDE because you want to, not because it's impossible to survive your mess without it.

/me shakes his fist at XML Spy addicts

Ponder,

j

Omaha's First-Ever Catalyst Workshop: Success!

And there was much rejoicing!

I thoroughly enjoyed presenting the workshop. Thanks for coming! We'll have to do it again (or an advanced / specific topics version) someday soonish?

See you in January!

Omaha's first-ever Catalyst Wed Dev Workshop

Trying a new format at next month's meeting:

http://jays.net/wiki/Catalyst_Workshop

Wish me luck! Should be fun. :)

Future Perl snuck up on me

Apparently I'm out of touch. Look at all this Perl6:: stuff implemented in Perl5. For example, Catalyst uses Perl6::Junction.

And Moose is starting to look like Perl7 to me. Here perigrin explains what this code is doing.

These ain't your momma's Perl. Heck, these ain't even 2005's Perl. :)

Term::ProgressBar

Here's a super easy CPAN module I started using last week. Ever have a long running process that a human is watching, and you want to add a progress bar? Here you go!

use Term::ProgressBar;
my $progress = Term::ProgressBar->new({
   name  => 'MySlowThing()',
   count => 100,
   ETA   => 'linear', 
});
$progress->max_update_rate(1);

for (1..100) {
   sleep 1;
   $progress->update($_);
}

Gives you a nifty ASCII timer bar with major and minor movements, updating clock, the whole nine. :)

MySlowThing():  26% [====        *     ]1m16s Left

GraphViz prettyness

My first mapping of bioinformatics alignment score to how dark a GraphViz line should appear, and switching colors based on other criteria... Perl++ GraphViz++

I don't think it's very informative, but there's no denying it's pretty. :)

Perl, Catalyst: $100M in sales this year

If I'm reading this spreadsheet correctly, it appears our $work website is on pace to book just shy of $100 million this year. Thanks Linux, GNU, Apache, Perl, CPAN!

Open source: A cast of thousands paying my mortgage, one patch at a time. I'm glad I can pitch in in small ways too! :)

Chiselling chunks off of BioPerl

There's been renewed conversation about breaking BioPerl up into smaller distributions. I'm trying to help, citing the Catalyst development model as an example of how the world could be better.

For some values of 'better'. -grin-

perl5 introspection (thanks Moose!)

Many dynamic languages have a way to get a list of all methods for an object built into them.

I don't think perl5 does. But Moose adds it. :)

$ cat j.pl
package X;
use Moose;
sub dosomething1 {}
sub dosomething2 {}
sub dosomething3 {}

package main;
my $x = X->new();
print join ", ", sort $x->meta->get_all_method_names;
print "\n";

$ perl j.pl
BUILDALL, BUILDARGS, DEMOLISHALL, DESTROY, DOES, does, 
dosomething1, dosomething2, dosomething3, dump, meta, new

Neato. :)

More info on Moose: http://moose.perl.org