librelist archives

« back to archive

job hooks

job hooks

From:
Ryan Carver
Date:
2010-03-21 @ 05:29
Hi, I'm working on a locking and retry plugin for resque (initially based on
Chris's locked_jobs [1])

http://github.com/rcarver/resque-lock-retry

After an initial implementation which overrode perform, Chris suggested
adding before_perform and after_perform callbacks for jobs. I've reworked
the plugin to do that, and now am looking for feedback on the hooks
themselves.

http://github.com/rcarver/resque-lock-retry/blob/master/lib/resque/job_hooks.rb

For this plugin, I didn't actually need before_perform or after_perform, but
instead around_perform and on_failure. All of the available hooks are:

* before_perform: Called with the job args before perform. If it raises
Resque::DontPerform, the job is aborted.
* after_perform: Called with the job args after it performs.
* around_perform: Called with the job args, then is expected to yield to
perform the job.
* on_failure: Called with the exception and job args if any exception occurs
while running the job (or hooks).

In all non-exception cases Resque::Job#perform returns true or false to
indicate whether the job was performed.

Anyone have opinion on this? Feedback on lock/retry welcome as well but I'd
like to first extract the hooks to resque proper.

Thanks,
 - Ryan


[1]
http://github.com/defunkt/resque/commit/7bbf3f06d0cccb5bdcbedc23e943cbe96b46cc53

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-03-23 @ 20:21
On Sat, Mar 20, 2010 at 10:29 PM, Ryan Carver <ryan@fivesevensix.com> wrote:

> For this plugin, I didn't actually need before_perform or after_perform, but
> instead around_perform and on_failure. All of the available hooks are:
>
> * before_perform: Called with the job args before perform. If it raises
> Resque::DontPerform, the job is aborted.

Sweet. What happens when a different exception is thrown - is it sent
along to the Failure backend and reported?

> In all non-exception cases Resque::Job#perform returns true or false to
> indicate whether the job was performed.

Beautiful.

Chris

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-03-23 @ 20:24
>
> Sweet. What happens when a different exception is thrown - is it sent
> along to the Failure backend and reported?
>


Yes. It's first caught and handled by the on_failure hook (if available),
then re-thrown.

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-03-23 @ 20:28
On Tue, Mar 23, 2010 at 1:24 PM, Ryan Carver <ryan@fivesevensix.com> wrote:

>> Sweet. What happens when a different exception is thrown - is it sent
>> along to the Failure backend and reported?
>
> Yes. It's first caught and handled by the on_failure hook (if available),
> then re-thrown.

Okay, wonderful.

Should the plugin and hooks API be documented in the README (probably
not), the wiki with a link from the README (maybe), or in a PLUGINS.md
document (probably)?

Chris

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-03-23 @ 20:32
Thumbs up on this api then? I'll port it to a fork of resque with some tests
asap.


Should the plugin and hooks API be documented in the README (probably
> not), the wiki with a link from the README (maybe), or in a PLUGINS.md
> document (probably)?


I'll start with a PLUGINS.md, then we can then link to that from the README
and/or wiki.

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-03-23 @ 20:58
Thumbs up.

On Tue, Mar 23, 2010 at 1:32 PM, Ryan Carver <ryan@fivesevensix.com> wrote:
> Thumbs up on this api then? I'll port it to a fork of resque with some tests
> asap.
>
>>
>> Should the plugin and hooks API be documented in the README (probably
>> not), the wiki with a link from the README (maybe), or in a PLUGINS.md
>> document (probably)?
>
> I'll start with a PLUGINS.md, then we can then link to that from the README
> and/or wiki.
>



-- 
Chris Wanstrath
http://github.com/defunkt

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-03-23 @ 23:02
I just modified the `Resque::Jobs::Locked` class:

http://github.com/defunkt/resque/commit/4f686cf2bd486fd5d48fe9a407ad6ade707ed1d8

It is now a module to be activated like so (in theory):

    class UpdateNetworkGraph
      extend Resque::Jobs::Locked

      def self.perform(repo_id)
        heavy_lifting
      end
    end

Because we're overriding methods instead of doing ActiveRecord-style
callbacks, functionality can not be chained. Is that good or bad?

Chris

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-03-23 @ 23:11
FYI, here's the job hooks implementation in progress...

https://github.com/rcarver/resque/compare/master...hooks


Because we're overriding methods instead of doing ActiveRecord-style
> callbacks, functionality can not be chained. Is that good or bad?
>

If you're ok with using super, and hope that plugin authors do it, it can be
done pretty easily. I just added some tests.

https://github.com/rcarver/resque/commit/0d1c97848c5548f51c64d8682e3acf9dcd031b99


  module BeforeOne
    def before_perform(history)
      super rescue NoMethodError
      history << :before_one
    end
  end

  module BeforeTwo
    def before_perform(history)
      super rescue NoMethodError
      history << :before_two
    end
  end

  class ManyBeforesJob
    extend BeforeOne
    extend BeforeTwo
    def self.perform(history)
      history << :perform
    end
  end

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-03-23 @ 23:38
On Tue, Mar 23, 2010 at 4:11 PM, Ryan Carver <ryan@fivesevensix.com> wrote:
> FYI, here's the job hooks implementation in progress...
> https://github.com/rcarver/resque/compare/master...hooks

Lookin' good. Reading the code made me realize an `extend` based
approach only works for jobs that inherit from Resque::Jobs, so maybe
it's better to just stick to subclasses.

Here's take #2, subclass style using the new hook API:

http://github.com/defunkt/resque/commit/7ef5579a1c87c310ebef37115788f9ce136e9825

I think doing the `super` dance with Ruby's module system could become
a nightmare for plugin authors. Your tests for before and after look
fine, but what about `around_perform`? Should we just stick to
inheritance?

Chris

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-03-24 @ 00:02
>
> Lookin' good. Reading the code made me realize an `extend` based
> approach only works for jobs that inherit from Resque::Jobs, so maybe
> it's better to just stick to subclasses.
>

Hmm, you lost me. How does extend required inheriting from Resque::Jobs? Do
you mean an AR-like callbacks? Then yeah it requires some kind of base class
- which isn't terrible I guess.

Actually, what if we had something like this:

module MyPlugin
  extend Resque::Hooks
  before_perform do
    ...
  end
end

Resque::Hooks would have to be pretty clever to manage hook lists for each
job (actually I think I went down this road once already and it got ugly).


Here's take #2, subclass style using the new hook API:
>
>
> http://github.com/defunkt/resque/commit/7ef5579a1c87c310ebef37115788f9ce136e9825
>
> I think doing the `super` dance with Ruby's module system could become
> a nightmare for plugin authors. Your tests for before and after look
> fine, but what about `around_perform`? Should we just stick to
> inheritance?


I agree it gets weird. But, I *really* like that you can include multiple
bits of functionality. The thing that drove this originally was implementing
both Lock and Retry. I have cases where one, the other, or both are useful.
These two happen to not conflict, but yeah it's possible. Also, I'm curious
how often `around_perform` is needed.

Long shot: is there a way to walk the `super` chain from outside the class?
Meaning, Resque::Job#perform could call the hidden methods instead of
requiring authors to call `super`.

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-03-24 @ 01:21
On Tue, Mar 23, 2010 at 5:02 PM, Ryan Carver <ryan@fivesevensix.com> wrote:

> Hmm, you lost me. How does extend required inheriting from Resque::Jobs? Do
> you mean an AR-like callbacks? Then yeah it requires some kind of base class
> - which isn't terrible I guess.

I completely confused myself. Ignore me.

> I agree it gets weird. But, I *really* like that you can include multiple
> bits of functionality. The thing that drove this originally was implementing
> both Lock and Retry. I have cases where one, the other, or both are useful.
> These two happen to not conflict, but yeah it's possible. Also, I'm curious
> how often `around_perform` is needed.

Confusion aside, I agree that composable plugins are awesome.

I wonder if it's worth checking out something like this:

http://github.com/wycats/simple-callbacks-refactor

Specifically:


http://github.com/wycats/simple-callbacks-refactor/blob/master/lib/new_callbacks.rb

> Long shot: is there a way to walk the `super` chain from outside the class?
> Meaning, Resque::Job#perform could call the hidden methods instead of
> requiring authors to call `super`.

You could do this:

https://gist.github.com/8d1bbe139a6d574540d5

Unfortunately `Module.ancestors` doesn't keep track of `extend` uses
like it does with `include`. Maybe there's another method I don't know
about for that.

-- 
Chris Wanstrath
http://github.com/defunkt

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-03-28 @ 07:18
How about this?

https://github.com/rcarver/resque/compare/hooks...hook-plugin

Pros:
 - Plugin authors don't need to worry about super
 - Both jobs and plugins can implement hooks and it behaves sensibly
(currently job hooks run last)

Cons:
 - Some metaprogramming
 - around_perform is kind of weird, but what's the alternative? It's the
reason we started worrying about super in the first place.


Chris: this is basically your plugin technique but uses `extend_object`
instead of `extended`.



On Tue, Mar 23, 2010 at 6:21 PM, Chris Wanstrath <chris@ozmm.org> wrote:

> On Tue, Mar 23, 2010 at 5:02 PM, Ryan Carver <ryan@fivesevensix.com>
> wrote:
>
> > Hmm, you lost me. How does extend required inheriting from Resque::Jobs?
> Do
> > you mean an AR-like callbacks? Then yeah it requires some kind of base
> class
> > - which isn't terrible I guess.
>
> I completely confused myself. Ignore me.
>
> > I agree it gets weird. But, I *really* like that you can include multiple
> > bits of functionality. The thing that drove this originally was
> implementing
> > both Lock and Retry. I have cases where one, the other, or both are
> useful.
> > These two happen to not conflict, but yeah it's possible. Also, I'm
> curious
> > how often `around_perform` is needed.
>
> Confusion aside, I agree that composable plugins are awesome.
>
> I wonder if it's worth checking out something like this:
>
> http://github.com/wycats/simple-callbacks-refactor
>
> Specifically:
>
>
> 
http://github.com/wycats/simple-callbacks-refactor/blob/master/lib/new_callbacks.rb
>
> > Long shot: is there a way to walk the `super` chain from outside the
> class?
> > Meaning, Resque::Job#perform could call the hidden methods instead of
> > requiring authors to call `super`.
>
> You could do this:
>
> https://gist.github.com/8d1bbe139a6d574540d5
>
> Unfortunately `Module.ancestors` doesn't keep track of `extend` uses
> like it does with `include`. Maybe there's another method I don't know
> about for that.
>
> --
> Chris Wanstrath
> http://github.com/defunkt
>

Re: [resque] job hooks

From:
Nicolás Sanguinetti
Date:
2010-03-28 @ 08:13
On Sun, Mar 28, 2010 at 4:18 AM, Ryan Carver <ryan@fivesevensix.com> wrote:
> How about this?
> https://github.com/rcarver/resque/compare/hooks...hook-plugin
> Pros:
>  - Plugin authors don't need to worry about super
>  - Both jobs and plugins can implement hooks and it behaves sensibly
> (currently job hooks run last)
> Cons:
>  - Some metaprogramming
>  - around_perform is kind of weird, but what's the alternative? It's the
> reason we started worrying about super in the first place.

For some reason gh redirects to my dashboard when I try to access
anything inside rcarver/resque, so I can't see that :-/

IMO, having to worry about super is not actually a problem, I see it
as a feature to be able to specify when to call "the other" plugins
(if any). Without seeing your code, I can't say nothing about it,
though.

> Chris: this is basically your plugin technique but uses `extend_object`
> instead of `extended`.
>
>
> On Tue, Mar 23, 2010 at 6:21 PM, Chris Wanstrath <chris@ozmm.org> wrote:
>>
>> On Tue, Mar 23, 2010 at 5:02 PM, Ryan Carver <ryan@fivesevensix.com>
>> wrote:
>>
>> > Hmm, you lost me. How does extend required inheriting from Resque::Jobs?
>> > Do
>> > you mean an AR-like callbacks? Then yeah it requires some kind of base
>> > class
>> > - which isn't terrible I guess.
>>
>> I completely confused myself. Ignore me.
>>
>> > I agree it gets weird. But, I *really* like that you can include
>> > multiple
>> > bits of functionality. The thing that drove this originally was
>> > implementing
>> > both Lock and Retry. I have cases where one, the other, or both are
>> > useful.
>> > These two happen to not conflict, but yeah it's possible. Also, I'm
>> > curious
>> > how often `around_perform` is needed.
>>
>> Confusion aside, I agree that composable plugins are awesome.
>>
>> I wonder if it's worth checking out something like this:
>>
>> http://github.com/wycats/simple-callbacks-refactor
>>
>> Specifically:
>>
>>
>> 
http://github.com/wycats/simple-callbacks-refactor/blob/master/lib/new_callbacks.rb
>>
>> > Long shot: is there a way to walk the `super` chain from outside the
>> > class?
>> > Meaning, Resque::Job#perform could call the hidden methods instead of
>> > requiring authors to call `super`.
>>
>> You could do this:
>>
>> https://gist.github.com/8d1bbe139a6d574540d5
>>
>> Unfortunately `Module.ancestors` doesn't keep track of `extend` uses
>> like it does with `include`. Maybe there's another method I don't know
>> about for that.
>>
>> --
>> Chris Wanstrath
>> http://github.com/defunkt
>
>

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-03-28 @ 16:01
My bad, it was private for some reason.


2010/3/28 Nicolás Sanguinetti <hi@nicolassanguinetti.info>

> On Sun, Mar 28, 2010 at 4:18 AM, Ryan Carver <ryan@fivesevensix.com>
> wrote:
> > How about this?
> > https://github.com/rcarver/resque/compare/hooks...hook-plugin
> > Pros:
> >  - Plugin authors don't need to worry about super
> >  - Both jobs and plugins can implement hooks and it behaves sensibly
> > (currently job hooks run last)
> > Cons:
> >  - Some metaprogramming
> >  - around_perform is kind of weird, but what's the alternative? It's the
> > reason we started worrying about super in the first place.
>
> For some reason gh redirects to my dashboard when I try to access
> anything inside rcarver/resque, so I can't see that :-/
>
> IMO, having to worry about super is not actually a problem, I see it
> as a feature to be able to specify when to call "the other" plugins
> (if any). Without seeing your code, I can't say nothing about it,
> though.
>
> > Chris: this is basically your plugin technique but uses `extend_object`
> > instead of `extended`.
> >
> >
> > On Tue, Mar 23, 2010 at 6:21 PM, Chris Wanstrath <chris@ozmm.org> wrote:
> >>
> >> On Tue, Mar 23, 2010 at 5:02 PM, Ryan Carver <ryan@fivesevensix.com>
> >> wrote:
> >>
> >> > Hmm, you lost me. How does extend required inheriting from
> Resque::Jobs?
> >> > Do
> >> > you mean an AR-like callbacks? Then yeah it requires some kind of base
> >> > class
> >> > - which isn't terrible I guess.
> >>
> >> I completely confused myself. Ignore me.
> >>
> >> > I agree it gets weird. But, I *really* like that you can include
> >> > multiple
> >> > bits of functionality. The thing that drove this originally was
> >> > implementing
> >> > both Lock and Retry. I have cases where one, the other, or both are
> >> > useful.
> >> > These two happen to not conflict, but yeah it's possible. Also, I'm
> >> > curious
> >> > how often `around_perform` is needed.
> >>
> >> Confusion aside, I agree that composable plugins are awesome.
> >>
> >> I wonder if it's worth checking out something like this:
> >>
> >> http://github.com/wycats/simple-callbacks-refactor
> >>
> >> Specifically:
> >>
> >>
> >>
> 
http://github.com/wycats/simple-callbacks-refactor/blob/master/lib/new_callbacks.rb
> >>
> >> > Long shot: is there a way to walk the `super` chain from outside the
> >> > class?
> >> > Meaning, Resque::Job#perform could call the hidden methods instead of
> >> > requiring authors to call `super`.
> >>
> >> You could do this:
> >>
> >> https://gist.github.com/8d1bbe139a6d574540d5
> >>
> >> Unfortunately `Module.ancestors` doesn't keep track of `extend` uses
> >> like it does with `include`. Maybe there's another method I don't know
> >> about for that.
> >>
> >> --
> >> Chris Wanstrath
> >> http://github.com/defunkt
> >
> >
>

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-03-30 @ 06:42
On Sun, Mar 28, 2010 at 12:18 AM, Ryan Carver <ryan@fivesevensix.com> wrote:

> http://github.com/rcarver/resque/compare/hooks...hook-plugin
>
> Pros:
>  - Plugin authors don't need to worry about super
>  - Both jobs and plugins can implement hooks and it behaves sensibly
> (currently job hooks run last)
>
> Cons:
>  - Some metaprogramming
>  - around_perform is kind of weird, but what's the alternative? It's the
> reason we started worrying about super in the first place.

Making people call `super` in their plugins has never sat right with
me because it's purely an implementation detail that our API is
exposing. Forgetting to include it can be quite dangerous, too - the
plugin chain will cease to exist and that could be potentially hard to
debug.

That said, I've been thinking about this and may have found an API I like:

https://gist.github.com/658dc0c1a6f0148b40bc

Pros:
- No super, plugins are still composable
- No metaprogramming - any Rubyist should be able to understand the
implementation
- Both jobs and plugins can implement hooks and it behaves sensibly
- Visibility into which hooks your job will run:
  pp Job.methods.grep(/_perform/)
- Every hook has a unique name so you can write a plugin which
selectively overwrites part of another plugin - perhaps enhancing its
hook or providing more functionality.

Cons:
- No around_perform

Maybe not having `around_perform` is a good thing? I'm unsure how we
can add it in any of these APIs without creating lot more complexity.

Here's a working around_perform example for this `before_peform_name`
style, though:
https://gist.github.com/ca76a33b9fb737920302

-- 
Chris Wanstrath
http://github.com/defunkt

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-03-30 @ 19:15
On Mon, Mar 29, 2010 at 11:42 PM, Chris Wanstrath <chris@ozmm.org> wrote:

> That said, I've been thinking about this and may have found an API I like:
>
> https://gist.github.com/658dc0c1a6f0148b40bc

Here's my modification of Ryan's "hook-plugin" branch to use this new style:

http://github.com/defunkt/resque/commit/e2d44974f55d3fe598a91749c322f85d4adc6530

The tests don't pass, but they never did for me in that branch :(

If everyone is agreed then I'll spend the time to make them pass and
finally merge this in.

-- 
Chris Wanstrath
http://github.com/defunkt

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-03-30 @ 21:10
>
> Here's my modification of Ryan's "hook-plugin" branch to use this new
> style:
>
>
> http://github.com/defunkt/resque/commit/e2d44974f55d3fe598a91749c322f85d4adc6530
>
> The tests don't pass, but they never did for me in that branch :(
>
> If everyone is agreed then I'll spend the time to make them pass and
> finally merge this in.


This looks awesome, great idea. Strangely, tests pass for me on both yours
and my hook-plugin branch. I just merged in your master into mine as well.
What failures are you getting?

Be sure to delete resque/plugin.rb too..

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-04-01 @ 00:51
On Tue, Mar 30, 2010 at 2:10 PM, Ryan Carver <ryan@fivesevensix.com> wrote:

> This looks awesome, great idea. Strangely, tests pass for me on both yours
> and my hook-plugin branch. I just merged in your master into mine as well.
>
> What failures are you getting?
> Be sure to delete resque/plugin.rb too..

Do we need plugin_test.rb? Removing that gets rid of most of my failures.

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-04-01 @ 01:03
Boom, merged into master. This'll make it into the 1.7 release -
thanks everyone!

On Wed, Mar 31, 2010 at 5:51 PM, Chris Wanstrath <chris@ozmm.org> wrote:
> On Tue, Mar 30, 2010 at 2:10 PM, Ryan Carver <ryan@fivesevensix.com> wrote:
>
>> This looks awesome, great idea. Strangely, tests pass for me on both yours
>> and my hook-plugin branch. I just merged in your master into mine as well.
>>
>> What failures are you getting?
>> Be sure to delete resque/plugin.rb too..
>
> Do we need plugin_test.rb? Removing that gets rid of most of my failures.
>



-- 
Chris Wanstrath
http://github.com/defunkt

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-04-01 @ 07:55
Awesome. Nice work on the super simple API.

I think there were a few nice tests in plugin_test dealing with multiple
hook'd methods - especially for the around case. I'll see about merging them
back in.

As those two test cases evolved, they more or less performed these duties:
- job_hook_test: how does each hook behave in isolation, and in what order
do they occur
- plugin_test: what happens when multiple *_perform_* methods are defined

Yeah, that could have been clearer...  ;)


On Wed, Mar 31, 2010 at 6:03 PM, Chris Wanstrath <chris@ozmm.org> wrote:

> Boom, merged into master. This'll make it into the 1.7 release -
> thanks everyone!
>
> On Wed, Mar 31, 2010 at 5:51 PM, Chris Wanstrath <chris@ozmm.org> wrote:
> > On Tue, Mar 30, 2010 at 2:10 PM, Ryan Carver <ryan@fivesevensix.com>
> wrote:
> >
> >> This looks awesome, great idea. Strangely, tests pass for me on both
> yours
> >> and my hook-plugin branch. I just merged in your master into mine as
> well.
> >>
> >> What failures are you getting?
> >> Be sure to delete resque/plugin.rb too..
> >
> > Do we need plugin_test.rb? Removing that gets rid of most of my failures.
> >
>
>
>
> --
> Chris Wanstrath
> http://github.com/defunkt
>

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-04-01 @ 17:08
Ok, I got this all sorted out.

First off, the tests were failing because they don't run when I do `rake`.
Not really sure what's up here, but for some reason only some of the
_test.rb's run. To sanity check, I used this to run the test suite **

find test/ -name "*_test.rb" | xargs -n1 ruby

More importantly, I restored the previous plugins_test.rb which dealt with
hook execution order. The failures in that suite were caused by an
undetermined order of hooks when doing
`job.methods.grep(/^before_perform/)`. That change is here:

http://github.com/rcarver/resque/compare/plugintests

which sorts the hook methods alphabetically.

Cool?


** Unrelated, I get this in worker_test.rb
  1) Failure:
test_sets__0_while_working(Resque__Worker)
    [test//worker_test.rb:212:in `test_sets__0_while_working'

/Users/rcarver/Work/Opensource/resque/test/../lib/resque/worker.rb:157:in
`process'

/Users/rcarver/Work/Opensource/resque/test/../lib/resque/worker.rb:123:in
`work'

/Users/rcarver/Work/Opensource/resque/test/../lib/resque/worker.rb:110:in
`loop'

/Users/rcarver/Work/Opensource/resque/test/../lib/resque/worker.rb:110:in
`work'
     test//worker_test.rb:210:in `test_sets__0_while_working']:
<"resque-1.7.0: Processing jobs since 1270141656"> expected but was
<"resque-1.7.0: Processing">.



On Thu, Apr 1, 2010 at 12:55 AM, Ryan Carver <ryan@fivesevensix.com> wrote:

> Awesome. Nice work on the super simple API.
>
> I think there were a few nice tests in plugin_test dealing with multiple
> hook'd methods - especially for the around case. I'll see about merging them
> back in.
>
> As those two test cases evolved, they more or less performed these duties:
> - job_hook_test: how does each hook behave in isolation, and in what order
> do they occur
> - plugin_test: what happens when multiple *_perform_* methods are defined
>
> Yeah, that could have been clearer...  ;)
>
>
> On Wed, Mar 31, 2010 at 6:03 PM, Chris Wanstrath <chris@ozmm.org> wrote:
>
>> Boom, merged into master. This'll make it into the 1.7 release -
>> thanks everyone!
>>
>> On Wed, Mar 31, 2010 at 5:51 PM, Chris Wanstrath <chris@ozmm.org> wrote:
>> > On Tue, Mar 30, 2010 at 2:10 PM, Ryan Carver <ryan@fivesevensix.com>
>> wrote:
>> >
>> >> This looks awesome, great idea. Strangely, tests pass for me on both
>> yours
>> >> and my hook-plugin branch. I just merged in your master into mine as
>> well.
>> >>
>> >> What failures are you getting?
>> >> Be sure to delete resque/plugin.rb too..
>> >
>> > Do we need plugin_test.rb? Removing that gets rid of most of my
>> failures.
>> >
>>
>>
>>
>> --
>> Chris Wanstrath
>> http://github.com/defunkt
>>
>
>

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-04-01 @ 18:13
On Thu, Apr 1, 2010 at 10:08 AM, Ryan Carver <ryan@fivesevensix.com> wrote:

> First off, the tests were failing because they don't run when I do `rake`.
> Not really sure what's up here, but for some reason only some of the
> _test.rb's run. To sanity check, I used this to run the test suite **
>
> find test/ -name "*_test.rb" | xargs -n1 ruby

Whoa, sorry about that. I don't remember why I had the funky test
loading but I think it may be because of the hack that starts and
stops Redis (which I've been meaning to pull out into its own
library).

`rake` should work now, it does something similar to your snippet.

> More importantly, I restored the previous plugins_test.rb which dealt with
> hook execution order. The failures in that suite were caused by an
> undetermined order of hooks when doing
> `job.methods.grep(/^before_perform/)`. That change is here:
> http://github.com/rcarver/resque/compare/plugintests
> which sorts the hook methods alphabetically.
> Cool?

Awesome, thanks!

> ** Unrelated, I get this in worker_test.rb

Hopefully this is fixed for you now - I had the error but the new
Rakefile made it go away.

Chris

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-04-01 @ 18:18
Rad. All is well.


On Thu, Apr 1, 2010 at 11:13 AM, Chris Wanstrath <chris@ozmm.org> wrote:

> On Thu, Apr 1, 2010 at 10:08 AM, Ryan Carver <ryan@fivesevensix.com>
> wrote:
>
> > First off, the tests were failing because they don't run when I do
> `rake`.
> > Not really sure what's up here, but for some reason only some of the
> > _test.rb's run. To sanity check, I used this to run the test suite **
> >
> > find test/ -name "*_test.rb" | xargs -n1 ruby
>
> Whoa, sorry about that. I don't remember why I had the funky test
> loading but I think it may be because of the hack that starts and
> stops Redis (which I've been meaning to pull out into its own
> library).
>
> `rake` should work now, it does something similar to your snippet.
>
> > More importantly, I restored the previous plugins_test.rb which dealt
> with
> > hook execution order. The failures in that suite were caused by an
> > undetermined order of hooks when doing
> > `job.methods.grep(/^before_perform/)`. That change is here:
> > http://github.com/rcarver/resque/compare/plugintests
> > which sorts the hook methods alphabetically.
> > Cool?
>
> Awesome, thanks!
>
> > ** Unrelated, I get this in worker_test.rb
>
> Hopefully this is fixed for you now - I had the error but the new
> Rakefile made it go away.
>
> Chris
>

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-04-01 @ 19:08
Resque::Plugin.lint(MyPlugin)

to make sure your plugin conforms to good plugin etiquette. Currently that
just means adding a unique name to the hook (`before_perform_lock` vs
`before_perform`)

http://github.com/rcarver/resque/compare/plugin

Useful? Just wanted this while updating my resque-lock-retry plugin to use
the new system.





On Thu, Apr 1, 2010 at 11:18 AM, Ryan Carver <ryan@fivesevensix.com> wrote:

> Rad. All is well.
>
>
> On Thu, Apr 1, 2010 at 11:13 AM, Chris Wanstrath <chris@ozmm.org> wrote:
>
>> On Thu, Apr 1, 2010 at 10:08 AM, Ryan Carver <ryan@fivesevensix.com>
>> wrote:
>>
>> > First off, the tests were failing because they don't run when I do
>> `rake`.
>> > Not really sure what's up here, but for some reason only some of the
>> > _test.rb's run. To sanity check, I used this to run the test suite **
>> >
>> > find test/ -name "*_test.rb" | xargs -n1 ruby
>>
>> Whoa, sorry about that. I don't remember why I had the funky test
>> loading but I think it may be because of the hack that starts and
>> stops Redis (which I've been meaning to pull out into its own
>> library).
>>
>> `rake` should work now, it does something similar to your snippet.
>>
>> > More importantly, I restored the previous plugins_test.rb which dealt
>> with
>> > hook execution order. The failures in that suite were caused by an
>> > undetermined order of hooks when doing
>> > `job.methods.grep(/^before_perform/)`. That change is here:
>> > http://github.com/rcarver/resque/compare/plugintests
>> > which sorts the hook methods alphabetically.
>> > Cool?
>>
>> Awesome, thanks!
>>
>> > ** Unrelated, I get this in worker_test.rb
>>
>> Hopefully this is fixed for you now - I had the error but the new
>> Rakefile made it go away.
>>
>> Chris
>>
>
>

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-04-02 @ 22:04
Sweet:


http://github.com/defunkt/resque-lock/commit/e541b65a84f10de8e8e68f48df1e33457b0f66a0

On Thu, Apr 1, 2010 at 12:08 PM, Ryan Carver <ryan@fivesevensix.com> wrote:
> Resque::Plugin.lint(MyPlugin)
> to make sure your plugin conforms to good plugin etiquette. Currently that
> just means adding a unique name to the hook (`before_perform_lock` vs
> `before_perform`)
> http://github.com/rcarver/resque/compare/plugin
> Useful? Just wanted this while updating my resque-lock-retry plugin to use
> the new system.
>
>
>
>
> On Thu, Apr 1, 2010 at 11:18 AM, Ryan Carver <ryan@fivesevensix.com> wrote:
>>
>> Rad. All is well.
>>
>> On Thu, Apr 1, 2010 at 11:13 AM, Chris Wanstrath <chris@ozmm.org> wrote:
>>>
>>> On Thu, Apr 1, 2010 at 10:08 AM, Ryan Carver <ryan@fivesevensix.com>
>>> wrote:
>>>
>>> > First off, the tests were failing because they don't run when I do
>>> > `rake`.
>>> > Not really sure what's up here, but for some reason only some of the
>>> > _test.rb's run. To sanity check, I used this to run the test suite **
>>> >
>>> > find test/ -name "*_test.rb" | xargs -n1 ruby
>>>
>>> Whoa, sorry about that. I don't remember why I had the funky test
>>> loading but I think it may be because of the hack that starts and
>>> stops Redis (which I've been meaning to pull out into its own
>>> library).
>>>
>>> `rake` should work now, it does something similar to your snippet.
>>>
>>> > More importantly, I restored the previous plugins_test.rb which dealt
>>> > with
>>> > hook execution order. The failures in that suite were caused by an
>>> > undetermined order of hooks when doing
>>> > `job.methods.grep(/^before_perform/)`. That change is here:
>>> > http://github.com/rcarver/resque/compare/plugintests
>>> > which sorts the hook methods alphabetically.
>>> > Cool?
>>>
>>> Awesome, thanks!
>>>
>>> > ** Unrelated, I get this in worker_test.rb
>>>
>>> Hopefully this is fixed for you now - I had the error but the new
>>> Rakefile made it go away.
>>>
>>> Chris
>>
>
>



-- 
Chris Wanstrath
http://github.com/defunkt

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-04-02 @ 22:26
rad, thanks!


On Fri, Apr 2, 2010 at 3:04 PM, Chris Wanstrath <chris@ozmm.org> wrote:

> Sweet:
>
>
> 
http://github.com/defunkt/resque-lock/commit/e541b65a84f10de8e8e68f48df1e33457b0f66a0
>
> On Thu, Apr 1, 2010 at 12:08 PM, Ryan Carver <ryan@fivesevensix.com>
> wrote:
> > Resque::Plugin.lint(MyPlugin)
> > to make sure your plugin conforms to good plugin etiquette. Currently
> that
> > just means adding a unique name to the hook (`before_perform_lock` vs
> > `before_perform`)
> > http://github.com/rcarver/resque/compare/plugin
> > Useful? Just wanted this while updating my resque-lock-retry plugin to
> use
> > the new system.
> >
> >
> >
> >
> > On Thu, Apr 1, 2010 at 11:18 AM, Ryan Carver <ryan@fivesevensix.com>
> wrote:
> >>
> >> Rad. All is well.
> >>
> >> On Thu, Apr 1, 2010 at 11:13 AM, Chris Wanstrath <chris@ozmm.org>
> wrote:
> >>>
> >>> On Thu, Apr 1, 2010 at 10:08 AM, Ryan Carver <ryan@fivesevensix.com>
> >>> wrote:
> >>>
> >>> > First off, the tests were failing because they don't run when I do
> >>> > `rake`.
> >>> > Not really sure what's up here, but for some reason only some of the
> >>> > _test.rb's run. To sanity check, I used this to run the test suite **
> >>> >
> >>> > find test/ -name "*_test.rb" | xargs -n1 ruby
> >>>
> >>> Whoa, sorry about that. I don't remember why I had the funky test
> >>> loading but I think it may be because of the hack that starts and
> >>> stops Redis (which I've been meaning to pull out into its own
> >>> library).
> >>>
> >>> `rake` should work now, it does something similar to your snippet.
> >>>
> >>> > More importantly, I restored the previous plugins_test.rb which dealt
> >>> > with
> >>> > hook execution order. The failures in that suite were caused by an
> >>> > undetermined order of hooks when doing
> >>> > `job.methods.grep(/^before_perform/)`. That change is here:
> >>> > http://github.com/rcarver/resque/compare/plugintests
> >>> > which sorts the hook methods alphabetically.
> >>> > Cool?
> >>>
> >>> Awesome, thanks!
> >>>
> >>> > ** Unrelated, I get this in worker_test.rb
> >>>
> >>> Hopefully this is fixed for you now - I had the error but the new
> >>> Rakefile made it go away.
> >>>
> >>> Chris
> >>
> >
> >
>
>
>
> --
> Chris Wanstrath
> http://github.com/defunkt
>

Re: [resque] job hooks

From:
Nicolás Sanguinetti
Date:
2010-03-30 @ 10:08
On Tue, Mar 30, 2010 at 3:42 AM, Chris Wanstrath <chris@ozmm.org> wrote:
> On Sun, Mar 28, 2010 at 12:18 AM, Ryan Carver <ryan@fivesevensix.com> wrote:
>
>> http://github.com/rcarver/resque/compare/hooks...hook-plugin
>>
>> Pros:
>>  - Plugin authors don't need to worry about super
>>  - Both jobs and plugins can implement hooks and it behaves sensibly
>> (currently job hooks run last)
>>
>> Cons:
>>  - Some metaprogramming
>>  - around_perform is kind of weird, but what's the alternative? It's the
>> reason we started worrying about super in the first place.
>
> Making people call `super` in their plugins has never sat right with
> me because it's purely an implementation detail that our API is
> exposing. Forgetting to include it can be quite dangerous, too - the
> plugin chain will cease to exist and that could be potentially hard to
> debug.
>
> That said, I've been thinking about this and may have found an API I like:
>
> https://gist.github.com/658dc0c1a6f0148b40bc
>
> Pros:
> - No super, plugins are still composable
> - No metaprogramming - any Rubyist should be able to understand the
> implementation
> - Both jobs and plugins can implement hooks and it behaves sensibly
> - Visibility into which hooks your job will run:
>  pp Job.methods.grep(/_perform/)
> - Every hook has a unique name so you can write a plugin which
> selectively overwrites part of another plugin - perhaps enhancing its
> hook or providing more functionality.

Yep, I like it, very easy :D

> Cons:
> - No around_perform
>
> Maybe not having `around_perform` is a good thing? I'm unsure how we
> can add it in any of these APIs without creating lot more complexity.
>
> Here's a working around_perform example for this `before_peform_name`
> style, though:
> https://gist.github.com/ca76a33b9fb737920302
>
> --
> Chris Wanstrath
> http://github.com/defunkt
>

Re: [resque] job hooks

From:
Nicolás Sanguinetti
Date:
2010-03-25 @ 01:46
On Tue, Mar 23, 2010 at 10:21 PM, Chris Wanstrath <chris@ozmm.org> wrote:
> On Tue, Mar 23, 2010 at 5:02 PM, Ryan Carver <ryan@fivesevensix.com> wrote:
>
>> Hmm, you lost me. How does extend required inheriting from Resque::Jobs? Do
>> you mean an AR-like callbacks? Then yeah it requires some kind of base class
>> - which isn't terrible I guess.
>
> I completely confused myself. Ignore me.
>
>> I agree it gets weird. But, I *really* like that you can include multiple
>> bits of functionality. The thing that drove this originally was implementing
>> both Lock and Retry. I have cases where one, the other, or both are useful.
>> These two happen to not conflict, but yeah it's possible. Also, I'm curious
>> how often `around_perform` is needed.
>
> Confusion aside, I agree that composable plugins are awesome.
>
> I wonder if it's worth checking out something like this:
>
> http://github.com/wycats/simple-callbacks-refactor
>
> Specifically:
>
> 
http://github.com/wycats/simple-callbacks-refactor/blob/master/lib/new_callbacks.rb
>
>> Long shot: is there a way to walk the `super` chain from outside the class?
>> Meaning, Resque::Job#perform could call the hidden methods instead of
>> requiring authors to call `super`.
>
> You could do this:
>
> https://gist.github.com/8d1bbe139a6d574540d5
>
> Unfortunately `Module.ancestors` doesn't keep track of `extend` uses
> like it does with `include`. Maybe there's another method I don't know
> about for that.
>
> --
> Chris Wanstrath
> http://github.com/defunkt
>

What's the problem with using plain-old regular ruby?

https://gist.github.com/9d37a90c954c040bde9f

To define a job, inherit from Resque::Job or whatever, and define a
#perform method. Plugins are modules that get included, define
callbacks or whatever, and call super. That makes it straightforward,
and let's me worry, and doesn't require me to understand some
mind-bending ruby metaprogramming. I can just read the code.

Keep It Simple, Smartguy.

In a related note, I can understand the appeal of saying "hey, any
object that responds to #perform can be a job", but then why not make
that #call, so we can just pass a Proc as a job? If you want me to
define a different structure than the default callable in ruby, I
don't see the problem with requiring me to inherit from a class given
class.

Just my ¢2

Re: [resque] job hooks

From:
Chris Wanstrath
Date:
2010-03-25 @ 17:45
2010/3/24 Nicolás Sanguinetti <hi@nicolassanguinetti.info>:

> What's the problem with using plain-old regular ruby?
>
> https://gist.github.com/9d37a90c954c040bde9f
>
> To define a job, inherit from Resque::Job or whatever, and define a
> #perform method. Plugins are modules that get included, define
> callbacks or whatever, and call super. That makes it straightforward,
> and let's me worry, and doesn't require me to understand some
> mind-bending ruby metaprogramming. I can just read the code.

This forces inheritance, which is a deal breaker for me.

> In a related note, I can understand the appeal of saying "hey, any
> object that responds to #perform can be a job", but then why not make
> that #call, so we can just pass a Proc as a job? If you want me to
> define a different structure than the default callable in ruby, I
> don't see the problem with requiring me to inherit from a class given
> class.

You can't serialize a proc, at least not with the standard
library. Special cases cause confusion - just ask anyone using Resque
who expected their payload to have symbol keys.

Chris

Re: [resque] job hooks

From:
Nicolás Sanguinetti
Date:
2010-03-25 @ 21:24
On Thu, Mar 25, 2010 at 2:45 PM, Chris Wanstrath <chris@ozmm.org> wrote:
> 2010/3/24 Nicolás Sanguinetti <hi@nicolassanguinetti.info>:
>
>> What's the problem with using plain-old regular ruby?
>>
>> https://gist.github.com/9d37a90c954c040bde9f
>>
>> To define a job, inherit from Resque::Job or whatever, and define a
>> #perform method. Plugins are modules that get included, define
>> callbacks or whatever, and call super. That makes it straightforward,
>> and let's me worry, and doesn't require me to understand some
>> mind-bending ruby metaprogramming. I can just read the code.
>
> This forces inheritance, which is a deal breaker for me.

I wonder why, but I suppose you have your reasons :)

In any case, look at

https://gist.github.com/9d37a90c954c040bde9f/6315d4a728646cce78178f2595a2eb8f9a380a44

It's simple, the only problem is that it adds Module#plugin, which
might conflict with other libraries who also thought about extending
the core classes. The fact that it defines a module on runtime isn't
very nice either (but I guess you will only be calling this in
"compile time" so it shouldn't be that big of a problem)

The API is a little weirder, but

https://gist.github.com/9d37a90c954c040bde9f/4075428f2dd321a924d94d339f6d99e87c583b76
at least doesn't fiddle with Module, so I guess I like it a bit more.

It's more or less the same idea behind what you were doing by defining
that class, but a bit easier on the eyes :P

Thoughts?

-foca

>
>> In a related note, I can understand the appeal of saying "hey, any
>> object that responds to #perform can be a job", but then why not make
>> that #call, so we can just pass a Proc as a job? If you want me to
>> define a different structure than the default callable in ruby, I
>> don't see the problem with requiring me to inherit from a class given
>> class.
>
> You can't serialize a proc, at least not with the standard
> library. Special cases cause confusion - just ask anyone using Resque
> who expected their payload to have symbol keys.

Fair enough :)

> Chris
>

Re: [resque] job hooks

From:
Ryan Carver
Date:
2010-03-25 @ 00:14
>
> You could do this:
>
> https://gist.github.com/8d1bbe139a6d574540d5



Nice, simple. I plugged this in and it seems to have one flaw, and it's
really just conceptual.

With the following example, I expect this result:

assert_equal [:before_one, :before_two, :perform], history

The problem is, what is the expected behavior if my job class itself defines
a before_perform hook? Options:

1. Run it first
2. Run it last
3. Don't run it (require hooks to be defined in plugins)

1) or 2) get messy because the last plugin will register itself in the
plugins list AND actually define before_perform on the class. 3) seems like
the cleanest option at this point, but I don't necessarily like forcing that
distinction - what do you think?

I have a few ideas to work around this, need to hack on it a bit...

  module BeforeOne
    extend Resque::Plugin
    def before_perform(history)
      history << :before_one
    end
  end

  module BeforeTwo
    extend Resque::Plugin
    def before_perform(history)
      history << :before_two
    end
  end

  class ManyBeforesJob
    extend BeforeOne
    extend BeforeTwo
    def self.perform(history)
      history << :perform
    end
  end




> Unfortunately `Module.ancestors` doesn't keep track of `extend` uses
> like it does with `include`. Maybe there's another method I don't know
> about for that.


Cool, something else to play with.