librelist archives

« back to archive

what to do in litmus when async tests time out

what to do in litmus when async tests time out

From:
Tom Yandell
Date:
2011-10-19 @ 21:15
Litmus currently has a pretty short (four second) default for async
operations in tests. When it times out and the assertions are subsequently
encountered (as happened to me today) I get a message like:

...
> [ PASS ] second created time same as initial
> [ PASS ] all second requests handled by webapp created at same time
> [ ERROR ] async operation "reload" timed out after 4 seconds
> Summary
> =======
> Result:
>     FAIL
>
>
> 
/Users/thomasyandell/node-projects/proton.js/node_modules/promised-io/promise.js:221
> throw error;
>            ^
> Error: assertion (test.is(..., "third set of responses received")) added
> to test run after it was finished
>     at [object Object]._checkRunning
> (/Users/thomasyandell/node-projects/litmus.js/lib/litmus.js:248:15)
> ...


It's tricky because node or the browser will continue to wait until there
are no outstanding events pending before exiting, unless you call
process.exit. My current thinking is that
TestRun.prototype.start/SuiteRun.prototype.start should return a promise
that is resolved when the timeout occurs and the command-line runner should
exit the process after it has printed. This doesn't help much for the
browser - I guess it should ignore assertion results that get added after
the run has completed following aync timeouts. I'll add a bug for that
unless you can think of a clever way round the problem.

Tom

Re: what to do in litmus when async tests time out

From:
Tom Yandell
Date:
2011-10-19 @ 21:24
Also, the interface for async timeouts sucks right now. Async tests look
like:

test.async('description', function (handle) {
    // ...
    handle.finish();
});

The timeout is an optional third parameter specified in seconds, which gets
lost as you put more code in the callback function :-( I'm also wondering
whether the handle should just be a promise that you resolve when you've
finished?

Tom

On 19 October 2011 22:15, Tom Yandell <tom@yandell.me.uk> wrote:

> Litmus currently has a pretty short (four second) default for async
> operations in tests. When it times out and the assertions are subsequently
> encountered (as happened to me today) I get a message like:
>
>  ...
>> [ PASS ] second created time same as initial
>> [ PASS ] all second requests handled by webapp created at same time
>> [ ERROR ] async operation "reload" timed out after 4 seconds
>> Summary
>> =======
>> Result:
>>     FAIL
>>
>>
>> 
/Users/thomasyandell/node-projects/proton.js/node_modules/promised-io/promise.js:221
>> throw error;
>>            ^
>> Error: assertion (test.is(..., "third set of responses received")) added
>> to test run after it was finished
>>     at [object Object]._checkRunning
>> (/Users/thomasyandell/node-projects/litmus.js/lib/litmus.js:248:15)
>> ...
>
>
> It's tricky because node or the browser will continue to wait until there
> are no outstanding events pending before exiting, unless you call
> process.exit. My current thinking is that
> TestRun.prototype.start/SuiteRun.prototype.start should return a promise
> that is resolved when the timeout occurs and the command-line runner should
> exit the process after it has printed. This doesn't help much for the
> browser - I guess it should ignore assertion results that get added after
> the run has completed following aync timeouts. I'll add a bug for that
> unless you can think of a clever way round the problem.
>
> Tom
>
>

Re: [usenode] Re: what to do in litmus when async tests time out

From:
Richard Hodgson
Date:
2011-10-22 @ 14:47
Something that I find annoying in other js test lib's is asynchronous 
tests not timing out at all. 

Could there be a default timeout that can be overridden per test 
(litmus.Test that is)?

It would still be nice to pass a timeout to async()... dare I suggest 
jQuery style variable argument positions?

The following are both supported:

    this.async('message', function () {...});

    this.async('message', 4, function () {...});

Not especially keen on it, however can't think of anything better.

Also, should timeouts be in seconds or milleseconds?

Richard

On 19 Oct 2011, at 22:24, Tom Yandell <tom@yandell.me.uk> wrote:

> Also, the interface for async timeouts sucks right now. Async tests look like:
> 
> test.async('description', function (handle) {
>     // ...
>     handle.finish();
> });
> 
> The timeout is an optional third parameter specified in seconds, which 
gets lost as you put more code in the callback function :-( I'm also 
wondering whether the handle should just be a promise that you resolve 
when you've finished?
> 
> Tom
> 
> On 19 October 2011 22:15, Tom Yandell <tom@yandell.me.uk> wrote:
> Litmus currently has a pretty short (four second) default for async 
operations in tests. When it times out and the assertions are subsequently
encountered (as happened to me today) I get a message like:
> ...
> [ PASS ] second created time same as initial
> [ PASS ] all second requests handled by webapp created at same time
> [ ERROR ] async operation "reload" timed out after 4 seconds
> Summary
> =======
> Result:
>     FAIL
> 
> 
/Users/thomasyandell/node-projects/proton.js/node_modules/promised-io/promise.js:221
> 					throw error;
>            ^
> Error: assertion (test.is(..., "third set of responses received")) added
to test run after it was finished
>     at [object Object]._checkRunning 
(/Users/thomasyandell/node-projects/litmus.js/lib/litmus.js:248:15)
> ...
> 
> It's tricky because node or the browser will continue to wait until 
there are no outstanding events pending before exiting, unless you call 
process.exit. My current thinking is that 
TestRun.prototype.start/SuiteRun.prototype.start should return a promise 
that is resolved when the timeout occurs and the command-line runner 
should exit the process after it has printed. This doesn't help much for 
the browser - I guess it should ignore assertion results that get added 
after the run has completed following aync timeouts. I'll add a bug for 
that unless you can think of a clever way round the problem.
> 
> Tom
> 
> 

Re: [usenode] Re: what to do in litmus when async tests time out

From:
Tom Yandell
Date:
2011-10-22 @ 21:47
Definitely milliseconds - don't know what I was thinking. Not sure about the
api bit, but I don't like where it is at the moment (after the function).
Maybe we should establish a convention? Perhaps: positional parameters for
stuff that's always passed, followed by options object (where applicable)
and then callback function (again, where applicable). This would then be:

    this.async('message', { timeout: 10000 }, function (done) {
        // ...
        done.resolve();
    });

Thoughts?

Tom

On 22 October 2011 15:47, Richard Hodgson <rightaboutnow@gmail.com> wrote:

> Something that I find annoying in other js test lib's is asynchronous tests
> not timing out at all.
>
> Could there be a default timeout that can be overridden per test
> (litmus.Test that is)?
>
> It would still be nice to pass a timeout to async()... dare I suggest
> jQuery style variable argument positions?
>
> The following are both supported:
>
>     this.async('message', function () {...});
>
>     this.async('message', 4, function () {...});
>
> Not especially keen on it, however can't think of anything better.
>
> Also, should timeouts be in seconds or milleseconds?
>
> Richard
>
> On 19 Oct 2011, at 22:24, Tom Yandell <tom@yandell.me.uk> wrote:
>
> **
> Also, the interface for async timeouts sucks right now. Async tests look
> like:
>
> test.async('description', function (handle) {
>     // ...
>     handle.finish();
> });
>
> The timeout is an optional third parameter specified in seconds, which gets
> lost as you put more code in the callback function :-( I'm also wondering
> whether the handle should just be a promise that you resolve when you've
> finished?
>
> Tom
>
> On 19 October 2011 22:15, Tom Yandell <tom@yandell.me.uk> wrote:
>
>> Litmus currently has a pretty short (four second) default for async
>> operations in tests. When it times out and the assertions are subsequently
>> encountered (as happened to me today) I get a message like:
>>
>>  ...
>>> [ PASS ] second created time same as initial
>>> [ PASS ] all second requests handled by webapp created at same time
>>> [ ERROR ] async operation "reload" timed out after 4 seconds
>>> Summary
>>> =======
>>> Result:
>>>     FAIL
>>>
>>>
>>> 
/Users/thomasyandell/node-projects/proton.js/node_modules/promised-io/promise.js:221
>>> throw error;
>>>            ^
>>> Error: assertion (test.is(..., "third set of responses received")) added
>>> to test run after it was finished
>>>     at [object Object]._checkRunning
>>> (/Users/thomasyandell/node-projects/litmus.js/lib/litmus.js:248:15)
>>> ...
>>
>>
>> It's tricky because node or the browser will continue to wait until there
>> are no outstanding events pending before exiting, unless you call
>> process.exit. My current thinking is that
>> TestRun.prototype.start/SuiteRun.prototype.start should return a promise
>> that is resolved when the timeout occurs and the command-line runner should
>> exit the process after it has printed. This doesn't help much for the
>> browser - I guess it should ignore assertion results that get added after
>> the run has completed following aync timeouts. I'll add a bug for that
>> unless you can think of a clever way round the problem.
>>
>> Tom
>>
>>
>

Re: [usenode] Re: what to do in litmus when async tests time out

From:
Tom Yandell
Date:
2011-10-24 @ 22:27
This is done in the aysnc-changes branch:

https://github.com/usenode/litmus.js/compare/master...async-changes

Thoughts?

Tom

On 22 October 2011 22:47, Tom Yandell <tom@yandell.me.uk> wrote:

> Definitely milliseconds - don't know what I was thinking. Not sure about
> the api bit, but I don't like where it is at the moment (after the
> function). Maybe we should establish a convention? Perhaps: positional
> parameters for stuff that's always passed, followed by options object (where
> applicable) and then callback function (again, where applicable). This would
> then be:
>
>     this.async('message', { timeout: 10000 }, function (done) {
>         // ...
>         done.resolve();
>     });
>
> Thoughts?
>
> Tom
>
>
> On 22 October 2011 15:47, Richard Hodgson <rightaboutnow@gmail.com> wrote:
>
>> Something that I find annoying in other js test lib's is asynchronous
>> tests not timing out at all.
>>
>> Could there be a default timeout that can be overridden per test
>> (litmus.Test that is)?
>>
>> It would still be nice to pass a timeout to async()... dare I suggest
>> jQuery style variable argument positions?
>>
>> The following are both supported:
>>
>>     this.async('message', function () {...});
>>
>>     this.async('message', 4, function () {...});
>>
>> Not especially keen on it, however can't think of anything better.
>>
>> Also, should timeouts be in seconds or milleseconds?
>>
>> Richard
>>
>> On 19 Oct 2011, at 22:24, Tom Yandell <tom@yandell.me.uk> wrote:
>>
>> **
>> Also, the interface for async timeouts sucks right now. Async tests look
>> like:
>>
>> test.async('description', function (handle) {
>>     // ...
>>     handle.finish();
>> });
>>
>> The timeout is an optional third parameter specified in seconds, which
>> gets lost as you put more code in the callback function :-( I'm also
>> wondering whether the handle should just be a promise that you resolve when
>> you've finished?
>>
>> Tom
>>
>> On 19 October 2011 22:15, Tom Yandell <tom@yandell.me.uk> wrote:
>>
>>> Litmus currently has a pretty short (four second) default for async
>>> operations in tests. When it times out and the assertions are subsequently
>>> encountered (as happened to me today) I get a message like:
>>>
>>>  ...
>>>> [ PASS ] second created time same as initial
>>>> [ PASS ] all second requests handled by webapp created at same time
>>>> [ ERROR ] async operation "reload" timed out after 4 seconds
>>>> Summary
>>>> =======
>>>> Result:
>>>>     FAIL
>>>>
>>>>
>>>> 
/Users/thomasyandell/node-projects/proton.js/node_modules/promised-io/promise.js:221
>>>> throw error;
>>>>            ^
>>>> Error: assertion (test.is(..., "third set of responses received"))
>>>> added to test run after it was finished
>>>>     at [object Object]._checkRunning
>>>> (/Users/thomasyandell/node-projects/litmus.js/lib/litmus.js:248:15)
>>>> ...
>>>
>>>
>>> It's tricky because node or the browser will continue to wait until there
>>> are no outstanding events pending before exiting, unless you call
>>> process.exit. My current thinking is that
>>> TestRun.prototype.start/SuiteRun.prototype.start should return a promise
>>> that is resolved when the timeout occurs and the command-line runner should
>>> exit the process after it has printed. This doesn't help much for the
>>> browser - I guess it should ignore assertion results that get added after
>>> the run has completed following aync timeouts. I'll add a bug for that
>>> unless you can think of a clever way round the problem.
>>>
>>> Tom
>>>
>>>
>>
>

Re: [usenode] Re: what to do in litmus when async tests time out

From:
Richard Hodgson
Date:
2011-10-26 @ 08:18
Looks good to me, I've warmed to the new optional params Api. 

Think we should merge some of these branches back master :-)

Richard

On 24 Oct 2011, at 23:27, Tom Yandell <tom@yandell.me.uk> wrote:

> This is done in the aysnc-changes branch:
> 
> https://github.com/usenode/litmus.js/compare/master...async-changes
> 
> Thoughts?
> 
> Tom
> 
> On 22 October 2011 22:47, Tom Yandell <tom@yandell.me.uk> wrote:
> Definitely milliseconds - don't know what I was thinking. Not sure about
the api bit, but I don't like where it is at the moment (after the 
function). Maybe we should establish a convention? Perhaps: positional 
parameters for stuff that's always passed, followed by options object 
(where applicable) and then callback function (again, where applicable). 
This would then be:
> 
>     this.async('message', { timeout: 10000 }, function (done) {
>         // ...
>         done.resolve();
>     });
> 
> Thoughts?
> 
> Tom
> 
> 
> On 22 October 2011 15:47, Richard Hodgson <rightaboutnow@gmail.com> wrote:
> Something that I find annoying in other js test lib's is asynchronous 
tests not timing out at all. 
> 
> Could there be a default timeout that can be overridden per test 
(litmus.Test that is)?
> 
> It would still be nice to pass a timeout to async()... dare I suggest 
jQuery style variable argument positions?
> 
> The following are both supported:
> 
>     this.async('message', function () {...});
> 
>     this.async('message', 4, function () {...});
> 
> Not especially keen on it, however can't think of anything better.
> 
> Also, should timeouts be in seconds or milleseconds?
> 
> Richard
> 
> On 19 Oct 2011, at 22:24, Tom Yandell <tom@yandell.me.uk> wrote:
> 
> Also, the interface for async timeouts sucks right now. Async tests look like:
> 
> test.async('description', function (handle) {
>     // ...
>     handle.finish();
> });
> 
> The timeout is an optional third parameter specified in seconds, which 
gets lost as you put more code in the callback function :-( I'm also 
wondering whether the handle should just be a promise that you resolve 
when you've finished?
> 
> Tom
> 
> On 19 October 2011 22:15, Tom Yandell <tom@yandell.me.uk> wrote:
> Litmus currently has a pretty short (four second) default for async 
operations in tests. When it times out and the assertions are subsequently
encountered (as happened to me today) I get a message like:
> ...
> [ PASS ] second created time same as initial
> [ PASS ] all second requests handled by webapp created at same time
> [ ERROR ] async operation "reload" timed out after 4 seconds
> Summary
> =======
> Result:
>     FAIL
> 
> 
/Users/thomasyandell/node-projects/proton.js/node_modules/promised-io/promise.js:221
> 					throw error;
>            ^
> Error: assertion (test.is(..., "third set of responses received")) added
to test run after it was finished
>     at [object Object]._checkRunning 
(/Users/thomasyandell/node-projects/litmus.js/lib/litmus.js:248:15)
> ...
> 
> It's tricky because node or the browser will continue to wait until 
there are no outstanding events pending before exiting, unless you call 
process.exit. My current thinking is that 
TestRun.prototype.start/SuiteRun.prototype.start should return a promise 
that is resolved when the timeout occurs and the command-line runner 
should exit the process after it has printed. This doesn't help much for 
the browser - I guess it should ignore assertion results that get added 
after the run has completed following aync timeouts. I'll add a bug for 
that unless you can think of a clever way round the problem.
> 
> Tom
> 
> 
> 
>