NUnit 2.5: What's In It Now

Hi Kelly,

Knowing your mail habits, I'm sending this back to the list. :-)

Let us know when the snapshot is ready. I'm CVS challenged
and short on time. :-(

OK.

Post by Charlie Poole
Both the Gui and the console now display more detailed
errors, failures, not runnable, ignored and skipped tests. While
not-runnable (invalid) tests are not counted as errors,

they display

Post by Charlie Poole
in red in the gui anyway so that they will get your attention.

If I were to implement an addin or something to do my
approach to Theory, which of these would you recommend for
the result of running a Theory and having Assume bail out of
the run? Do any of them make sense?

I see two possible choices, neither of which is currently
available:

1) Inconclusive, which has been requested as a feature.
It makes sense that "Inconclusive" could be OK in a
Theory situation - provided other data is supplied - but
not OK in a normal Test.

2) No result at all, with the Theory giving only one result
if it passes and reporting specific failures if it fails.
This is how I tend to think of it. It means that two
implementations, one of them filtering the data
externally and one internally will give the same results.

Post by Charlie Poole
Test methods may have arguments (if you provide them) and return
TestCase and DataSource.

Have you described DataSource anywhere besides this email? I
can guess how it works from what you say below. I have
noticed that the documentation page is lagging the current
release, is there some way I could help out with that? Just
let me know what you want documented, or I can point out what
I think is missing.

It's in the docs, which will be part of the snapshot. Finishing
them up is actually what is keeping me from releasing right now.

Post by Charlie Poole
TestCase provides inline arguments and may also specify

an expected

Post by Charlie Poole
exception, expected return value, test name and description.

Way cool.

Post by Charlie Poole
DataSource may take a string or a Type. A string is the name of a
static property returning an IEnumerable.
A Type must contain such a property - the first one found is used.

Interesting approach to DataSource. Still doesn't give a
compile time error. If the type had to implement the
IEnumerable interface instead of just having an IEnumerable
property, then could you have a compile time error reported?

I agree. Of course, we would have to actually instantiate
the object in that case. I'll see about adding this in
the next drop.

Post by Charlie Poole
In either case, the IEnumerable should return either an

object array

Post by Charlie Poole
- if only args are provided - or an object with properties

similar to

Post by Charlie Poole
those of the TestCase attribute. This allows the provider

to supply

Post by Charlie Poole
a return value, expected exception, etc.
TestCase and DataSource may be mixed on a method.

This sounds like a case for a little bit of documentation.
Give me a simple sample, and I can probably run from there.

See http://nunit.org/?p=testCase&r=2.5 and
http://nunit.org/?p=dataSource&r=2.5

Note that these pages have been updated, but others not yet.

Post by Charlie Poole
A new extension point "ParameterProviders" is available

for addins

Post by Charlie Poole
that provide test case data to paramterized methods.

I would kill to have a new extension point that would allow
me to put an intercept in that would allow me to do Assume...
It would be really simple, I think.

There is one: since Tests handle their own exceptions, you
just need to create a test type that handles this. You might
call it "Theory" :-)

Post by Charlie Poole
I think we are at a point where a preliminary release of

some kind

Post by Charlie Poole
is called for. After that...
WHAT REMAINS TO DO
I'm looking at Gary's Theory extension to see if it fits

into this

Post by Charlie Poole
timeframe.

I know that Assume is off your list for 2.5. But would it be
possible to implement a couple of minor extension points that
would allow for it in an Addin? It doesn't seem like that
would be too hard, or take away very much from what Theory
might develop into in the future.

Answering this separately.

Charlie

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Register now and save $200. Hurry, offer ends at 11:59 p.m.,
Monday, April 7! Use priority code J8TLD2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Kelly Anderson

2008-04-07 01:53:30 UTC

Post by Charlie Poole
Hi Kelly,
Knowing your mail habits, I'm sending this back to the list. :-)

Uh, good call.. :-)

Post by Charlie Poole
Both the Gui and the console now display more detailed
errors, failures, not runnable, ignored and skipped tests. While
not-runnable (invalid) tests are not counted as errors,

they display

Post by Charlie Poole
in red in the gui anyway so that they will get your attention.

I see two possible choices, neither of which is currently
1) Inconclusive, which has been requested as a feature.
It makes sense that "Inconclusive" could be OK in a
Theory situation - provided other data is supplied - but
not OK in a normal Test.

Ok. That's one possibility. Sounds good.

Post by Charlie Poole
2) No result at all, with the Theory giving only one result
if it passes and reporting specific failures if it fails.
This is how I tend to think of it. It means that two
implementations, one of them filtering the data
externally and one internally will give the same results.

There's also the case where none of the data passed in gets past the
Assumes. That's supposed to be a failure, but I suppose that is
already covered by fail.

Post by Charlie Poole
Test methods may have arguments (if you provide them) and return
TestCase and DataSource.

It's in the docs, which will be part of the snapshot. Finishing
them up is actually what is keeping me from releasing right now.

Post by Charlie Poole
TestCase provides inline arguments and may also specify

an expected

Post by Charlie Poole
exception, expected return value, test name and description.

Way cool.

Post by Charlie Poole
DataSource may take a string or a Type. A string is the name of a
static property returning an IEnumerable.
A Type must contain such a property - the first one found is used.

I agree. Of course, we would have to actually instantiate
the object in that case. I'll see about adding this in
the next drop.

Well, since you can't use the DataSource for more than one thing
anyway, this seems like a cleaner approach.

Post by Charlie Poole
This sounds like a case for a little bit of documentation.
Give me a simple sample, and I can probably run from there.

See http://nunit.org/?p=testCase&r=2.5 and
http://nunit.org/?p=dataSource&r=2.5
Note that these pages have been updated, but others not yet.

That helps a lot.

Post by Charlie Poole
A new extension point "ParameterProviders" is available

for addins

Post by Charlie Poole
that provide test case data to paramterized methods.

I would kill to have a new extension point that would allow
me to put an intercept in that would allow me to do Assume...
It would be really simple, I think.

There is one: since Tests handle their own exceptions, you
just need to create a test type that handles this. You might
call it "Theory" :-)

We'll have to discuss this further. :-)

Perhaps we can start a new thread for this?

Post by Charlie Poole
I'm looking at Gary's Theory extension to see if it fits

into this

Post by Charlie Poole
timeframe.

Answering this separately.

Ok. By the way, I have run into several instances recently where I
found Assume would have been useful and appropriate inside of regular
data driven tests.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Register now and save $200. Hurry, offer ends at 11:59 p.m.,
Monday, April 7! Use priority code J8TLD2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Kelly Anderson

2008-04-07 02:58:19 UTC

Post by Charlie Poole
Hi Kelly,

It seems as if there may be one or more misunderstandings
or what I'm saying - and maybe I'm also missing what you
are trying to say. Let me try to clarify this point by
point.
1) Assume is off my list as an independent feature, which
is how you proposed it. In fact, taken in that light,
it's off my list forever, not just for 2.5.

I'm going to assume that you looked at my implementation of Assume.
That is, it was exactly like Assert, except that it threw a different
exception. How that exception gets handled by the framework is an
independent question.

Would your implementation of Assume be different than that? Or, are
you simply saying that you don't want to release Assume without also
releasing Theory. Do they need to be tied together to be useful?

I guess I just don't understand if you didn't like the way I
implemented Assume, or if you just have difficulty with doing it
without some fuller approach to Theory first.

I'm not trying to be confrontational, I just really don't understand
the sense in which you are objecting to putting Assume in earlier
rather than later...

Consider this use case of a data test (which is real, I did it
Thursday)... I want to iterate over all the files in a directory, and
validate things about the files if they are the right type and
version. I could do this with your proposed 2.5 like this:

public static IEnumerable AFiles
{
get
{
DirectoryInfo di = new DirectoryInfo("c:/data");
List<FileInfo> list = new List<FileInfo>();
FileInfo[] files = di.GetFiles("*.*");
foreach(FileInfo fi in files)
{
if (TypeOfFileIsA(fi))
{
object [] parms = new object[1];
parms[0] = fi;
list.Add(parms);
}
}
return list;
}
}

[DataSource("AFiles")]
public void TestAFiles(object a)
{
FileInfo fi = (FileInfo)a;
// Assert something about the file of type A.
}

public static IEnumerable BFiles
{
get
{
DirectoryInfo di = new DirectoryInfo("c:/data");
List<FileInfo> list = new List<FileInfo>();
FileInfo[] files = di.GetFiles("*.*");
foreach(FileInfo fi in files)
{
if (TypeOfFileIsB(fi))
{
object [] parms = new object[1];
parms[0] = fi;
list.Add(parms);
}
}
return list;
}
}

[DataSource("BFiles")]
public void TestBFiles(object a)
{
FileInfo fi = (FileInfo)a;
// Assert something about the file of type B.
}

====================================================
Now, if you have Assume, then you can refactor the above code to this:

public static IEnumerable Files
{
get
{
DirectoryInfo di = new DirectoryInfo("c:/data");
List<FileInfo> list = new List<FileInfo>();
FileInfo[] files = di.GetFiles("*.*");
foreach(FileInfo fi in files)
{
object [] parms = new object[1];
parms[0] = fi;
list.Add(parms);
}
return list;
}
}

[DataSource("Files")]
public void TestAFiles(object a)
{
FileInfo fi = (FileInfo)a;
Assume.That(TypeOfFileIsA(fi));
// Assert something about the file of type A.
}

[DataSource("Files")]
public void TestBFiles(object a)
{
FileInfo fi = (FileInfo)a;
Assume.That(TypeOfFileIsB(fi));
// Assert something about the file of type B.
}

=============================================
Now, I don't know about you, but I like the second version better. And
this is very clearly data testing, not Theory.

Post by Charlie Poole
2) As I mentionned in the earlier note, it's not hard
for someone to implement Theory as an extension using
the existing extension points. Gary has done that, in fact.

I'm not really talking about Theory, I'm talking about Assume. It's
great what Gary has done, but without Assume, it's only half the
story.

Post by Charlie Poole
3) What you were proposing in earlier discussions came
across to me - perhaps wrongly - as a sort of Potemkin
Theory implementation. That's what I reacted to, not
to the notion of you or anyone creating a true Theory
implementation as an addin. Go to it if you like. It's
just not in /my/ workplan for 2.5.

Yes, that's what I understood.

Post by Charlie Poole
4) Earlier, in reacting to what I perceive as your
trying to push more and more into 2.5, at the expense
of getting started with 3.0, I reacted strongly.
Unfortunately, that reaction was taken by others as
discouraging their efforts. Nothing could be farther
from my intent.

I understand the tension between putting more stuff in 2.5 and getting
on with 3.0, I do.

Post by Charlie Poole
I don't want to discourage you - not even, :-) - from
doing stuff either. In fact I encourage it. Have you
tried out Gary's Theory extension? Does it work for you?
Does it need something? Can you contribute whatever
that is? ... See, I'm encouraging you. :-)

I have not yet tried Gary's extension, but I intend to do so as soon
as I have a chance. Things have been a bit busy lately.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Register now and save $200. Hurry, offer ends at 11:59 p.m.,
Monday, April 7! Use priority code J8TLD2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-07 04:20:53 UTC

Hi Kelly,

Post by Kelly Anderson
I'm going to assume that you looked at my implementation of Assume.
That is, it was exactly like Assert, except that it threw a
different exception. How that exception gets handled by the
framework is an independent question.

I didn't, but it works as I figured it would work. The implementation
part is pretty straightforward.

I guess I'm not able to separate the /intention/ of the feature from
the implementation as easily as you are. Simply saying that it's
a different exception and the framework can do whatever it wants
with it begs the question of what you intend to be done with it.

It seems to me that the natural meaning of Assume.That only
makes sense in the context of a Theory implementation where
the programmer does not directly control the inputs. If the
programmer controls the inputs, then Assume makes no sense
except as a different kind of error - you gave me bad data.

Post by Kelly Anderson
Would your implementation of Assume be different than that?

No - but what I do with the exception may differ.

Post by Kelly Anderson
Or, are you simply saying that you don't want to release
Assume without also releasing Theory. Do they need to be tied
together to be useful?

That's my view. And we aren't talking about years here, so
I don't see the sense of rushing into it.

Post by Kelly Anderson
I guess I just don't understand if you didn't like the way I
implemented Assume, or if you just have difficulty with doing
it without some fuller approach to Theory first.

I think that it's a bad idea to implement Assume as an
independent feature. I'm not stopping you from implementing
it, since you do think it's a good idea, but you seem to
be trying to convince me to do it.

Post by Kelly Anderson
I'm not trying to be confrontational, I just really don't
understand the sense in which you are objecting to putting
Assume in earlier rather than later...

Separately rather than "as a part of" would be my terms.

Post by Kelly Anderson
Consider this use case of a data test (which is real, I did
it Thursday)... I want to iterate over all the files in a
directory, and validate things about the files if they are
the right type and version. I could do this with your

<snipped long example/>

Your example works fine without Assume - just substitute
"if" for "Assume.That"

2) As I mentionned in the earlier note, it's not hard for

someone to

implement Theory as an extension using the existing

extension points.

Gary has done that, in fact.

I'm not really talking about Theory, I'm talking about
Assume. It's great what Gary has done, but without Assume,
it's only half the story.

So modify it. Or write your own. Or write something else
that incorporates Assume. It will need some kind of test that
interprets the exception - just as only NUnitTestMethods
know what an AssertionException means.

3) What you were proposing in earlier discussions came

across to me

- perhaps wrongly - as a sort of Potemkin Theory implementation.
That's what I reacted to, not to the notion of you or

anyone creating

a true Theory implementation as an addin. Go to it if you

like. It's

just not in /my/ workplan for 2.5.

Yes, that's what I understood.

4) Earlier, in reacting to what I perceive as your trying to push
more and more into 2.5, at the expense of getting started

with 3.0, I

reacted strongly.
Unfortunately, that reaction was taken by others as discouraging
their efforts. Nothing could be farther from my intent.

I understand the tension between putting more stuff in 2.5
and getting on with 3.0, I do.

I don't want to discourage you - not even, :-) - from doing stuff
either. In fact I encourage it. Have you tried out Gary's Theory
extension? Does it work for you?
Does it need something? Can you contribute whatever that is? ...
See, I'm encouraging you. :-)

I have not yet tried Gary's extension, but I intend to do so
as soon as I have a chance. Things have been a bit busy lately.

Well that's one thing we have in common. :-)

Charlie

Post by Kelly Anderson
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by the 2008 JavaOne(SM)
Conference Register now and save $200. Hurry, offer ends at
11:59 p.m., Monday, April 7! Use priority code J8TLD2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java
.sun.com/javaone
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer

Kelly Anderson

2008-04-07 04:56:10 UTC

On Sun, Apr 6, 2008 at 10:20 PM, Charlie Poole

Post by Charlie Poole
I guess I'm not able to separate the /intention/ of the feature from
the implementation as easily as you are. Simply saying that it's
a different exception and the framework can do whatever it wants
with it begs the question of what you intend to be done with it.

I just want an exception that I can throw that means the Test exits
without penalty.

Post by Charlie Poole
It seems to me that the natural meaning of Assume.That only
makes sense in the context of a Theory implementation where
the programmer does not directly control the inputs. If the
programmer controls the inputs, then Assume makes no sense
except as a different kind of error - you gave me bad data.

If the programmer is using a random number generator, or anything just
a little complex like reading data files that someone else may be
generating, then he loses total control over the input. That's part of
what you run into doing data testing.

Post by Kelly Anderson
Would your implementation of Assume be different than that?

No - but what I do with the exception may differ.

That's why I asked for an extension point at that location in the
code... If I can do it another way, then that's fine too.

Post by Kelly Anderson
Or, are you simply saying that you don't want to release
Assume without also releasing Theory. Do they need to be tied
together to be useful?

That's my view. And we aren't talking about years here, so
I don't see the sense of rushing into it.

Ok, I can deal with that.

Post by Kelly Anderson
I guess I just don't understand if you didn't like the way I
implemented Assume, or if you just have difficulty with doing
it without some fuller approach to Theory first.

I wrote the code... it's not like the last time where I was trying to
get you to do it. All you have to do is accept my contribution. :-)

Post by Kelly Anderson
I'm not trying to be confrontational, I just really don't
understand the sense in which you are objecting to putting
Assume in earlier rather than later...

Separately rather than "as a part of" would be my terms.

Ok, I just think it's useful separately. We obviously have a
difference of opinion, and that makes the world go round. Where I have
difficulty is that it "feels" like an emotional response rather than a
technical one. That is probably a limitation of the medium more than
anything else, but I just can't grok your technical argument because
it seems to be based on "I haven't thought about it enough yet" and
not much more. I'm done beating this dead horse. I'm sure we'll get it
right in 3.0, or perhaps close to right in an Addin.

<snipped long example/>
Your example works fine without Assume - just substitute
"if" for "Assume.That"

With that argument, then there is no need for Assume even with a full
blown 3.0 Theory implementation. Unless it's just a semantic issue. My
lack of emotion about BDD should be a clue how I feel about that sort
of argument :-)

2) As I mentionned in the earlier note, it's not hard for

someone to

implement Theory as an extension using the existing

extension points.

Gary has done that, in fact.

I'm not really talking about Theory, I'm talking about
Assume. It's great what Gary has done, but without Assume,
it's only half the story.

I did write it. I guess I integrated it too tightly into the core
though. Perhaps I should take it up with Gary and see if he can
incorporate my Assume into his Theory... if he hasn't done something
similar already.

I have not yet tried Gary's extension, but I intend to do so
as soon as I have a chance. Things have been a bit busy lately.

Well that's one thing we have in common. :-)

I'll see if I can fit it in this week...

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Register now and save $200. Hurry, offer ends at 11:59 p.m.,
Monday, April 7! Use priority code J8TLD2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-07 18:00:00 UTC

Hi Kelly,

Post by Kelly Anderson
I just want an exception that I can throw that means the Test
exits without penalty.

Can't you just return?

Post by Kelly Anderson
If the programmer is using a random number generator, or
anything just a little complex like reading data files that
someone else may be generating, then he loses total control
over the input. That's part of what you run into doing data testing.

Agreed. The question is whether one should be encouraging programmers
to do things like that. Jim and Brad have answered absolutely no.
I have said "when testing a theory." You are saying "whatever the
programmer wants." These are different philosophical views.

Post by Kelly Anderson
Would your implementation of Assume be different than that?

No - but what I do with the exception may differ.

That's why I asked for an extension point at that location in
the code... If I can do it another way, then that's fine too.

I don't understand what point you mean. Exceptions are not handled
at all by the core of nunit. They are handled by extensions.

Post by Kelly Anderson
I wrote the code... it's not like the last time where I was
trying to get you to do it. All you have to do is accept my
contribution. :-)

Where is the code? I must have missed something.

Post by Kelly Anderson
Where I have difficulty is that it "feels" like an emotional
response rather than a technical one.

There is a third category, that of goals and objectives. I think
you may be equating "non-technical" with "emotional" - as we
techies sometimes do.

That said, there comes to be an emotional level, arising
when one's argumets don't seem to be heard.

Post by Kelly Anderson
That is probably a
limitation of the medium more than anything else, but I just
can't grok your technical argument because it seems to be
based on "I haven't thought about it enough yet" and not much
more.

First off, I don't think I've even tried to make a technical
argument. I'm trying to operate at a somewhat higher
level in order to decide /whether/ to include a feature.
Technical arguments belong in the discussion of /how/ to
implement a feature - stuff like "Should we use an exception
or something else."

Secondly, as the guy who takes personal responsibility for
everythin that goes into a release, I feel that "I haven't
thought about it enough" is a darned good reason to delay
doing something.

Thirdly, you are putting a lot of energy into getting
what you want done this month rather than next month,
and that seems a bit emotional to me too.

Post by Kelly Anderson
I'm done beating this dead horse. I'm sure we'll get it
right in 3.0, or perhaps close to right in an Addin.

There is no reason it can't be done right - or the way
you want - in an Addin. In fact, if it can't be, then
it can't be done in 3.0, which will consist entirely of
addins. What you want - to the extent I grasp it - seems
to be perfectly easy to implement in the existing Addin
structure.

If you want to re-cast this discussion to "how can I do it"
rather than "why doesn't NUnit do X?" I'd be glad to help.

Post by Charlie Poole
Your example works fine without Assume - just substitute "if" for
"Assume.That"

With that argument, then there is no need for Assume even
with a full blown 3.0 Theory implementation. Unless it's just
a semantic issue. My lack of emotion about BDD should be a
clue how I feel about that sort of argument :-)

It was your example. :-) What I'm saying here is that you
have not managed to make me understand /how/ what you want
is different from an if statement.

2) As I mentionned in the earlier note, it's not hard for >

someone to > > implement Theory as an extension using the

existing

Post by Kelly Anderson
extension points.

Gary has done that, in fact.

I did write it. I guess I integrated it too tightly into the
core though. Perhaps I should take it up with Gary and see if
he can incorporate my Assume into his Theory... if he hasn't
done something similar already.

As I said, I may have missed something here. I knew you had written
some code, but I thought it was only on-the-job and not available for
us to see. If you let me know (again?) where it is, I'll take
a look and maybe that will clarify things.

Charlie

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Register now and save $200. Hurry, offer ends at 11:59 p.m.,
Monday, April 7! Use priority code J8TLD2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Kelly Anderson

2008-04-08 22:38:38 UTC

On Mon, Apr 7, 2008 at 12:00 PM, Charlie Poole

Post by Charlie Poole
Hi Kelly,

Post by Kelly Anderson
I just want an exception that I can throw that means the Test
exits without penalty.

Can't you just return?

I suppose. For me it's a question of expressiveness. Yes, returning is
incrementally faster than exception handling. It's just not as
expressive in my opinion.

I guess I'm pragmatic in favor of flexibility. If people can use tools
in different ways than the author might have intended, in ways that
are useful to them, why is that a bad thing?

Post by Kelly Anderson
Would your implementation of Assume be different than that?

No - but what I do with the exception may differ.

That's why I asked for an extension point at that location in
the code... If I can do it another way, then that's fine too.

I don't understand what point you mean. Exceptions are not handled
at all by the core of nunit. They are handled by extensions.

See my post to Gary a minute ago. In the Framework the Assert
exception is handled, I want to handle the Assume exception in a
similar way.

Post by Kelly Anderson
I wrote the code... it's not like the last time where I was
trying to get you to do it. All you have to do is accept my
contribution. :-)

Where is the code? I must have missed something.

I emailed it to you. Would you like it again?

Post by Kelly Anderson
Where I have difficulty is that it "feels" like an emotional
response rather than a technical one.

There is a third category, that of goals and objectives. I think
you may be equating "non-technical" with "emotional" - as we
techies sometimes do.
That said, there comes to be an emotional level, arising
when one's argumets don't seem to be heard.

I admit to feeling a slight tinge of frustration, but I'm trying to
make my arguments on a technical basis... Ok, perhaps not just a
technical basis, but rather on the basis of expressiveness. That's not
exactly a technical argument either. :-)

When you have Theory with a Theory Explorer, then Assume rises to a
technical level.

Ok. Fair enough. I guess I've been trying to goad you into a technical
argument, and it isn't working.

Post by Charlie Poole
Secondly, as the guy who takes personal responsibility for
everythin that goes into a release, I feel that "I haven't
thought about it enough" is a darned good reason to delay
doing something.

Ok.

Post by Charlie Poole
Thirdly, you are putting a lot of energy into getting
what you want done this month rather than next month,
and that seems a bit emotional to me too.

Ok. That's a fair observation.

Post by Kelly Anderson
I'm done beating this dead horse. I'm sure we'll get it
right in 3.0, or perhaps close to right in an Addin.

Ok, I could not figure out how to do it in the current structure
without changing the code in the nunit.framework, specifically in
ProcessException.

Post by Charlie Poole
If you want to re-cast this discussion to "how can I do it"
rather than "why doesn't NUnit do X?" I'd be glad to help.

I'm super happy with that approach. How do I do what's now done in
ProcessException in an Addin?

Post by Charlie Poole
Your example works fine without Assume - just substitute "if" for
"Assume.That"

With that argument, then there is no need for Assume even
with a full blown 3.0 Theory implementation. Unless it's just
a semantic issue. My lack of emotion about BDD should be a
clue how I feel about that sort of argument :-)

It was your example. :-) What I'm saying here is that you
have not managed to make me understand /how/ what you want
is different from an if statement.

It differs only in expressiveness. You could say exactly the same for
ExpectedException... because you can write a try catch block and do
your own Assert.Fail after the catch.

Post by Kelly Anderson
I did write it. I guess I integrated it too tightly into the
core though. Perhaps I should take it up with Gary and see if
he can incorporate my Assume into his Theory... if he hasn't
done something similar already.

Sure, I'll send it again to your individual address.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Register now and save $200. Hurry, offer ends at 11:59 p.m.,
Monday, April 7! Use priority code J8TLD2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-09 00:51:31 UTC

Hi Kelly,

I just want an exception that I can throw that means the Test >

exits without penalty.
Can't you just return?

I suppose. For me it's a question of expressiveness. Yes,
returning is incrementally faster than exception handling.
It's just not as expressive in my opinion.

Actually, I don't find it more expressive - which is probably why
we aren't agreeing. To me, it looks as if it ought to do something
more than just return, so I waste energy trying to figure it out.
Then I learn it's just a way for the test to succeed. I guess
expressiveness is a matter of opinion.

If the programmer is using a random number generator, or >

anything just a little complex like reading data files that
someone
else may be generating, then he loses total control > over

the input.

Post by Kelly Anderson
That's part of what you run into doing data testing.
Agreed. The question is whether one should be encouraging

programmers

Post by Kelly Anderson
to do things like that. Jim and Brad have answered absolutely no.
I have said "when testing a theory." You are saying "whatever the
programmer wants." These are different philosophical views.

I guess I'm pragmatic in favor of flexibility. If people can
use tools in different ways than the author might have
intended, in ways that are useful to them, why is that a bad thing?

That's what I said - that you have a different view. I didn't say
it was a bad thing, did I?

Post by Kelly Anderson
Would your implementation of Assume be different than that?

No - but what I do with the exception may differ.

That's why I asked for an extension point at that location in >

the code... If I can do it another way, then that's fine too.
I don't understand what point you mean. Exceptions are not

handled

Post by Kelly Anderson
at all by the core of nunit. They are handled by extensions.

See my post to Gary a minute ago. In the Framework the Assert
exception is handled, I want to handle the Assume exception
in a similar way.

They are handled by NUnitTestCase, which is not part of the
core of NUnit - it's just another extension like you might write.

I wrote the code... it's not like the last time where I was >

trying to get you to do it. All you have to do is accept my >
contribution. :-)
Where is the code? I must have missed something.

I emailed it to you. Would you like it again?

Got it now.

Where I have difficulty is that it "feels" like an emotional >

response rather than a technical one.
There is a third category, that of goals and objectives. I

think you

Post by Kelly Anderson
may be equating "non-technical" with "emotional" - as we techies
sometimes do.
That said, there comes to be an emotional level, arising

when one's

Post by Kelly Anderson
argumets don't seem to be heard.

I admit to feeling a slight tinge of frustration, but I'm
trying to make my arguments on a technical basis... Ok,
perhaps not just a technical basis, but rather on the basis
of expressiveness. That's not exactly a technical argument either. :-)
When you have Theory with a Theory Explorer, then Assume
rises to a technical level.

I don't understand that.

That is probably a
limitation of the medium more than anything else, but I just >

can't grok your technical argument because it seems to be
based on
"I haven't thought about it enough yet" and not much > more.
First off, I don't think I've even tried to make a technical
argument. I'm trying to operate at a somewhat higher level

in order

Post by Kelly Anderson
to decide /whether/ to include a feature.
Technical arguments belong in the discussion of /how/ to

implement a

Post by Kelly Anderson
feature - stuff like "Should we use an exception or

something else."
Ok. Fair enough. I guess I've been trying to goad you into a
technical argument, and it isn't working.

Post by Kelly Anderson
Secondly, as the guy who takes personal responsibility for

everythin

Post by Kelly Anderson
that goes into a release, I feel that "I haven't thought about it
enough" is a darned good reason to delay doing something.

Ok.

Post by Kelly Anderson
Thirdly, you are putting a lot of energy into getting

what you want

Post by Kelly Anderson
done this month rather than next month, and that seems a bit
emotional to me too.

Ok. That's a fair observation.

I'm done beating this dead horse. I'm sure we'll get it

right in
3.0, or perhaps close to right in an Addin.
There is no reason it can't be done right - or the way

you want - in

Post by Kelly Anderson
an Addin. In fact, if it can't be, then it can't be done in 3.0,
which will consist entirely of addins. What you want - to

the extent

Post by Kelly Anderson
I grasp it - seems to be perfectly easy to implement in

the existing

Post by Kelly Anderson
Addin structure.

Ok, I could not figure out how to do it in the current
structure without changing the code in the nunit.framework,
specifically in ProcessException.

ProcessException is in nunit.core, in TestMethod or TestCase -
I forget which at the moment. But it's not truly "core" - that
is it's not part of the stuff we extend. Rather, it is itself
an extension. When you write your own extension, with your own
test case, then you can handle any exception you like in any
way you like.

In principle, this is no different from the exceptions that
signal a failure. There is no place in NUnit that recognizes
those exceptions for all tests. NUnitTestMethod recogizes the
particular exception that it knows about. CsUnitTestMethod -
to take one example - recogizes a completely different exception.

Post by Kelly Anderson
If you want to re-cast this discussion to "how can I do it"
rather than "why doesn't NUnit do X?" I'd be glad to help.

I'm super happy with that approach. How do I do what's now
done in ProcessException in an Addin?

By creating a test case that contains the code to process that
kind of exception.

Post by Charlie Poole
Your example works fine without Assume - just

substitute "if"

Post by Kelly Anderson
for > > "Assume.That"

With that argument, then there is no need for Assume

even > with a

Post by Kelly Anderson
full blown 3.0 Theory implementation. Unless it's just > a

semantic

Post by Kelly Anderson
issue. My lack of emotion about BDD should be a > clue how I feel
about that sort of argument :-)
It was your example. :-) What I'm saying here is that you

have not

Post by Kelly Anderson
managed to make me understand /how/ what you want is

different from

Post by Kelly Anderson
an if statement.

It differs only in expressiveness. You could say exactly the
same for ExpectedException... because you can write a try
catch block and do your own Assert.Fail after the catch.

OK, then the discussion turns on whether one finds it
expressive in the sense you are using it.

I did write it. I guess I integrated it too tightly into the
core though. Perhaps I should take it up with Gary and

see if > he

Post by Kelly Anderson
can incorporate my Assume into his Theory... if he hasn't > done
something similar already.
As I said, I may have missed something here. I knew you

had written

Post by Kelly Anderson
some code, but I thought it was only on-the-job and not

available for

Post by Kelly Anderson
us to see. If you let me know (again?) where it is, I'll

take a look

Post by Kelly Anderson
and maybe that will clarify things.

Sure, I'll send it again to your individual address.

No need now - I found it and responded separately

Charlie

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Kelly Anderson

2008-04-09 02:07:01 UTC

On Tue, Apr 8, 2008 at 6:51 PM, Charlie Poole

Post by Charlie Poole
Hi Kelly,
Actually, I don't find it more expressive - which is probably why
we aren't agreeing. To me, it looks as if it ought to do something
more than just return, so I waste energy trying to figure it out.
Then I learn it's just a way for the test to succeed. I guess
expressiveness is a matter of opinion.

That's really interesting. Do you find Theory to be more expressive? I
thought that part of what made Theory more expressive was Assume. I
guess it might be a matter of what makes sense to you... for example I
don't find this code:

[Test]
public void x()
{
Assert.Throws<ArgumentException>(new TestSnippet(MethodThatThrows) );
}

To be more expressive than:

[Test]
[ExpectedException(typeof(ArgumentException))]
{
MethodThatThrows();
}

But I guess maybe that's just me. It may also be that I'm not yet used
to the delegate syntax, and I may grow to like it as I have with
Assert.That()...

Of course, there is also the possibility that you might grow to like
Assume.That, if you used it as much as I used Assert.That...

Post by Kelly Anderson
I guess I'm pragmatic in favor of flexibility. If people can
use tools in different ways than the author might have
intended, in ways that are useful to them, why is that a bad thing?

That's what I said - that you have a different view. I didn't say
it was a bad thing, did I?

No, you didn't. :-)

Post by Kelly Anderson
See my post to Gary a minute ago. In the Framework the Assert
exception is handled, I want to handle the Assume exception
in a similar way.

They are handled by NUnitTestCase, which is not part of the
core of NUnit - it's just another extension like you might write.

But to write that extension would mean that Assume would only work
within [KellyTests] not [Test]... and I'd really like it to work for
any other sort of test as well... Maybe that's asking too much of the
extensibility framework.

Post by Kelly Anderson
I admit to feeling a slight tinge of frustration, but I'm
trying to make my arguments on a technical basis... Ok,
perhaps not just a technical basis, but rather on the basis
of expressiveness. That's not exactly a technical argument either. :-)
When you have Theory with a Theory Explorer, then Assume
rises to a technical level.

I don't understand that.

With a Theory Explorer, you can't predict what the parameters might
be, therefore, you would have to have Assume. If you just used
if(condition) return; then you wouldn't have sure knowledge that the
Theory Explorer had generated cases that reached the code to be truly
explored, that is, the code after the Assume.That's... This could lead
to bad theories. For example, if I wrote this theory...

private bool IsEven(int number)
{
return true; // wrong
}

[Theory]
public void AllEvenNumbersShouldDivideToFullNumbers(int evenNumber)
{
if (!IsEven(evenNumber))
return;
Assert.That(evenNumber/2, Is.EqualTo( (int) ( (double) evenNumber)/2.0 ) );
}

You see that I messed up the implementation of IsEven, but I would
never know it because the Theory would always pass. However, if I
wrote it with Assume like so...

[Theory]
public void AllEvenNumbersShouldDivideToFullNumbers(int evenNumber)
{
Assume.That (IsEven(evenNumber));
Assert.That(evenNumber/2, Is.EqualTo( (int) ( (double) evenNumber)/2.0));
}

Then the framework would tell me that IsEven was implemented
incorrectly, because the test would never complete successfully as the
Assume would fire every time.

That seems like a technical argument, rather than an argument of expressiveness.

Post by Kelly Anderson
Ok, I could not figure out how to do it in the current
structure without changing the code in the nunit.framework,
specifically in ProcessException.

It's in TestMethod. Ok, then I just misunderstood the architecture. I
would just have to subclass TestCase, and write my own kind of
ProcessException...

Post by Charlie Poole
In principle, this is no different from the exceptions that
signal a failure. There is no place in NUnit that recognizes
those exceptions for all tests. NUnitTestMethod recogizes the
particular exception that it knows about. CsUnitTestMethod -
to take one example - recogizes a completely different exception.

Ok, I think I "get it" now. I really would like for all kinds of
TestCase subclasses to be able to handle Assume, just like they all
can handle Assert (I suppose that they all just choose to handle
Assert at this point, if I'm understanding)... but that is probably
not going to happen.

Post by Charlie Poole
If you want to re-cast this discussion to "how can I do it"
rather than "why doesn't NUnit do X?" I'd be glad to help.

I'm super happy with that approach. How do I do what's now
done in ProcessException in an Addin?

By creating a test case that contains the code to process that
kind of exception.

Ok.

Post by Kelly Anderson
It differs only in expressiveness. You could say exactly the
same for ExpectedException... because you can write a try
catch block and do your own Assert.Fail after the catch.

OK, then the discussion turns on whether one finds it
expressive in the sense you are using it.

Right. Got it.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-09 17:20:38 UTC

Hi Kelly,

Post by Charlie Poole
Actually, I don't find it more expressive - which is

probably why we

Post by Charlie Poole
aren't agreeing. To me, it looks as if it ought to do

something more

Post by Charlie Poole
than just return, so I waste energy trying to figure it out.
Then I learn it's just a way for the test to succeed. I guess
expressiveness is a matter of opinion.

That's really interesting. Do you find Theory to be more
expressive? I thought that part of what made Theory more
expressive was Assume. I guess it might be a matter of what

My point was that Assume as a part of Theory does NOT merely
return success. It's not the word that I find unexpressive,
it's that it doesn't seem to express what you are doing.

It's like a method name, which expresses something, but
not what the method really does. Usually, we just change
the name of the method. In this case, I think we want
Assume to do something other than what you're having
it do. I don't want to implement it and then have to
change the semantics at a later time.

[Test]
public void x()
{
Assert.Throws<ArgumentException>(new
TestSnippet(MethodThatThrows) ); }
[Test]
[ExpectedException(typeof(ArgumentException))]
{
MethodThatThrows();
}
But I guess maybe that's just me. It may also be that I'm not
yet used to the delegate syntax, and I may grow to like it as
I have with Assert.That()...

This is on a different plain AFAIC. It's a well-tested syntax,
used in other apps and has been requested for NUnit for years.
I'm not expressing a preference here, I;m responding to a request.

And anyway, you're using the .NET 1.1 syntax. It looks lots
better as
Assert.Throws<ArgumentException>( MethodThatThrows );

Which is why I turned down the requests till now. :-)

Of course, there is also the possibility that you might grow
to like Assume.That, if you used it as much as I used Assert.That...

I like it! I like it! I don't like it as a synonym for Succeed.

See my post to Gary a minute ago. In the Framework the Assert >

exception is handled, I want to handle the Assume exception > in a
similar way.
They are handled by NUnitTestCase, which is not part of

the core of

Post by Charlie Poole
NUnit - it's just another extension like you might write.

But to write that extension would mean that Assume would only
work within [KellyTests] not [Test]... and I'd really like it
to work for any other sort of test as well... Maybe that's
asking too much of the extensibility framework.

If we had the ablility to extend extensions, then it could
work for an NUnitTestMethod. As I made clear at the start of
this, we cannot have extensions on extensions without the
major rework of the addin architecture planned for 3.0.

I could be wrong, but I think you are having difficulty
accepting that NUnitTestMethod is itself an extension. If
so, consider what should happen if you use Assume.That in
a csunit test method.

Post by Charlie Poole
I don't understand that.

In my view, that's not Theory Explorer, it's Theory. I believe
that the base implementation of Theory should provide for mixing
up available parameters. The notion of a spearate program to
suggest additional parameters (Theory Explorer) is an important
idea and one we may see implemented some day. But a supplier
of parameters for Theory is not going to be deterministic,
at least in my view, the way a TestCase is.

So yes, I believe something performing the function you assign
to Assume.That is needed in a Theory. I don't believe it has
to be Assume.That, but it is definitely a likely candidate.
Since it is likely that Assume.That will be used with Theory,
and that it will have a different meaning than what you have
given it, I don't want to start using it with your meaning
right now.

private bool IsEven(int number)
{
return true; // wrong
}
[Theory]
public void AllEvenNumbersShouldDivideToFullNumbers(int evenNumber) {
if (!IsEven(evenNumber))
return;
Assert.That(evenNumber/2, Is.EqualTo( (int) ( (double)
evenNumber)/2.0 ) ); }
You see that I messed up the implementation of IsEven, but I
would never know it because the Theory would always pass.
However, if I wrote it with Assume like so...
[Theory]
public void AllEvenNumbersShouldDivideToFullNumbers(int evenNumber) {
Assume.That (IsEven(evenNumber));
Assert.That(evenNumber/2, Is.EqualTo( (int) ( (double)
evenNumber)/2.0)); }
Then the framework would tell me that IsEven was implemented
incorrectly, because the test would never complete
successfully as the Assume would fire every time.

That's not what your Assume.That does, however. We're not
(I'm not) talking about the general design of Theory here,
but about your particular implementation which equates
to the use of return.

That seems like a technical argument, rather than an argument
of expressiveness.

Again - YOUR Assert.That expresses something it doesn't
deliver. Giving users a method that seems to do something
but doesn't is not such a good idea.

Assert.Succeed() has been requested and I'm inclined to
implement it, since it's trivial. Would you be equally
happy using Assert.Succeed in place of your non-Theory
uses of Assume.That?

Post by Charlie Poole
ProcessException is in nunit.core, in TestMethod or TestCase - I
forget which at the moment. But it's not truly "core" -

that is it's

Post by Charlie Poole
not part of the stuff we extend. Rather, it is itself an

extension.

Post by Charlie Poole
When you write your own extension, with your own test

case, then you

Post by Charlie Poole
can handle any exception you like in any way you like.

It's in TestMethod. Ok, then I just misunderstood the
architecture. I would just have to subclass TestCase, and
write my own kind of ProcessException...

Or take a look at what methods are virtual.

Ideally, I'd like to see extensions subclass Test, but it's
hard to pass up all the pre-implemented stuff. I've been
guilty of falling into the inheritance trap myself. In
3.0, I want to look at the versioning issue for addins.

Post by Charlie Poole
In principle, this is no different from the exceptions

that signal a

Post by Charlie Poole
failure. There is no place in NUnit that recognizes those

exceptions

Post by Charlie Poole
for all tests. NUnitTestMethod recogizes the particular exception
that it knows about. CsUnitTestMethod - to take one example -
recogizes a completely different exception.

Ok, I think I "get it" now. I really would like for all kinds
of TestCase subclasses to be able to handle Assume, just like
they all can handle Assert (I suppose that they all just
choose to handle Assert at this point, if I'm
understanding)... but that is probably not going to happen.

Bear in mind that TestCase and TestMethod were not written to
be inherited from by Addins. It's just an expedient thing we
are all doing as a quick way to get work done. I'm hoping
to see us get away from this as the addin thing matures.

Even so, NOT all TestCase-derived clases handle the same Assert.
For example, if you use an NUnit Assert in a csUnit test class,
it comes up as an unexpected exception, not a failure. That's
because CsUnitTestMethod doesn't understand NUnit's exception.

This isn't a bug: it's the way it has to work. Different
frameworks handle different exceptions. The different
exception types are not what is understood in common among
all tests under NUnit: it's the TestResults that are
created by the individual test types.

In an earlier version of NUnit's extensibility, there was
a concept that each test type fell into under a "TestFramework"
and could tell NUnit under which frameworks it should be
active. That was too complicated and I took it out, although
there is still some code left over that allows you to list
frameworks that have been loaded. It could be that we will
want to reconsider this decision in the future, so we could
say AssumeException is recognized by all test types that
are under the NUnitTestFramework. Right now, that's not
possible.

Charlie

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Kelly Anderson

2008-04-09 21:15:55 UTC

On Wed, Apr 9, 2008 at 11:20 AM, Charlie Poole

Post by Charlie Poole
Hi Kelly,

Post by Charlie Poole
Actually, I don't find it more expressive - which is

probably why we

Post by Charlie Poole
aren't agreeing. To me, it looks as if it ought to do

something more

Post by Charlie Poole
than just return, so I waste energy trying to figure it out.
Then I learn it's just a way for the test to succeed. I guess
expressiveness is a matter of opinion.

That's really interesting. Do you find Theory to be more
expressive? I thought that part of what made Theory more
expressive was Assume. I guess it might be a matter of what

My point was that Assume as a part of Theory does NOT merely
return success. It's not the word that I find unexpressive,
it's that it doesn't seem to express what you are doing.
It's like a method name, which expresses something, but
not what the method really does. Usually, we just change
the name of the method. In this case, I think we want
Assume to do something other than what you're having
it do. I don't want to implement it and then have to
change the semantics at a later time.

I guess that makes sense. I'm just not sure I can think of what it
would do differently, other than how the framework deals with it when
the exception fires. There is no way to return from the middle of a
function other than an exception (that I can think of) so I don't know
how else you would implement something like assume in the middle of
the code. You could somehow do it before the method is called, but
that separates the assumption from the assertions, which I find less
expressive than having them close together in the code.

The proximity of the Assumes to the Asserts seems fundamental to me in
terms of expressiveness. Does that make sense to you in the context of
Theory?

This is on a different plain AFAIC. It's a well-tested syntax,
used in other apps and has been requested for NUnit for years.
I'm not expressing a preference here, I;m responding to a request.

Ok.

Post by Charlie Poole
And anyway, you're using the .NET 1.1 syntax. It looks lots
better as
Assert.Throws<ArgumentException>( MethodThatThrows );
Which is why I turned down the requests till now. :-)

Still looks a bit funny, but I suppose I could get used to it.

Of course, there is also the possibility that you might grow
to like Assume.That, if you used it as much as I used Assert.That...

I like it! I like it! I don't like it as a synonym for Succeed.

Sigh.

See my post to Gary a minute ago. In the Framework the Assert >

exception is handled, I want to handle the Assume exception > in a
similar way.
They are handled by NUnitTestCase, which is not part of

the core of

Post by Charlie Poole
NUnit - it's just another extension like you might write.

But to write that extension would mean that Assume would only
work within [KellyTests] not [Test]... and I'd really like it
to work for any other sort of test as well... Maybe that's
asking too much of the extensibility framework.

Ok, just my previous misunderstanding of the architecture, which I
THINK is now straightened out.

Post by Charlie Poole
I could be wrong, but I think you are having difficulty
accepting that NUnitTestMethod is itself an extension. If
so, consider what should happen if you use Assume.That in
a csunit test method.

Ok I get it now.

Post by Charlie Poole
I don't understand that.

I think it's important when RUNNING a Theory for the run to be
repeatable, thus deterministic (if my definition of deterministic is
the same as yours). When you go exploring the space a Theory covers,
then it's important to be non-deterministic (but report the case(s)
where the Theory fails so that new cases can be added to the Theory
data source.)

Post by Charlie Poole
So yes, I believe something performing the function you assign
to Assume.That is needed in a Theory. I don't believe it has
to be Assume.That, but it is definitely a likely candidate.
Since it is likely that Assume.That will be used with Theory,
and that it will have a different meaning than what you have
given it, I don't want to start using it with your meaning
right now.

Ok, that makes a certain amount of sense, I just don't see the meaning
as being quite as significant as you are indicating. This is of course
just opinion when you get to that point.

That's not what your Assume.That does, however. We're not
(I'm not) talking about the general design of Theory here,
but about your particular implementation which equates
to the use of return.

My implementation throws AssumeException, which is not equivalent to
return. My implementation of the catch of AssumeException in the test
case makes it equivalent, but I'm kind of separating Assume.That from
the handling of the exception. If you're talking about how the
exception is handled, then point taken. But I believe throwing the
exception has the right implementation for Theory...

That seems like a technical argument, rather than an argument of expressiveness.

Again - YOUR Assert.That expresses something it doesn't
deliver. Giving users a method that seems to do something
but doesn't is not such a good idea.
Assert.Succeed() has been requested and I'm inclined to
implement it, since it's trivial. Would you be equally
happy using Assert.Succeed in place of your non-Theory
uses of Assume.That?

As long as I could use a Constraint with Assert.Succeed... but then it
reads funny
Assert.Succeed(Is.EqualTo(f)); ???

If you just do Assert.Succeed() that doesn't give you the same level
of expressiveness. If you did

Succeed.If(Is.EqualTo(f));

That would seem grammatically and syntactically similar to what I'm proposing.

Post by Charlie Poole
ProcessException is in nunit.core, in TestMethod or TestCase - I
forget which at the moment. But it's not truly "core" -

that is it's

Post by Charlie Poole
not part of the stuff we extend. Rather, it is itself an

extension.

Post by Charlie Poole
When you write your own extension, with your own test

case, then you

Post by Charlie Poole
can handle any exception you like in any way you like.

It's in TestMethod. Ok, then I just misunderstood the
architecture. I would just have to subclass TestCase, and
write my own kind of ProcessException...

Or take a look at what methods are virtual.
Ideally, I'd like to see extensions subclass Test, but it's
hard to pass up all the pre-implemented stuff. I've been
guilty of falling into the inheritance trap myself. In
3.0, I want to look at the versioning issue for addins.

More interfaces, less inheritance! :-)

Post by Charlie Poole
In principle, this is no different from the exceptions

that signal a

Post by Charlie Poole
failure. There is no place in NUnit that recognizes those

exceptions

Post by Charlie Poole
for all tests. NUnitTestMethod recogizes the particular exception
that it knows about. CsUnitTestMethod - to take one example -
recogizes a completely different exception.

Ok, I think I "get it" now. I really would like for all kinds
of TestCase subclasses to be able to handle Assume, just like
they all can handle Assert (I suppose that they all just
choose to handle Assert at this point, if I'm
understanding)... but that is probably not going to happen.

I agree. More interfaces could help.

Post by Charlie Poole
Even so, NOT all TestCase-derived clases handle the same Assert.
For example, if you use an NUnit Assert in a csUnit test class,
it comes up as an unexpected exception, not a failure. That's
because CsUnitTestMethod doesn't understand NUnit's exception.

Ok, I get it now :-)

Post by Charlie Poole
This isn't a bug: it's the way it has to work. Different
frameworks handle different exceptions. The different
exception types are not what is understood in common among
all tests under NUnit: it's the TestResults that are
created by the individual test types.

I thought they were orthogonal concerns before.

Post by Charlie Poole
In an earlier version of NUnit's extensibility, there was
a concept that each test type fell into under a "TestFramework"
and could tell NUnit under which frameworks it should be
active. That was too complicated and I took it out, although
there is still some code left over that allows you to list
frameworks that have been loaded. It could be that we will
want to reconsider this decision in the future, so we could
say AssumeException is recognized by all test types that
are under the NUnitTestFramework. Right now, that's not
possible.

I agree that it is not currently possible. The question is whether it
is eventually desirable, and that's a separate question.

I assume a good design might have exception types orthogonal to test
types, but there may be cases I haven't considered, so I'm not ready
to assert my assumption. :-)

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-10 00:58:16 UTC

Hi Kelly,

Post by Charlie Poole
My point was that Assume as a part of Theory does NOT

merely return

Post by Charlie Poole
success. It's not the word that I find unexpressive, it's that it
doesn't seem to express what you are doing.

I'm researching that - it's part of figuring out how to
communicate results without exceptions.

But that's not really my point...

You could somehow do it before the method is called, but that
separates the assumption from the assertions, which I find
less expressive than having them close together in the code.

I'm just not willing to consider the throwing of the exception
separately from what is done with it by the framework. You've
been saying you want the assumptionexception to have meaning
in all tests. If that were to be so, I'd want that meaning
to be the same for all tests and I don't really know what
the meaning might be.

The proximity of the Assumes to the Asserts seems fundamental
to me in terms of expressiveness. Does that make sense to you
in the context of Theory?

I like that aspect of David's implementation and it was my
first choice of how to go. But I won't jump from that to
the notion that no other way of doing it is possible, or
maybe even better. Gary has implemented a different approach,
for example. I want to write a non-trivial program using his
implementation. Then I want to write my own. Then I feel like
I'll be qualified to have a preference.

Post by Kelly Anderson
Of course, there is also the possibility that you might

grow > to

Post by Charlie Poole
like Assume.That, if you used it as much as I used Assert.That...
I like it! I like it! I don't like it as a synonym for Succeed.

Sigh.

???

Post by Charlie Poole
I could be wrong, but I think you are having difficulty accepting
that NUnitTestMethod is itself an extension. If so, consider what
should happen if you use Assume.That in a csunit test method.

Ok I get it now.

Cool.

Post by Charlie Poole
In my view, that's not Theory Explorer, it's Theory. I

believe that

Post by Charlie Poole
the base implementation of Theory should provide for mixing up
available parameters. The notion of a spearate program to suggest
additional parameters (Theory Explorer) is an important

idea and one

Post by Charlie Poole
we may see implemented some day. But a supplier of parameters for
Theory is not going to be deterministic, at least in my

view, the way

Post by Charlie Poole
a TestCase is.

I think it's important when RUNNING a Theory for the run to
be repeatable, thus deterministic (if my definition of
deterministic is the same as yours). When you go exploring
the space a Theory covers, then it's important to be
non-deterministic (but report the case(s) where the Theory
fails so that new cases can be added to the Theory data source.)

You're right. Theory is probably going to be deterministic. Just
not under the direct control of the programmer as are normal
data-driven tests.

Post by Charlie Poole
That's not what your Assume.That does, however. We're not

(I'm not)

Post by Charlie Poole
talking about the general design of Theory here, but about your
particular implementation which equates to the use of return.

My implementation throws AssumeException, which is not
equivalent to return. My implementation of the catch of
AssumeException in the test case makes it equivalent, but I'm
kind of separating Assume.That from the handling of the
exception. If you're talking about how the exception is
handled, then point taken. But I believe throwing the
exception has the right implementation for Theory...

Probably. But it's not the main point IMO. As a user, I
don't throw an exception. I just Assume something. You're
saying "As a user, I want to be able to make an assumption
and have it verified. If it succeeds, the test continues.
If it fails, the test terminates and succeeds." This
doesn't make a lot of sense.

I think we have to design the entire interaction here,
not just focus on the implementation details.

Post by Charlie Poole
Assert.Succeed() has been requested and I'm inclined to implement
it, since it's trivial. Would you be equally happy using
Assert.Succeed in place of your non-Theory uses of Assume.That?

As requested, it's just a command, like Assert.Fail or Assert.Ignore.

Succeed.If(Is.EqualTo(f));

For that example, you could do
if ( a == b ) Assert.Succeed();
but yes, I can see that a constraint syntax might be useful
for some cases.

That would seem grammatically and syntactically similar to
what I'm proposing.

Yeah, but it actually says what it does.

Post by Charlie Poole
Even so, NOT all TestCase-derived clases handle the same Assert.
For example, if you use an NUnit Assert in a csUnit test

class, it

Post by Charlie Poole
comes up as an unexpected exception, not a failure. That's because
CsUnitTestMethod doesn't understand NUnit's exception.

Ok, I get it now :-)

Great.

I agree that it is not currently possible. The question is
whether it is eventually desirable, and that's a separate question.

That's been decided. It will be possible in 3.0.

Charlie

I assume a good design might have exception types orthogonal
to test types, but there may be cases I haven't considered,
so I'm not ready to assert my assumption. :-)
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by the 2008 JavaOne(SM)
Conference Don't miss this year's exciting event. There's
still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java
.sun.com/javaone
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer

Kelly Anderson

2008-04-10 15:32:16 UTC

On Wed, Apr 9, 2008 at 6:58 PM, Charlie Poole

return from the middle of a function other than an exception
(that I can think of) so I don't know how else you would
implement something like assume in the middle of the code.

I'm researching that - it's part of figuring out how to
communicate results without exceptions.

I suppose you could call a function that somehow didn't return to the
caller... not sure how though.

Post by Charlie Poole
But that's not really my point...
I'm just not willing to consider the throwing of the exception
separately from what is done with it by the framework. You've
been saying you want the assumptionexception to have meaning
in all tests. If that were to be so, I'd want that meaning
to be the same for all tests and I don't really know what
the meaning might be.

I have given up on that request completely. I see your point now that
I understand the framework and context better. You are right. I don't
necessarily want Assume to work for Test anymore, or even
IterativeTest. I would like it for Theory still, and will work with
Gary to see that it gets into his extension.

The proximity of the Assumes to the Asserts seems fundamental
to me in terms of expressiveness. Does that make sense to you
in the context of Theory?

Ok, that works for me.

Post by Kelly Anderson
Of course, there is also the possibility that you might

grow > to

like Assume.That, if you used it as much as I used Assert.That...
I like it! I like it! I don't like it as a synonym for Succeed.

Sigh.

???

You're hard to read sometimes. :-) That's all.

I think it's important when RUNNING a Theory for the run to
be repeatable, thus deterministic (if my definition of
deterministic is the same as yours). When you go exploring
the space a Theory covers, then it's important to be
non-deterministic (but report the case(s) where the Theory
fails so that new cases can be added to the Theory data source.)

You're right. Theory is probably going to be deterministic. Just
not under the direct control of the programmer as are normal
data-driven tests.

Ok, then we're on the same page.

My implementation throws AssumeException, which is not
equivalent to return. My implementation of the catch of
AssumeException in the test case makes it equivalent, but I'm
kind of separating Assume.That from the handling of the
exception. If you're talking about how the exception is
handled, then point taken. But I believe throwing the
exception has the right implementation for Theory...

Why not? That's what Assume means in all the literature. The only
variant is that if the test terminates in ALL cases of testing the
Theory, then it is not a success.

Post by Charlie Poole
I think we have to design the entire interaction here,
not just focus on the implementation details.

Ok. I suppose the part you are interested in talking about is how to
go about specifying the data set going into the Theory. I agree that's
an important thing to talk about too. I've seen some of MbUnit's
approach to the problem. Some of it I like, some of it I think is
syntactically odd.

If you just do Assert.Succeed() that doesn't give you the
same level of expressiveness. If you did

As requested, it's just a command, like Assert.Fail or Assert.Ignore.

Ok...

Succeed.If(Is.EqualTo(f));

For that example, you could do
if ( a == b ) Assert.Succeed();
but yes, I can see that a constraint syntax might be useful
for some cases.

It might also be nice to:

Assert.Fail(Constraint);

and so forth, but it does lead to the rather ungrammatic:

Assert.Fail(Is.EqualTo(4));

That would seem grammatically and syntactically similar to
what I'm proposing.

Yeah, but it actually says what it does.

Ok, ok.

I agree that it is not currently possible. The question is
whether it is eventually desirable, and that's a separate question.

That's been decided. It will be possible in 3.0.

Super.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-10 19:18:18 UTC

Hi Kelly,

Post by Kelly Anderson
You're hard to read sometimes. :-) That's all.

I work hard to have no hidden meanings beyond what I say.
Oddly, that sometimes confuses folks.

Post by Charlie Poole
Probably. But it's not the main point IMO. As a user, I

don't throw

Post by Charlie Poole
an exception. I just Assume something. You're saying "As a user, I
want to be able to make an assumption and have it verified. If it
succeeds, the test continues.
If it fails, the test terminates and succeeds." This

doesn't make a

Post by Charlie Poole
lot of sense.

Why not? That's what Assume means in all the literature. The
only variant is that if the test terminates in ALL cases of
testing the Theory, then it is not a success.

I don't think so. I believe you are mixing two things here:
the Theory and the method that runs the test. NUnit need not
know about the individual method results - depending on the
implementation. That's not to say we don't care about the
data on which the theory failed - we want to know that -
but we don't have to consider every invocation of the
method a test.

You may recall I wrote about this alternative to creating
multiple tests in an earlier thread.

Post by Kelly Anderson
Succeed.If(Is.EqualTo(f));

For that example, you could do
if ( a == b ) Assert.Succeed(); but yes, I can see that a
constraint syntax might be useful for some cases.

Assert.Fail(Constraint);
Assert.Fail(Is.EqualTo(4));

Not clear what you're comparing 4 to here.

Anyway, we have that already. You write it
Assert.That(x, Is.EqualTo(4));

Charlie

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Gary Evans

2008-04-07 12:43:04 UTC

Hi Kelly,

2) As I mentionned in the earlier note, it's not hard> > for someone to implement Theory as an extension using> > the existing extension points. Gary has done that, in fact.> > I'm not really talking about Theory, I'm talking about Assume. It's> great what Gary has done, but without Assume, it's only half the> story.>

Consider this use case of a data test (which is real, I did it> Thursday)... I want to iterate over all the files in a directory, and> validate things about the files if they are the right type and> version. I could do this with your proposed 2.5 like this:> > public static IEnumerable AFiles> {> get> {> DirectoryInfo di = new DirectoryInfo("c:/data");> List<FileInfo> list = new List<FileInfo>();> FileInfo[] files = di.GetFiles("*.*");> foreach(FileInfo fi in files)> {> if (TypeOfFileIsA(fi))> {> object [] parms = new object[1];> parms[0] = fi;> list.Add(parms);> }> }> return list;> }> }> > [DataSource("AFiles")]> public void TestAFiles(object a)> {> FileInfo fi = (FileInfo)a;> // Assert something about the file of type A.> }> > public static IEnumerable BFiles> {> get> {> DirectoryInfo di = new DirectoryInfo("c:/data");> List<FileInfo> list = new List<FileInfo>();> FileInfo[] files = di.GetFiles("*.*");> foreach(FileInfo fi in files)> {> if (TypeOfFileIsB(fi))> {> object [] parms = new object[1];> parms[0] = fi;> list.Add(parms);> }> }> return list;> }> }> > [DataSource("BFiles")]> public void TestBFiles(object a)> {> FileInfo fi = (FileInfo)a;> // Assert something about the file of type B.> }> > ====================================================> Now, if you have Assume, then you can refactor the above code to this:> > public static IEnumerable Files> {> get> {> DirectoryInfo di = new DirectoryInfo("c:/data");> List<FileInfo> list = new List<FileInfo>();> FileInfo[] files = di.GetFiles("*.*");> foreach(FileInfo fi in files)> {> object [] parms = new object[1];> parms[0] = fi;> list.Add(parms);> }> return list;> }> }> > [DataSource("Files")]> public void TestAFiles(object a)> {> FileInfo fi = (FileInfo)a;> Assume.That(TypeOfFileIsA(fi));> // Assert something about the file of type A.> }> > [DataSource("Files")]> public void TestBFiles(object a)> {> FileInfo fi = (FileInfo)a;> Assume.That(TypeOfFileIsB(fi));> // Assert something about the file of type B.> }> > =============================================> Now, I don't know about you, but I like the second version better. And> this is very clearly data testing, not Theory.>

I can't see that this gives us much, we're using exceptions to control program flow, which is slow, and we only save a line of code, i.e we could have

if (!TypeOfFileIsB(fi))
return;

and it may be confusing for the users once they are used to seeing Assumes for Theories? In a theory, Assumptions are powerful as they tell us the cases for which we know our theory is not valid, but putting them in a general test feels strange.
This is only my opinioni and gut-feeling though, it might be that users are totally happy with this? I don't know...

Cheers,
Gary
_________________________________________________________________
Win 100s of Virgin Experience days with BigSnapSearch.com
http://www.bigsnapsearch.com

Charlie Poole

2008-04-07 18:17:23 UTC

Hi Gary,

Post by Charlie Poole
2) As I mentionned in the earlier note, it's not hard
for someone to implement Theory as an extension using
the existing extension points. Gary has done that, in fact.

I'm not really talking about Theory, I'm talking about Assume. It's
great what Gary has done, but without Assume, it's only half the
story.

My extension has assumes. My syntax is a bit worse than Assume.That, but the
functionality's there. It would take you about 5 minutes to make the changes
that you recently proposed to the list, but to do it in my extension; it's
just a case of putting the changes in TheoryMethod instead of
NUnitTestMethod.

That's the idea... If a particular test type understands a certain
exception, then the code to
interpret it belongs in the implementatio of that test type.

It's a little hard to see in 2.5, with the Test hierarchy living in the core
assembly, but
NUnitTestMethod and NUnitTestFixture are not really part of the "core" of
NUnit, they
are extensions like any other, except for the fact that they are added to
the list
of extensions in a different way. In 3.0, they will be much more like other
extensions
and this will be easier to see.

My implementation differs a little bit - your assume was simply a filtering,
but mine catches cases where none of the assumptions were exercised etc.
Nevertheless, it'd still be simple to change it to use Assume.That and catch
an AssumptionException, but all in the extension, and you could change the
behaviour to your preference.

I know you were asking Charlie the question, but I thought I'd chip in my 2p
on this (sorry for the diversion).

It seems worthwhile to me - hopefully to others as well. :-)

I was a bit lazy when I wrote my prototype Theory extension, my assumptions
were just a separate method that was called prior to the theory being
executed, but but I could have written them to work using the Assume.That()
syntax the same as you recently posted to the list. Your change was to be in
the NUnitTestMethod (I think, I'm working off the top of my head) to catch
the AssumptionException (maybe not correct name again).

That's an interesting approach. Way back, I was writing that there might be
more than one way to handle the detection of good data for a theory. It
sounds like you invented a way.

However, We've all derived from NUnitTestMethod in our paramaterized test
extensions, to handle the case of having a test with data, in my theory
extension I called it TheoryMethod (I think). and that would be the ideal
place to put the Assume functionality? i.e. it doesn't need to be in the
base class.

For the record, the lower down the hierarchy you derive from, the better off
you'll be. But of course, you'll have to write more
code, of course. It depends how you view Theory: as a special kind of NUnit
test, or as a sibling. If you derive from NUnitTestMethod,
then you will get whatever behavior it adds later on. That could be a
feature or a bug, depending.

I think how you've written the Assume.That() and the exception handling is
good, but I think that we don't want Assumes to live in NUnitTestMethod -
does it make sense for a user to write an Assume in a vanilla unit test
(i.e. not even a RowTest or a DataSource test?). I haven't looked at the 2.5
sources yet, but I'm assuming that there is a derived class that the RowTest
and DataSource creates, ParameterizedTestMethod? Maybe that class, or a
Theory class deriving from that would be the better place to put this - but
this can be shipped with the Theory extension, and doesn't have to be in the
core/framework of NUnit, unless there's some reporting mechanism missing
(i.e. if we wanted a different way to report assumption failures, if we
bothered putting that functionality in).

In the current code, it's all handled in NUnitTestCaseBuilder and the new
ParameterProviders extension. The actual cases are
still NUnitTestMethods and the suite that holds the multiple tests is a
ParameterizedTestMethodSuite. I think some refactoring will be called for,
but I'd like some other eyes on the code for a bit of review first.

In part, the current structure is driven by my initial tests being for a
single execution of a parameterized test case, which I
represent as a single test, rather than as a suite with one member. It may
be that this "feature" will drop out at some point.

<snipped Kelly's example/>

I can't see that this gives us much, we're using exceptions to control
program flow, which is slow, and we only save a line of code, i.e we could
have

if (!TypeOfFileIsB(fi))
return;

That's how I saw it as well.

and it may be confusing for the users once they are used to seeing Assumes
for Theories? In a theory, Assumptions are powerful as they tell us the
cases for which we know our theory is not valid, but putting them in a
general test feels strange.
This is only my opinioni and gut-feeling though, it might be that users are
totally happy with this? I don't know...

+1

Charlie

Charlie

Cheers,
Gary

_____

News, Sports, Entertainment and Weather on your mobile. Text MSN to
<http://mobile.uk.msn.com/pc/msn_content.aspx> 63463 Now.

Kelly Anderson

2008-04-08 22:26:36 UTC

Post by Charlie Poole
Hi Kelly,

Post by Charlie Poole
2) As I mentionned in the earlier note, it's not hard
for someone to implement Theory as an extension using
the existing extension points. Gary has done that, in fact.

I'm not really talking about Theory, I'm talking about Assume. It's
great what Gary has done, but without Assume, it's only half the
story.

I have no problem with you putting Assume.That into your extension. If
you need assistance with it let me know, but I think it should be
pretty easy to do. If you need, I can email you another copy or
whatever.

Post by Charlie Poole
My implementation differs a little bit - your assume was simply a
filtering, but mine catches cases where none of the assumptions were
exercised etc. Nevertheless, it'd still be simple to change it to use
Assume.That and catch an AssumptionException, but all in the extension, and
you could change the behaviour to your preference.

I think it's important to know when ALL runs of a Theory resulted in
AssumptionExceptions... how does it show up in the UI?

Post by Charlie Poole
I know you were asking Charlie the question, but I thought I'd chip in my
2p on this (sorry for the diversion).

No problem. This is a mailing list, not a private conversation!! :-)

Post by Charlie Poole
I was a bit lazy when I wrote my prototype Theory extension, my assumptions
were just a separate method that was called prior to the theory being
executed, but but I could have written them to work using the Assume.That()
syntax the same as you recently posted to the list. Your change was to be in
the NUnitTestMethod (I think, I'm working off the top of my head) to catch
the AssumptionException (maybe not correct name again).

I think Assume.That works better than what you're describing in that
it keeps the Assumptions closer to the Theory, increasing readability.
My opinion is that readability is one of the more important
contributions Theory makes, so this is important imho.

Post by Charlie Poole
However, We've all derived from NUnitTestMethod in our paramaterized test
extensions, to handle the case of having a test with data, in my theory
extension I called it TheoryMethod (I think). and that would be the ideal
place to put the Assume functionality? i.e. it doesn't need to be in the
base class.

Ok.

Post by Charlie Poole
I think how you've written the Assume.That() and the exception handling is
good, but I think that we don't want Assumes to live in NUnitTestMethod -
does it make sense for a user to write an Assume in a vanilla unit test
(i.e. not even a RowTest or a DataSource test?).

I don't know what you mean here... Assume and Assert are an orthogonal
concern to different Test types. I might be missing something as my
brain is a bit hammered today (dang ole day job anyway).

Post by Charlie Poole
I haven't looked at the 2.5
sources yet, but I'm assuming that there is a derived class that the RowTest
and DataSource creates, ParameterizedTestMethod? Maybe that class, or a
Theory class deriving from that would be the better place to put this - but
this can be shipped with the Theory extension, and doesn't have to be in the
core/framework of NUnit, unless there's some reporting mechanism missing
(i.e. if we wanted a different way to report assumption failures, if we
bothered putting that functionality in).

If you only want Assume to be caught in Theory, then maybe you can
separate it into the separate kind of test. If you want to maintain
orthogonality, then Assume catching has to go into the core. (if I
understand things right). It's the catching of the AssumeException
that needs a new extension point, not the throwing of it.
Specifically, in ProcessException:

protected internal virtual void ProcessException(Exception
exception, TestCaseResult testResult)
{
if (IsAssumeException(exception))
{
// TODO: Perhaps add text to some output stream.
testResult.Success();
return;
}

if (!ExceptionExpected)
{
RecordException(exception, testResult);
return;
}

If there were some sort of extension point where this could call an
external thing something like this:

protected internal virtual void ProcessException(Exception
exception, TestCaseResult testResult)
{
if (ExceptionHandledByExtensionPoint(exception,testResult))
{
return;
}

if (!ExceptionExpected)
{
RecordException(exception, testResult);
return;
}

Then I think we could implement Assume such that it would be
orthogonal to the Test type. I think that would be a good thing...

<example snipped>

Post by Kelly Anderson
=============================================
Now, I don't know about you, but I like the second version better. And
this is very clearly data testing, not Theory.

I can't see that this gives us much, we're using exceptions to control
program flow, which is slow, and we only save a line of code, i.e we could
have
if (!TypeOfFileIsB(fi))
return;

I don't think it's as expressive. That's all. It also reports the test
as having succeeded, rather than just not being applicable, which is a
difference that doesn't jive with Theory... (although that's what my
Assume does too because there isn't a test result type that's really
appropriate to the result yet.)

Post by Charlie Poole
and it may be confusing for the users once they are used to seeing Assumes
for Theories? In a theory, Assumptions are powerful as they tell us the
cases for which we know our theory is not valid, but putting them in a
general test feels strange.
This is only my opinioni and gut-feeling though, it might be that users are
totally happy with this? I don't know...

I think it's a matter of pragmatism. Is it useful to Assume in data
tests, or even regular tests? I think it may be in some rare
conditions. Others may not agree. That's fine, it's just an opinion.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Register now and save $200. Hurry, offer ends at 11:59 p.m.,
Monday, April 7! Use priority code J8TLD2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-09 01:09:14 UTC

Hi Kelly,

Post by Gary Evans
I think how you've written the Assume.That() and the exception
handling is good, but I think that we don't want Assumes to live in
NUnitTestMethod - does it make sense for a user to write an

Assume in

Post by Gary Evans
a vanilla unit test (i.e. not even a RowTest or a DataSource test?).

I don't know what you mean here... Assume and Assert are an
orthogonal concern to different Test types. I might be
missing something as my brain is a bit hammered today (dang
ole day job anyway).

I'm not getting what you mean by "orthogonal concern to different
test types."

Post by Gary Evans
I haven't looked at the 2.5
sources yet, but I'm assuming that there is a derived class

that the

Post by Gary Evans
RowTest and DataSource creates, ParameterizedTestMethod? Maybe that
class, or a Theory class deriving from that would be the

better place

Post by Gary Evans
to put this - but this can be shipped with the Theory

extension, and

Post by Gary Evans
doesn't have to be in the core/framework of NUnit, unless

there's some

Post by Gary Evans
reporting mechanism missing (i.e. if we wanted a different way to
report assumption failures, if we bothered putting that

functionality in).
If you only want Assume to be caught in Theory, then maybe
you can separate it into the separate kind of test. If you
want to maintain orthogonality, then Assume catching has to
go into the core. (if I understand things right). It's the
catching of the AssumeException that needs a new extension
point, not the throwing of it.

IFF you want every test type - now and future - in NUnit to
understand the AssumeException and handle it in the same way,
then you would need to modify the core nunit test runner.

Currently, there are no extension points that modify the
behavior of a test at runtime and designing one would be
a big deal - best deferred to 3.0 IMO.

Current extension points all operate at Load time except
for EventListener, which can't affect test behavior. The
way to impact runtime in this architecture is to create
a special test type at load time, which then does its
thing upon test execution.

Personally, I don't think it's desirable to allow an
extension to affect every conceivable type of test,
whether existing now or written later on. This could
be very confusing to users.

The core of NUnit understands a very small set of
concepts. You can run a test. It gives a result.
The result may be success, failure, or a few other
things. The way to handle this issue within the
core of NUnit is to try to figure out whether we
have a new return type or not and what it is. The
fact that it comes from some exception is the
job of a particular type of test.

Post by Gary Evans
if (!TypeOfFileIsB(fi))
return;

I don't think it's as expressive. That's all. It also reports
the test as having succeeded, rather than just not being
applicable, which is a difference that doesn't jive with
Theory... (although that's what my Assume does too because
there isn't a test result type that's really appropriate to
the result yet.)

My issue with the expressiveness of this is with any implementation
that treats the exception as a success. Assume.That doesn't express
success for me. In an implementation where it did something else,
then I might find it expressive.

Post by Kelly Anderson
I think it's a matter of pragmatism. Is it useful to Assume
in data tests, or even regular tests? I think it may be in
some rare conditions. Others may not agree. That's fine, it's
just an opinion.

In regular tests, your Assume.That would mean something like
Succeed.If. That may be useful. If so we'll add it. In fact,
we already have a request for Assert.Succeed.

Charlie

Kelly Anderson

2008-04-09 02:17:50 UTC

On Tue, Apr 8, 2008 at 7:09 PM, Charlie Poole

Post by Charlie Poole
Hi Kelly,

Assume in

Post by Gary Evans
a vanilla unit test (i.e. not even a RowTest or a DataSource test?).

I don't know what you mean here... Assume and Assert are an
orthogonal concern to different Test types. I might be
missing something as my brain is a bit hammered today (dang
ole day job anyway).

I'm not getting what you mean by "orthogonal concern to different
test types."

What I was thinking was that both Assume and Assert should be handled
by Test, DataTest, IterativeTest, etc., etc. independently of each of
them individually writing code to handle them. But that's not how the
current architecture works, apparently.

Post by Charlie Poole
IFF you want every test type - now and future - in NUnit to
understand the AssumeException and handle it in the same way,
then you would need to modify the core nunit test runner.

That is what I wanted. Apparently not what you want. That is a matter
of opinion, which is fine.

Post by Charlie Poole
Currently, there are no extension points that modify the
behavior of a test at runtime and designing one would be
a big deal - best deferred to 3.0 IMO.

Agreed.

Post by Charlie Poole
Current extension points all operate at Load time except
for EventListener, which can't affect test behavior. The
way to impact runtime in this architecture is to create
a special test type at load time, which then does its
thing upon test execution.

Right, got that now.

Post by Charlie Poole
Personally, I don't think it's desirable to allow an
extension to affect every conceivable type of test,
whether existing now or written later on. This could
be very confusing to users.

Potentially, since we don't know what types of tests there might be.
Is it true that ALL test types understand Assert or not? Do you have
to write code to understand Assert in each TestCase type?

Post by Charlie Poole
The core of NUnit understands a very small set of
concepts. You can run a test. It gives a result.
The result may be success, failure, or a few other
things. The way to handle this issue within the
core of NUnit is to try to figure out whether we
have a new return type or not and what it is. The
fact that it comes from some exception is the
job of a particular type of test.

It would be nice to have another choice of test result types, but
that's the only thing needed for Theory, and you've already talked
about adding it, I think.

Post by Charlie Poole
My issue with the expressiveness of this is with any implementation
that treats the exception as a success. Assume.That doesn't express
success for me. In an implementation where it did something else,
then I might find it expressive.

What else would you have it do? I suppose you could collect together
the set of Theory runs that resulted in Assume being thrown, and
that's a good thing, in fact, I think I left a TODO: in the code I
sent you about that. What other things?

Post by Charlie Poole
In regular tests, your Assume.That would mean something like
Succeed.If. That may be useful. If so we'll add it. In fact,
we already have a request for Assert.Succeed.

I think I would like that. It would also be nice if the other test
result types could be thrown directly. Assert.Ignore() for example?
Maybe that's already in there, I don't recall.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-09 17:42:27 UTC

Hi Kelly,

I don't know what you mean here... Assume and Assert are an >

orthogonal concern to different Test types. I might be > missing
something as my brain is a bit hammered today (dang > ole day job
anyway).
I'm not getting what you mean by "orthogonal concern to different
test types."

What I was thinking was that both Assume and Assert should be
handled by Test, DataTest, IterativeTest, etc., etc.
independently of each of them individually writing code to
handle them. But that's not how the current architecture
works, apparently.

I believe you are thinking of extensions that are intended by
their author to be sub-types of an NUnitTestMethod. But NUnit
supports writing tests that have nothing whatsoever to do with
NUnitTestMethod and it's those types where I think it would
not make sense.

IFF you want every test type - now and future - in NUnit to
understand the AssumeException and handle it in the same way, then
you would need to modify the core nunit test runner.

That is what I wanted. Apparently not what you want. That is
a matter of opinion, which is fine.

This is a fairly large area of discussion. Who should get to
decide whether a particular extension applies to a particular
kind of test? The extension author? The author of the original
test type? The user? IMO, they should all have a say. That's
why I want to move as many such policy decisions out of the
core as I possibly can, leaving a small set of central
concepts.

Take a simple example: Should you be able to extend
RowTest? Maybe yes, maybe no. It depends on whether
RowTest is intended to be a pure emulation of MbUnit's
RowTest or just another way to express data. There isn't
a simple answer that applies to all types of tests - heck,
there may not even be a simple answer for RowTest.

Personally, I don't think it's desirable to allow an extension to
affect every conceivable type of test, whether existing now or
written later on. This could be very confusing to users.

Potentially, since we don't know what types of tests there might be.
Is it true that ALL test types understand Assert or not? Do
you have to write code to understand Assert in each TestCase type?

I assume by "understand Assert" you mean "interpret
an AssertionException as a test failure." The answer
is no. Each type of test has to decide whether it's
tests failed.

We can imagine a test that returned true on success,
false on failure. Such a test class would probably
contain code like
if ( retval )
result.Success();
else
result.Failure();
What's "core" here is the TestResult and its methods.
That's why TestResult is in nunit.core.interfaces.

Post by Kelly Anderson
It would be nice to have another choice of test result types,
but that's the only thing needed for Theory, and you've
already talked about adding it, I think.

There are currently three types defined in the ResultState
enum and, logically speaking, four others in the RunState
enum. I plan to add the "implied" results (Ignored, Skipped,
Explicit, NotRunnable) to ResultState as well. [As an example,
RunState.Igonored means "this test should be ignored" while
ResultState.Ignored meas "this test was ignored." They don't
always match, since you can ignore a test dynamically.]

I see a need for an Inconclusive result state, which may
or may not be enough for Theory - we'll see as we do it.

Post by Kelly Anderson
What else would you have it do? I suppose you could collect
together the set of Theory runs that resulted in Assume being
thrown, and that's a good thing, in fact, I think I left a
TODO: in the code I sent you about that. What other things?

I wasn't recommending - just commenting. But in a Theory
implementation, I think we need some way to indicate that
the data was not suitable for the Theory. Either we plan to
never pass it bad data (in which case Assume should be an error)
or we filter it externally (in which case Assume might not be
needed) or we let the test filter it, in which case we need
to give some indication that says "insufficient evidence."

In the last case, there is a sub-category of possibilities,
depending on whether NUnit calls the Theory once or multiple
times - i.e. once per data point.

There are many ways I can imagine to skin this cat and I
don't want to presume one of them too early, particularly
since it's trivial to experiment with them as addins. If
we were to add Assume.That to NUnit prematurely, we would
be stifling further investigation, IME.

In regular tests, your Assume.That would mean something like
Succeed.If. That may be useful. If so we'll add it. In fact, we
already have a request for Assert.Succeed.

I think I would like that. It would also be nice if the other
test result types could be thrown directly. Assert.Ignore()
for example?
Maybe that's already in there, I don't recall.

In the case of Ignore, it is.

Charlie

Post by Kelly Anderson
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by the 2008 JavaOne(SM)
Conference Don't miss this year's exciting event. There's
still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java
.sun.com/javaone
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer

Charlie Poole

2008-04-08 16:08:56 UTC

Hi Kelly,

Post by Kelly Anderson
I did write it. I guess I integrated it too tightly into the core
though. Perhaps I should take it up with Gary and see if he can
incorporate my Assume into his Theory... if he hasn't done

something similar already.
As I said, I may have missed something here. I knew you had
written some code, but I thought it was only on-the-job and
not available for us to see. If you let me know (again?)
where it is, I'll take a look and maybe that will clarify things.

Ah sorry, now I realize what you are talking about.

You did't do an extension, but a complete change to the NUnit
codebase. We even had some discussion around the best way
to send patches.

I wasn't relating it to this discussion, because
1) I got it on the nunitv3 list
2) It was a change to NUnit itself, not an extension
3) It was late and I wasn't making connections

Or all of the above - take yer cherce. :-)

Anyway, yes, this is indeed "too tightly integrated" since
it modifies some basic stuff in NUnit. More exactly, you
modified another extension - the one that does NUnitTestMethods.

I have been finding this a hard concept to get across: NUnit's
own tests are extensions themselves, and are not intended to
be the only kind of extensions that run on NUnit. That's a
bit hard to see when most current extensions inherit from
NUnit's internal extensions, but that wasn't how it was
supposed to work.

In particular the class NUnitFramework is supposed to hold
the basic behavior of /NUnit/ tests - not of all tests. You'll
see that my csUnit extension has a similar class that holds
it's different behavior. In fact so does your IterativeTestExtension.

So the part you are changing - where you have asked for new
extension points to be added - /is/ an extension. It's not
a place for extension points until we have the ability for
extensions themselves to have extension points - in 3.0.

As Gary has discussed, it would be trivial to put the same
code you wrote into a special test type, where it would
work exactly as it works in the code you sent. Then you
would be sending me a 20K extension rather than 2M of
NUnit source code!

If your feature is intended to be a kind of NUnit test
then you should probably derive the test from NUnitTestMethod,
although that will expose you to future changes in
NUnitTestMethod. It may be better to inherit from TestMethod
or ParameterizedTestMethod - which doesn't yet exist, but
which I'm thinking of facoring out.

Similarly, your builder could inherit from NUnitTestCaseBuilder
if you want to get all its behavior, now and in the future.

Sorry for the confusion - I thought we were talking about
something entirely different.

Charlie

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Register now and save $200. Hurry, offer ends at 11:59 p.m.,
Monday, April 7! Use priority code J8TLD2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Kelly Anderson

2008-04-09 23:40:25 UTC

On Wed, Apr 9, 2008 at 4:46 PM, Charlie Poole

Post by Charlie Poole
Hi Kelly,

That would seem to indicate that you want the Exception types
to be orthogonal to the Test types. That's something worth
thinking about for sure, but I don't pretend to know the
answer at this point.

"Orthogonal" to me would mean that you can decide the two things
independently, not that all test types use the same exceptions.

Fair enough.

Post by Charlie Poole
This really isn't a preference of mine - one way or the other -
it's just a simple fact on the ground. For example, a csUnit
test indicates failure by throwing csunit.SomeExceptionIForgot,
while NUnit test cases throw NUnit.Framework.Exception and
Microsoft throws one with a really long name. Would it make
sense to say that NUnit should "understand" the MS exception?
I don't think so.

No, probably not. But why impose that level of independence upon our
own set of tests? I can see when we are trying to emulate another
testing framework that it needs to be kept independent. But within our
own Test, IterativeTest, etc. why do we need that same level of
independence?

Apparently the new MbUnit has much more orthogonality than
NUnit, or even previous versions of MbUnit. They can combine
all kinds of things in ways that the authors might not have
expected. I'm not entirely sure that's a good thing, as it
*might* lead to results that would be hard to predict (for
the author and the user). Then again, it does give more flexibility.

I'm pretty sure that MbUnit has precisely the degree of orthogonality
that the authors intended - no more no less. Same for NUnit.

Their intention apparently was to have more orthogonality than NUnit does.

Post by Charlie Poole
There is a difference between providing a facility for orthogonal
application of a feature and requiring it. What you are proposing
is to require all future extensions to accept your extension and
I disagree with that philosophically.

I agree with you now that I understand things better.

Right. But they could interpret AssertionException
differently, which would be confusing if you as a user
thought the concepts were orthogonal when they weren't. In
designing this, the principle of least surprise becomes very
important. For me, I never conceived of a tie between
Exception types thrown by Assert or Assume and test types.
As we increase the number of test types, we risk confusing
users if we don't explicitly state that each test type deals
with different exception types differently. This is basic to
how it all works, and I didn't get it until you beat it into
my head here. I can't imagine that a casual user would get
that concept, and now that we have more than one test type,
it becomes a much more important concept to get.
Maybe not quite so much now, because both of the supported
test types do support AssertException, but that just serves
to reinforce my previous idea that they were somehow
orthogonal. Do you see where I'm coming from in usability terms?

Casual users don't deal with the exception mechanism. They deal
with tests failing and succeeding. If I have an assert fail,
and I'm linked with the csUnit dll, it throws a csUnit type
of exception. If I'm linked with the NUnit.Framework dll,
then it throws an nunit type of exception. This is completely
transparent - but it can only be transparent because the test
types are smart about it.

How do I, as a casual user, learn that I can't call Assume.That inside
of a Test? I could assume that it would in a particular way, and it
wouldn't. How do I discover that fact as a casual user?

Post by Charlie Poole
I think the problem here is that you are thinking of extensibility
as applied only to NUnitTestMethods. Within that domain, what you
say is reasonable.

Yes, that's what I was talking about.

Post by Charlie Poole
It's just not the whole domain of NUnit
extensibility. When you say that something should be pushed down
into the core of NUnit, then you are saying it is common to
NUnit, csUnit and VSTS tests today and possibly JUnit, xUnit.net
and others in the future. At that level of generality, there
is commonality, but it's pretty abstract.

Ok, I get that and agree.

Post by Charlie Poole
Because mbUnit has never defined extensibility to include "able
to run tests from other frameworks" they are able to get away
with a simpler implementation. THere is complexity here, but
it isn't accidental - it's rooted in the problem being solved.

So, do we want to create an MbUnit test runner as an extension to NUnit?

Then there would be isomorphism in the two structures.

There really isn't a need for isomorphism. The two structures
should really mean different things - one is on a test and
is a characteristic of the test, the other is on a result
and is a characteristic of a single execution of the test.

Ok.

Post by Charlie Poole
In the last case, there is a sub-category of possibilities,
depending on whether NUnit calls the Theory once or

multiple times -

Post by Charlie Poole
i.e. once per data point.

It doesn't feel very Theoryish to only call it once. :-)

Of course, the test method would be called many times, but
the Theory isn't the test method. I believe the original
implementation simply runs a Theory and gets a single
result.

Ok, that's a new idea. I figured you would want to know the specific
data that made the Theory fail so that you could do something about
it. I'm not sure how to present that if running the Theory is an
atomic operation, but it does make me think about things in a
potentially different way.

Ok, does it stifle further investigation to add Assume.That
to Gary's Theory Addin? I don't think so, but people might
just assume that's the only way to do things.

No. We can have all sorts of addins. I think our discussion
suffers because context matters to me: whether we are talking
about 2.5 or 3.0, whether we are talking about an addin or
the "official" implementation of NUnit. You are more
pragmatic, and it makes it hard for you to see why someone
would care about such things - or that's how it appears.

I would agree that I am pragmatic, and trying to get things done. You
are trying to preserve the integrity of NUnit over the long haul.
These are not entirely compatible ways to look at the world, and that
leads to lively discussion. Neither is bad, just different. I do mix
up the 2.5 and 3.0 stuff fairly freely, and that causes some
confusion. The point though is that if we know where we want 3.0 to
go, it can help in figuring out what can and cannot be put into 2.5.

So why not let people throw all the result kinds directly?
Who knows how useful that could end up being? I don't.

I already said I was implementing the ability to set the
test result to success and probably inconclusive. We don't
want to encourage people to throw exceptions, however,
because that's an implementation detail that could change.

Ok. I'd be interested to hear the alternative you may have in mind.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-10 01:28:16 UTC

Hi Kelly,

Would it make sense to say that NUnit
should "understand" the MS exception?
I don't think so.

No, probably not. But why impose that level of independence
upon our own set of tests? I can see when we are trying to
emulate another testing framework that it needs to be kept
independent. But within our own Test, IterativeTest, etc. why
do we need that same level of independence?

As I see it, having a group of test types that act the same,
implies the ability to add extensions to a single test type,
aka extensions on extensions. Therefore, 3.0 not 2.5.

Being able to do this sort of thing is exactly why I want
to move to a more sophisticated addin architecture rather
than waste time trying to kluge it using our current one.

I'm pretty sure that MbUnit has precisely the degree of

orthogonality

that the authors intended - no more no less. Same for NUnit.

Their intention apparently was to have more orthogonality
than NUnit does.

That's hard to say, since NUnit has never had the data sources
before - orthogonal or not. We agreed that it should be possible
to use any data source with any test in any combination. We now
have that - in 2.5.

There is a difference between providing a facility for orthogonal
application of a feature and requiring it. What you are

proposing is

to require all future extensions to accept your extension and I
disagree with that philosophically.

I agree with you now that I understand things better.

OK.

Post by Kelly Anderson
How do I, as a casual user, learn that I can't call
Assume.That inside of a Test? I could assume that it would in
a particular way, and it wouldn't. How do I discover that
fact as a casual user?

The test would give an error due to the unexpected exception.

Post by Kelly Anderson
When you say

that something should be pushed down into the core of

NUnit, then you

are saying it is common to NUnit, csUnit and VSTS tests today and
possibly JUnit, xUnit.net and others in the future. At

that level of

generality, there is commonality, but it's pretty abstract.

Ok, I get that and agree.

OK.

Because mbUnit has never defined extensibility to include

"able to

run tests from other frameworks" they are able to get away with a
simpler implementation. THere is complexity here, but it isn't
accidental - it's rooted in the problem being solved.

So, do we want to create an MbUnit test runner as an
extension to NUnit?

I think we want to have something that gives the functionality,
but I'm not sure we want to go as far as a test runner. There
are three levels we could do this at:

1) Have attributes and tests that look like MbUnit's
2) Simulate loading of mbUnit tests, execute their actual
asertions and interpret the results.
3) Write some sort of adapter that runs the tests under
the actual mbUnit runner and interpret the resultss.

(1) is how Andreas' RowTest works - as well as MbUnit's
emulation of NUnit.
(2) is how my csUnit and (in process) vsts addins work.
(3) is how Gallio works

For now, I'm sticking to (1) and (2) but we may go farther
in NUnit 3.0.

Of course, the test method would be called many times, but the
Theory isn't the test method. I believe the original

implementation

simply runs a Theory and gets a single result.

Ok, that's a new idea. I figured you would want to know the
specific data that made the Theory fail so that you could do
something about it. I'm not sure how to present that if
running the Theory is an atomic operation, but it does make
me think about things in a potentially different way.

It seems to be how David Saff implemented it and it was my
assumption. The fact that you want more info if it fails
does't work against it. The single test would just return
an array of results.

Post by Kelly Anderson
I would agree that I am pragmatic, and trying to get things
done. You are trying to preserve the integrity of NUnit over
the long haul.

Actually, I'm doing both at different moments. Perhaps its
not always clear which mode I'm in.

Post by Kelly Anderson
These are not entirely compatible ways to look at the world,
and that leads to lively discussion. Neither is bad, just
different. I do mix up the 2.5 and 3.0 stuff fairly freely,
and that causes some confusion. The point though is that if
we know where we want 3.0 to go, it can help in figuring out
what can and cannot be put into 2.5.

Have you ever been a release manager? It's a job that requires
you to be very tough about putting things in that may destabilize
the release. That's where we are now. We have a release close
to going beta, which must go out this month or wait till
mid-summer, due to my schedule. Which do you pick? :-)

So why not let people throw all the result kinds directly?
Who knows how useful that could end up being? I don't.

I already said I was implementing the ability to set the test
result to success and probably inconclusive. We don't want to
encourage people to throw exceptions, however, because that's an
implementation detail that could change.

Ok. I'd be interested to hear the alternative you may have in mind.

"Reporting test results without exceptions" has been on the work plan
for a while. If I knew how to do it the workplan would say something
specific. I'm not above putting something on a plan that I have not
yet figured out how to do.

Charlie

Post by Kelly Anderson
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by the 2008 JavaOne(SM)
Conference Don't miss this year's exciting event. There's
still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java
.sun.com/javaone
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer

Kelly Anderson

2008-04-10 15:41:53 UTC

On Wed, Apr 9, 2008 at 7:28 PM, Charlie Poole

Post by Charlie Poole
Hi Kelly,
As I see it, having a group of test types that act the same,
implies the ability to add extensions to a single test type,
aka extensions on extensions. Therefore, 3.0 not 2.5.
Being able to do this sort of thing is exactly why I want
to move to a more sophisticated addin architecture rather
than waste time trying to kluge it using our current one.

Ok, I think I see your point now.

Post by Charlie Poole
I'm pretty sure that MbUnit has precisely the degree of

orthogonality

Post by Charlie Poole
that the authors intended - no more no less. Same for NUnit.

Their intention apparently was to have more orthogonality
than NUnit does.

The data sources changes the nature of the framework in interesting
ways, for sure. It pushes for orthogonality, but is that always right?
I don't know.

The test would give an error due to the unexpected exception.

That would help, IF the error were specific enough... something like

"Assume is not supported within [Test], it is for use with [Theory] only."

Or something to that extent. I don't like the dependence of Test on
Theory even in a message, but how else do you get them on the right
track quickly?

Post by Kelly Anderson
So, do we want to create an MbUnit test runner as an
extension to NUnit?

I think we want to have something that gives the functionality,
but I'm not sure we want to go as far as a test runner.

If we're doing it for csUnit and MSTest, why would MbUnit be a second
class citizen? Just curious. Is it because it's hard, or confusing, or
what?

Post by Charlie Poole
There
1) Have attributes and tests that look like MbUnit's
2) Simulate loading of mbUnit tests, execute their actual
asertions and interpret the results.
3) Write some sort of adapter that runs the tests under
the actual mbUnit runner and interpret the resultss.

I was thinking along the lines of 3.

Post by Charlie Poole
(1) is how Andreas' RowTest works - as well as MbUnit's
emulation of NUnit.

Kind of weird to have NUnit run MbUnit to simulate running NUnit... :-)

Post by Charlie Poole
(2) is how my csUnit and (in process) vsts addins work.
(3) is how Gallio works
For now, I'm sticking to (1) and (2) but we may go farther
in NUnit 3.0.

I wouldn't expect anything on this line until 3.0. Adding some
data-driven stuff in 2.5 is really good, but not emulation of MbUnit.

Post by Kelly Anderson
Ok, that's a new idea. I figured you would want to know the
specific data that made the Theory fail so that you could do
something about it. I'm not sure how to present that if
running the Theory is an atomic operation, but it does make
me think about things in a potentially different way.

It seems to be how David Saff implemented it and it was my
assumption. The fact that you want more info if it fails
does't work against it. The single test would just return
an array of results.

Yes, it's just a bit different than I was thinking.

Post by Kelly Anderson
I would agree that I am pragmatic, and trying to get things
done. You are trying to preserve the integrity of NUnit over
the long haul.

Actually, I'm doing both at different moments. Perhaps its
not always clear which mode I'm in.

Sometimes. :-)

Let's get it out. :-) Then Gary and I can play around with Theory in
Addin land.

Post by Kelly Anderson
Ok. I'd be interested to hear the alternative you may have in mind.

Fair enough.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-10 19:28:30 UTC

Hi Kelly,

How do I, as a casual user, learn that I can't call >

Assume.That

inside of a Test? I could assume that it would in > a

particular way,

and it wouldn't. How do I discover that > fact as a casual user?
The test would give an error due to the unexpected exception.

That would help, IF the error were specific enough... something like
"Assume is not supported within [Test], it is for use with
[Theory] only."

It would say UnexpectedExcepton: AssumeException with any message
you had provided and a stack trace. The user might have to wonder
about it a bit before realizing that Assume is throwing the
exception. That's a drawback, but I can't think of any other way
to do it without lettin NUnit know that Assume exists.

NUnit doesn't know anything about Assert either, btw.

Post by Kelly Anderson
If we're doing it for csUnit and MSTest, why would MbUnit be
a second class citizen? Just curious. Is it because it's
hard, or confusing, or what?

csUnit and MS Unit tests provide only one type of test. So
emulating them just requires an addin with a new test type.
At least on the surface, mbUnit seems to provide many types
of tests, so it's probably easier to emulate them separately,
rather than to emulate "mbUnit" as a whole.

Additionally, there seems to be no advantage for NUnit in
emulating mbUnit as a framework, as opposed to emulating
the individual fixture types.

There
1) Have attributes and tests that look like MbUnit's
2) Simulate loading of mbUnit tests, execute their actual
asertions and interpret the results.
3) Write some sort of adapter that runs the tests under
the actual mbUnit runner and interpret the resultss.

I was thinking along the lines of 3.

It's certainly possible.

(1) is how Andreas' RowTest works - as well as MbUnit's
emulation of NUnit.

Kind of weird to have NUnit run MbUnit to simulate running
NUnit... :-)

One reaso why emulating All of mbUnit doewsn't make sense to me.

(2) is how my csUnit and (in process) vsts addins work.
(3) is how Gallio works
For now, I'm sticking to (1) and (2) but we may go farther

in NUnit

3.0.

I wouldn't expect anything on this line until 3.0. Adding
some data-driven stuff in 2.5 is really good, but not
emulation of MbUnit.

I may add one or two fixture-type emulations, merely as a way
of exercising the addin facility. Or maybe not - it depends on time.

Post by Kelly Anderson
Let's get it out. :-) Then Gary and I can play around with Theory in
Addin land.

Continue to do it now as well - there may be changes needed in 2.5
to make it work.

Charlie

Post by Kelly Anderson
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by the 2008 JavaOne(SM)
Conference Don't miss this year's exciting event. There's
still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java
.sun.com/javaone
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer

Gary Evans

2008-04-10 15:25:44 UTC

I don't understand that.> > >> > > With a Theory Explorer, you can't predict what the parameters> > > might be, therefore, you would have to have Assume. If you just used> > > if(condition) return; then you wouldn't have sure knowledge> > > that the Theory Explorer had generated cases that reached the> > > code to be truly explored, that is, the code after the> > > Assume.That's... This could lead to bad theories. For> > > example, if I wrote this theory...> >> > In my view, that's not Theory Explorer, it's Theory. I believe> > that the base implementation of Theory should provide for mixing> > up available parameters. The notion of a spearate program to> > suggest additional parameters (Theory Explorer) is an important> > idea and one we may see implemented some day. But a supplier> > of parameters for Theory is not going to be deterministic,> > at least in my view, the way a TestCase is.> > I think it's important when RUNNING a Theory for the run to be> repeatable, thus deterministic (if my definition of deterministic is> the same as yours). When you go exploring the space a Theory covers,> then it's important to be non-deterministic (but report the case(s)> where the Theory fails so that new cases can be added to the Theory> data source.)>

My implementation throws AssumeException, which is not equivalent to> return. My implementation of the catch of AssumeException in the test> case makes it equivalent, but I'm kind of separating Assume.That from> the handling of the exception. If you're talking about how the> exception is handled, then point taken. But I believe throwing the> exception has the right implementation for Theory...>

I disagree about an exception for assumptions necessarily being the right implementation. For a start, there's a performance hit, and secondly, it means that we're going to bomb out of the first assumption - there's no way of seeing whether the data would be invalid for the other assumptions that follow the first failing assumption (I'm not necessarily saying we want that, but exceptions make that impossible).

I agree that having an Assume.That() in the code is a nicer syntax, but you could implement this by having your TheoryMethod set a static flag, then call Invoke, have Assume.That() set the flag and then, if any exception is thrown, it's OK if the flag is set. Of course, having statics means that we can't run the theories in parallel on different threads, so we probably wouldn't want to go down that route, but I'm using the example to say that an exception isn't necessarily the best way to implement the Assume.That() syntax.

Cheers,
Gary
_________________________________________________________________
Win 100s of Virgin Experience days with BigSnapSearch.com
http://www.bigsnapsearch.com

Charlie Poole

2008-04-10 19:09:52 UTC

Hi Gary,

Post by Kelly Anderson
I think it's important when RUNNING a Theory for the run to be
repeatable, thus deterministic (if my definition of deterministic is
the same as yours). When you go exploring the space a Theory covers,
then it's important to be non-deterministic (but report the case(s)
where the Theory fails so that new cases can be added to the Theory
data source.)

I thought that a theory should be deterministic, at least with the user
supplied data. I didn't think it would be as "rigid" as the dynamic test
data. That provides the data in "parameter sets" - i.e. sets of data that we
know work together, one of which is the expected output.
i.e. MyTest(int a, int b, int expected)

whereas I've seen theory syntax expressed (can't remember where) - where you
specify the set of data for each parameter INDEPENDENTLY, and then the tool
exercises the theory with whatever permutations it sees fit.

That's how the original (Java) implementation works. I think, there's a lot
of power in that, even without actively
searching for corner cases. The only .NET implementation I know - except for
yours - is in xunit.net, which
sticks with parameter sets.

MbUnit, OTOH, has both types as part of its data-driven tests - and without
any particular reference to
theory. I think that speaks to mbunit's tester-oriented origins.

Is this what you mean by not being deterministic? I'd expect it to exercise
every possible permutation, but can see how with large sets of data that may
be unfeasible.

There are strategies for maximizing coverage while not using every
permutation - google pairwise testing.

This does show, however, that with even user-supplied data (if you use a
scheme like this) you might want/need assumptions even for user supplied
data and not just a theory explorer - as the assumptions may be valid for
only certain combinations of user input.

Yes.

Post by Kelly Anderson
My implementation throws AssumeException, which is not equivalent to

return. My implementation of the catch of AssumeException in the test
case makes it equivalent, but I'm kind of separating Assume.That from
the handling of the exception. If you're talking about how the
exception is handled, then point taken. But I believe throwing the
exception has the right implementation for Theory...

I disagree about an exception for assumptions necessarily being the right
implementation. For a start, there's a performance hit, and secondly, it
means that we're going to bomb out of the first assumption - there's no way
of seeing whether the data would be invalid for the other assumptions that
follow the first failing assumption (I'm not necessarily saying we want
that, but exceptions make that impossible).

This is a subset of the general question "how do tests report results
without throwing" One way - which may not be possible in
managed code - is to do a two-level return, leaving a result on the stack.
Another is to simply store up results somewhere for
interpretation after the eventual return. I think we need to experiment with
a few options before we'll know for sure.

Charlie

I agree that having an Assume.That() in the code is a nicer syntax, but you
could implement this by having your TheoryMethod set a static flag, then
call Invoke, have Assume.That() set the flag and then, if any exception is
thrown, it's OK if the flag is set. Of course, having statics means that we
can't run the theories in parallel on different threads, so we probably
wouldn't want to go down that route, but I'm using the example to say that
an exception isn't necessarily the best way to implement the Assume.That()
syntax.

Cheers,
Gary

_____

News, Sports, Entertainment and Weather on your mobile. Text MSN to
<http://mobile.uk.msn.com/pc/msn_content.aspx> 63463 Now.

Gary Evans

2008-04-10 21:04:31 UTC

Hi guys,

"Reporting test results without exceptions" has been on the work plan> for a while. If I knew how to do it the workplan would say something> specific. I'm not above putting something on a plan that I have not> yet figured out how to do.>

What sort of results are you after reporting? Currently we either have the case where no exception is thrown, and if the method is decorated with an ExpectedException then there's a failure, and the converse (if an exception is thrown and we don't have an ExpectedException attribute).

I'd like to see the results of multiple asserts, and maybe for an assertion failure report that its failed without breaking out of the test (this may replace a lot of the data-driven testing stuff)

Cheers,
Gary
_________________________________________________________________
Win 100s of Virgin Experience days with BigSnapSearch.com
http://www.bigsnapsearch.com

Charlie Poole

2008-04-10 21:10:03 UTC

Post by Charlie Poole
"Reporting test results without exceptions" has been on the work plan
for a while. If I knew how to do it the workplan would say something
specific. I'm not above putting something on a plan that I have not
yet figured out how to do.

What sort of results are you after reporting? Currently we either have the
case where no exception is thrown, and if the method is decorated with an
ExpectedException then there's a failure, and the converse (if an exception
is thrown and we don't have an ExpectedException attribute).

I'd like to see the results of multiple asserts, and maybe for an assertion
failure report that its failed without breaking out of the test (this may
replace a lot of the data-driven testing stuff)

Cheers,
Gary

_____

News, Sports, Entertainment and Weather on your mobile. Text MSN to
<http://mobile.uk.msn.com/pc/msn_content.aspx> 63463 Now.

Gary Evans

2008-04-10 21:29:55 UTC

Hi Gary,

For example, you write
Assert.That( 2 + 2 , Is.EqualTo( 5) );
I'd like the test result that says "Expected: 4 But was: 5" to go back to NUnit
without throwing an AssertionException.

Why you ask?

Because then, if you wrote
Assert.That( 2+2, Is.EqualTo( 4) );
I could give NUnit that successful result, and continue with the test.

Reporting of successful asserts is a long-standing request and reporting
failure via an exception seems wrong to me, since failures are expected
in a test framework.

That would not change how other exceptions are reported, however.

Any ideas?

Charlie

I like this :) it is a bit limiting that an assertion failure prevents the rest of the test from running, in some cases. I say some cases 'cause a lot of tests are written at the minute that rely on this functionality.

If we're in a loop checking inputs against expected output, you might be interested in all of the loop results - whether successful or not. But in other tests a failure at the top of a test may mean that there's not much point carrying on.

But then again, maybe Assert.That should return the result of the condition, so you could have

if (!Assert.That(myCondition))
{
// there's not much point continuing the test
return;
}

Cheers,
Gary
_________________________________________________________________
Amazing prizes every hour with Live Search Big Snap
http://www.bigsnapsearch.com

Charlie Poole

2008-04-10 22:56:10 UTC

Hi Gary,

I like this :) it is a bit limiting that an assertion failure prevents the
rest of the test from running, in some cases. I say some cases 'cause a lot
of tests are written at the minute that rely on this functionality.

If we're in a loop checking inputs against expected output, you might be
interested in all of the loop results - whether successful or not. But in
other tests a failure at the top of a test may mean that there's not much
point carrying on.

But then again, maybe Assert.That should return the result of the condition,
so you could have

if (!Assert.That(myCondition))
{
// there's not much point continuing the test
return;
}

That puts it in the user's hands, which is good, but requires an if test.

My thought was to have

1) a way to report ongoing progress without returning

2) a way to return "twice" from within an Assert or other call

The first is fairly easy - some sort of list of intermediate results that
is maintained in the framework and accessed from the core via
reflection - it could use only built-in .NET types to avoid any
versioning issues.

The second is either hard or impossible. :-) You can do it
by stack manipulation in unmanaged code, but I think it
would require a .NET facility to make it happen in
managed code.

Charlie

Kelly Anderson

2008-04-11 00:26:24 UTC

On Thu, Apr 10, 2008 at 3:10 PM, Charlie Poole

Post by Charlie Poole
Hi Gary,
For example, you write
Assert.That( 2 + 2 , Is.EqualTo( 5) );
I'd like the test result that says "Expected: 4 But was: 5" to go back to
NUnit
without throwing an AssertionException.
Why you ask?
Assert.That( 2+2, Is.EqualTo( 4) );
I could give NUnit that successful result, and continue with the test.
Reporting of successful asserts is a long-standing request and reporting
failure via an exception seems wrong to me, since failures are expected
in a test framework.
That would not change how other exceptions are reported, however.
Any ideas?

Assert could be implemented such that if it were successful, it could
append the successful assertion to an output reporting mechanism,
while still throwing exceptions for failure. You already have to catch
other exceptions anyway as you say.

I think it is very bad to continue execution after an Assertion has
failed. That changes what assert has meant for thirty plus years in
the C/C++/C# world. That would be like putting a game boy console in
place of the steering wheel, accelerator and breaks of a car IMHO.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Charlie Poole

2008-04-11 03:03:09 UTC

Hi Kelly,

Post by Kelly Anderson
Assert could be implemented such that if it were successful,
it could append the successful assertion to an output
reporting mechanism, while still throwing exceptions for
failure. You already have to catch other exceptions anyway as you say.

Sounds like a possibility.

Post by Kelly Anderson
I think it is very bad to continue execution after an
Assertion has failed. That changes what assert has meant for
thirty plus years in the C/C++/C# world. That would be like
putting a game boy console in place of the steering wheel,
accelerator and breaks of a car IMHO.

Not sure where you got that. There are no plans to continue
after a failure.

Charlie

Post by Kelly Anderson
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by the 2008 JavaOne(SM)
Conference Don't miss this year's exciting event. There's
still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java
.sun.com/javaone
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer

Kelly Anderson

2008-04-12 01:24:54 UTC

On Thu, Apr 10, 2008 at 9:03 PM, Charlie Poole

Post by Charlie Poole
Hi Kelly,

Sounds like a possibility.

I would think you would want a separate "stream" for successful
Asserts from unsuccessful Asserts anyway.

Not sure where you got that. There are no plans to continue
after a failure.

That's what it sounded like Gary was suggesting. I think it would be a bad idea.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Erlis Vidal

2008-04-11 17:09:37 UTC

Hi everyone,

I'm reading your posts and I would like to give you this idea, because maybe
this could help.

If we can "disable" the break on failing behavior within a method call, and
then the option for "enable" it again, maybe this could help. It's the
programmer's choice. This could help in the case I'm testing in a loop the
inputs values and the expected results. With some mechanism to retrieve the
results (failing or success) after re-enable the default behavior you could
have what you are looking for. Maybe the "enable" method could return a kind
of report class. If the code never was "enabled" then NUnit print the report
class at the end of the execution.

I hope I can help you guys; I really appreciate your job.

Sincerely

Erlis

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Charlie
Poole
Sent: Thursday, April 10, 2008 5:10 PM
To: 'Gary Evans'; nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi Gary,

For example, you write

Assert.That( 2 + 2 , Is.EqualTo( 5) );

I'd like the test result that says "Expected: 4 But was: 5" to go back to
NUnit

without throwing an AssertionException.

Why you ask?

Because then, if you wrote

Assert.That( 2+2, Is.EqualTo( 4) );

I could give NUnit that successful result, and continue with the test.

Reporting of successful asserts is a long-standing request and reporting

failure via an exception seems wrong to me, since failures are expected

in a test framework.

That would not change how other exceptions are reported, however.

Any ideas?

Charlie

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Gary
Evans
Sent: Thursday, April 10, 2008 2:05 PM
To: nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi guys,

What sort of results are you after reporting? Currently we either have the
case where no exception is thrown, and if the method is decorated with an
ExpectedException then there's a failure, and the converse (if an exception
is thrown and we don't have an ExpectedException attribute).

I'd like to see the results of multiple asserts, and maybe for an assertion
failure report that its failed without breaking out of the test (this may
replace a lot of the data-driven testing stuff)

Cheers,
Gary

_____

News, Sports, Entertainment and Weather on your mobile. Text MSN to
<http://mobile.uk.msn.com/pc/msn_content.aspx> 63463 Now.

Charlie Poole

2008-04-11 18:36:21 UTC

Hi Erlis,

That sounds like a possible option. As you probably know, with 2.5, you can
simply provide the data
to the test via arguments and attributes and NUnit will create multiple
calls to your method. However,
there are other cases where looping in the test makes the most sense.

Charlie

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Erlis
Vidal
Sent: Friday, April 11, 2008 10:10 AM
To: nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi everyone,

I'm reading your posts and I would like to give you this idea, because maybe
this could help.

If we can "disable" the break on failing behavior within a method call, and
then the option for "enable" it again, maybe this could help. It's the
programmer's choice. This could help in the case I'm testing in a loop the
inputs values and the expected results. With some mechanism to retrieve the
results (failing or success) after re-enable the default behavior you could
have what you are looking for. Maybe the "enable" method could return a kind
of report class. If the code never was "enabled" then NUnit print the report
class at the end of the execution.

I hope I can help you guys; I really appreciate your job.

Sincerely

Erlis

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Charlie
Poole
Sent: Thursday, April 10, 2008 5:10 PM
To: 'Gary Evans'; nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi Gary,

For example, you write

Assert.That( 2 + 2 , Is.EqualTo( 5) );

I'd like the test result that says "Expected: 4 But was: 5" to go back to
NUnit

without throwing an AssertionException.

Why you ask?

Because then, if you wrote

Assert.That( 2+2, Is.EqualTo( 4) );

I could give NUnit that successful result, and continue with the test.

Reporting of successful asserts is a long-standing request and reporting

failure via an exception seems wrong to me, since failures are expected

in a test framework.

That would not change how other exceptions are reported, however.

Any ideas?

Charlie

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Gary
Evans
Sent: Thursday, April 10, 2008 2:05 PM
To: nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi guys,

What sort of results are you after reporting? Currently we either have the
case where no exception is thrown, and if the method is decorated with an
ExpectedException then there's a failure, and the converse (if an exception
is thrown and we don't have an ExpectedException attribute).

I'd like to see the results of multiple asserts, and maybe for an assertion
failure report that its failed without breaking out of the test (this may
replace a lot of the data-driven testing stuff)

Cheers,
Gary

_____

News, Sports, Entertainment and Weather on your mobile. Text MSN to
<http://mobile.uk.msn.com/pc/msn_content.aspx> 63463 Now.

Jeff Brown

2008-04-11 19:13:17 UTC

I call this feature Assert.Multiple. It creates a context within which
multiple assertion failures are allowed.
When the context is exited, all asserts are reported together.

This allows you to glue together many asserts that are really about
verifying some more complex state piecewise.

Jeff.

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Erlis
Vidal
Sent: Friday, April 11, 2008 10:10 AM
To: nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi everyone,

I'm reading your posts and I would like to give you this idea, because maybe
this could help.

If we can "disable" the break on failing behavior within a method call, and
then the option for "enable" it again, maybe this could help. It's the
programmer's choice. This could help in the case I'm testing in a loop the
inputs values and the expected results. With some mechanism to retrieve the
results (failing or success) after re-enable the default behavior you could
have what you are looking for. Maybe the "enable" method could return a kind
of report class. If the code never was "enabled" then NUnit print the report
class at the end of the execution.

I hope I can help you guys; I really appreciate your job.

Sincerely

Erlis

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Charlie
Poole
Sent: Thursday, April 10, 2008 5:10 PM
To: 'Gary Evans'; nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi Gary,

For example, you write

Assert.That( 2 + 2 , Is.EqualTo( 5) );

I'd like the test result that says "Expected: 4 But was: 5" to go back to
NUnit

without throwing an AssertionException.

Why you ask?

Because then, if you wrote

Assert.That( 2+2, Is.EqualTo( 4) );

I could give NUnit that successful result, and continue with the test.

Reporting of successful asserts is a long-standing request and reporting

failure via an exception seems wrong to me, since failures are expected

in a test framework.

That would not change how other exceptions are reported, however.

Any ideas?

Charlie

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Gary
Evans
Sent: Thursday, April 10, 2008 2:05 PM
To: nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi guys,

What sort of results are you after reporting? Currently we either have the
case where no exception is thrown, and if the method is decorated with an
ExpectedException then there's a failure, and the converse (if an exception
is thrown and we don't have an ExpectedException attribute).

I'd like to see the results of multiple asserts, and maybe for an assertion
failure report that its failed without breaking out of the test (this may
replace a lot of the data-driven testing stuff)

Cheers,
Gary

_____

News, Sports, Entertainment and Weather on your mobile. Text MSN to 63463
Now. <http://mobile.uk.msn.com/pc/msn_content.aspx>

Jeff Brown

2008-04-11 19:48:05 UTC

You can capture a stack trace at will using the
System.Diagnostics.StackTrace class.

The trick will be filtering and pruning the trace down so that showing
traces for multiple asserts does not get out of hand. The traces should
effectively be scoped within the context of the Assert.Multiple as if a
catch block appeared there (but it does not).

In principle, we could just cut the trace down to one line: the location of
the assert that failed within the Assert.Multiple block.

As for syntax, we have a few options such as:

using (Assert.Multiple)
{
// more asserts
}

Assert.Multiple(delegate
{
// more asserts
}

(or the same thing using the lamdba syntax)

In terms of actual implementation, there's a hack for this feature in MbUnit
v2 that a couple of people seem to be using now. We will be adding the
feature more formally in MbUnit v3 soon.

I started messing around with this idea a year or two ago along with a few
others to assist with grouping related asserts anhd providing more
contextually relevant information on the whole.

Jeff.

_____

From: Vaughn, Clifton [mailto:***@bestbuy.com]
Sent: Friday, April 11, 2008 12:30 PM
To: Jeff Brown; nunit-***@lists.sourceforge.net
Subject: RE: [nunit-developer] NUnit 2.5: What's In It Now

The challenge with this approach, and I'm all for it btw, is that you lose
the context information (stack trace) so you can go to the offending line of
code. Unless of course your assert.Multiple just caught the exception and
then outputted it so that it can be used in VS or other tool.

Thanks

cliff

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Jeff
Brown
Sent: Friday, April 11, 2008 2:13 PM
To: nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

I call this feature Assert.Multiple. It creates a context within which
multiple assertion failures are allowed.

When the context is exited, all asserts are reported together.

This allows you to glue together many asserts that are really about
verifying some more complex state piecewise.

Jeff.

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Erlis
Vidal
Sent: Friday, April 11, 2008 10:10 AM
To: nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi everyone,

I'm reading your posts and I would like to give you this idea, because maybe
this could help.

If we can "disable" the break on failing behavior within a method call, and
then the option for "enable" it again, maybe this could help. It's the
programmer's choice. This could help in the case I'm testing in a loop the
inputs values and the expected results. With some mechanism to retrieve the
results (failing or success) after re-enable the default behavior you could
have what you are looking for. Maybe the "enable" method could return a kind
of report class. If the code never was "enabled" then NUnit print the report
class at the end of the execution.

I hope I can help you guys; I really appreciate your job.

Sincerely

Erlis

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Charlie
Poole
Sent: Thursday, April 10, 2008 5:10 PM
To: 'Gary Evans'; nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi Gary,

For example, you write

Assert.That( 2 + 2 , Is.EqualTo( 5) );

I'd like the test result that says "Expected: 4 But was: 5" to go back to
NUnit

without throwing an AssertionException.

Why you ask?

Because then, if you wrote

Assert.That( 2+2, Is.EqualTo( 4) );

I could give NUnit that successful result, and continue with the test.

Reporting of successful asserts is a long-standing request and reporting

failure via an exception seems wrong to me, since failures are expected

in a test framework.

That would not change how other exceptions are reported, however.

Any ideas?

Charlie

_____

From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Gary
Evans
Sent: Thursday, April 10, 2008 2:05 PM
To: nunit-***@lists.sourceforge.net
Subject: Re: [nunit-developer] NUnit 2.5: What's In It Now

Hi guys,

What sort of results are you after reporting? Currently we either have the
case where no exception is thrown, and if the method is decorated with an
ExpectedException then there's a failure, and the converse (if an exception
is thrown and we don't have an ExpectedException attribute).

I'd like to see the results of multiple asserts, and maybe for an assertion
failure report that its failed without breaking out of the test (this may
replace a lot of the data-driven testing stuff)

Cheers,
Gary

_____

News, Sports, Entertainment and Weather on your mobile. Text MSN to 63463
Now. <http://mobile.uk.msn.com/pc/msn_content.aspx>

Gary Evans

2008-04-10 21:25:04 UTC

return from the middle of a function other than an exception> > > (that I can think of) so I don't know how else you would> > > implement something like assume in the middle of the code.> >> > I'm researching that - it's part of figuring out how to> > communicate results without exceptions.> > I suppose you could call a function that somehow didn't return to the> caller... not sure how though.>

I've been reading up a little bit on continuations (read this post: http://blogs.msdn.com/wesdyer/archive/2007/12/22/continuation-passing-style.aspx)

I'm still trying to make sense of this, I don't know whether F# would let us do more funky stuff like this?

As an aside - with theories, do we want our assumptions to return us immediately from the function (i.e. filtering), or flag that the theory code is expected to fail and then actually execute the code to check that it fails (i.e. validate the assumption).

I have given up on that request completely. I see your point now that> I understand the framework and context better. You are right. I don't> necessarily want Assume to work for Test anymore, or even> IterativeTest. I would like it for Theory still, and will work with> Gary to see that it gets into his extension.>

I like the Assume.That() syntax, but implementing it using exceptions will reduce the functionality currently in my extension. Maybe I could use a ThreadStatic flag?

The proximity of the Assumes to the Asserts seems fundamental> > > to me in terms of expressiveness. Does that make sense to you> > > in the context of Theory?> >> > I like that aspect of David's implementation and it was my> > first choice of how to go. But I won't jump from that to> > the notion that no other way of doing it is possible, or> > maybe even better. Gary has implemented a different approach,> > for example. I want to write a non-trivial program using his> > implementation. Then I want to write my own. Then I feel like> > I'll be qualified to have a preference.> > Ok, that works for me.>

I think one of the weaknesses of my approach is that the Assumes aren't inline with the code, but if we want to also think of theory explorers, we may want to "advertise" our assumptions in a different way, so that the tools can build up the datasets more intelligently?

Probably. But it's not the main point IMO. As a user, I> > don't throw an exception. I just Assume something. You're> > saying "As a user, I want to be able to make an assumption> > and have it verified. If it succeeds, the test continues.> > If it fails, the test terminates and succeeds." This> > doesn't make a lot of sense.> > Why not? That's what Assume means in all the literature. The only> variant is that if the test terminates in ALL cases of testing the> Theory, then it is not a success.>

On my blog posts, I was saying that we would want to verify assumes (see my paragraph/reply a bit above to see what I mean). This isn't mentioned in the literature (that I can remember), and I'm not sure how useful this is in real life, but the idea of my extension was to keep all options open (well, put more functionality like this in, which could be removed later if it doesn't prove to be useful, rather than limit the functionality too soon - that's why I called it a prototype theory implementation).
Cheers,
Gary
_________________________________________________________________
Welcome to the next generation of Windows Live
http://www.windowslive.co.uk/get-live

Charlie Poole

2008-04-10 23:21:10 UTC

Hi Gary,

I've been reading up a little bit on continuations (read this post:
http://blogs.msdn.com/wesdyer/archive/2007/12/22/continuation-passing-style.
aspx)

It's cool stuff, but I'm not sure it helps us since it would require the
user - at least as I understand it - to
use the funky syntax rather than hiding it from him.

I'm still trying to make sense of this, I don't know whether F# would let us
do more funky stuff like this?

As an aside - with theories, do we want our assumptions to return us
immediately from the function (i.e. filtering), or flag that the theory code
is expected to fail and then actually execute the code to check that it
fails (i.e. validate the assumption).

I don't think failing an assumption implies that the test would fail if run.
It could just
be outside the scope of the theory.

Post by Kelly Anderson
I have given up on that request completely. I see your point now that
I understand the framework and context better. You are right. I don't
necessarily want Assume to work for Test anymore, or even
IterativeTest. I would like it for Theory still, and will work with
Gary to see that it gets into his extension.

I like the Assume.That() syntax, but implementing it using exceptions will
reduce the functionality currently in my extension. Maybe I could use a
ThreadStatic flag?

How would that work? That is, who would set it and who would honor it?

Post by Kelly Anderson
The proximity of the Assumes to the Asserts seems fundamental
to me in terms of expressiveness. Does that make sense to you
in the context of Theory?

Ok, that works for me.

I think one of the weaknesses of my approach is that the Assumes aren't
inline with the code, but if we want to also think of theory explorers, we
may want to "advertise" our assumptions in a different way, so that the
tools can build up the datasets more intelligently?

Interesting idea.

Post by Charlie Poole
Probably. But it's not the main point IMO. As a user, I
don't throw an exception. I just Assume something. You're
saying "As a user, I want to be able to make an assumption
and have it verified. If it succeeds, the test continues.
If it fails, the test terminates and succeeds." This
doesn't make a lot of sense.

Why not? That's what Assume means in all the literature. The only
variant is that if the test terminates in ALL cases of testing the
Theory, then it is not a success.

On my blog posts, I was saying that we would want to verify assumes (see my
paragraph/reply a bit above to see what I mean). This isn't mentioned in the
literature (that I can remember), and I'm not sure how useful this is in
real life, but the idea of my extension was to keep all options open (well,
put more functionality like this in, which could be removed later if it
doesn't prove to be useful, rather than limit the functionality too soon -
that's why I called it a prototype theory implementation).

That's the diff between an experiment and a release - or so I've learned -
if you put functionality
into a release, you're pretty much stuck with it. :-)

Charlie

Cheers,
Gary

_____

Have you played Fishticuffs? Get fish-slapping on Messenger
<http://www.fishticuffs.co.uk>

Kelly Anderson

2008-04-11 01:50:22 UTC