Discussion:
Data-driven Tests in NUnit 2.5
Charlie Poole
2008-03-17 01:02:49 UTC
Permalink
Hi All,

We've had much discussion on the need for parameterized
or data-driven tests in 2.5. I thought I'd outline here
some thoughts about where I see it going.

[NOTE: This is separate from the discussion of Theories,
which is taking place on NUnitV3 currently. In fact, I'm
writing this to put down my thoughts about data-driven
tests before I jump into the Theories disussion.]
From all that has been said and my own research and
experimentation, I think there is a need for two
types of data-driven tests:

1. Those whose data arguments are specified right in
the metadata through a custom attribute.

2. Those whose data is provided separately - from a
file, an array or collection or an iterator.

For purposes of this discusssion, I'm calling the first
category "static" data, the second "dynamic." Bear in
mind it's actually all dynamic, but what I mean by
"static" in this context is that the actual values
used are contained in the metadata. For "dynamic"
data, the metadata only contains instructions for
how to get the data - a method name for example.

For static data, NUnit can construct separate
test cases at load time for each set of arguments.
For dynamic data, this would be possible with some
changes, but I'm making a design decision not to
do it at this point.

That means that test cases created with static data
will show up in the tree of tests individually.
Those constructed with dynamic data will not show
up separately, but will produce separate results
when the tests are run. Tests are already capable
of returning multiple results in NUnit but the
Gui will need to be modified to display them.

Internally, NUnit will support an extension point
for providing data to tests. The extension will
return one of two things to NUnit:

1) An actual data source, if the data is static.
2) An object for use at runtime to get the actual
data source, if the data is dynamic.

The actual interfaces involved still need to be
worked out.

For static test methods, I'm thinking of supporting
the following natively:

[TestFixture]
public class StaticDataSample
{
[TestCase( 1000, 10, 100.0000)]
[TestCase(-1000, 10, -100.0000)]
[TestCase( 1000, 7, 142.85715)]
[TestCase( 1000, 0.00001, 100000000)]
[TestCase(4195835, 3145729, 1.3338196)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator / denominator, 0.00001);
}
}

If you're familiar with RowTest, you'll see that this is
essentially the same thing, with TestCase substituted for Row.
In fact I took the example from Andreas and I think we can
incorporate his code right into NUnit.

For dynamic data, I'm thinking of supporting something
like this...

[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator / denominator, 0.00001);
}
}

Where MyDataSource is a class provided by the user. I'm
still working on the details of this, so please throw in
your ideas. I'd like to have a typesafe interface for
data, rather than using a string.

More to come on this - your comments are welcome.

Charlie



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Charlie Poole
2008-03-17 01:08:46 UTC
Permalink
I left out one thing - a question.

If a fixture contains testcases like this...

[TestCase( 1000, 10, 100.0000)]
[TestCase(-1000, 10, -100.0000)]
[TestCase( 1000, 7, 142.85715)]
[TestCase( 1000, 0.00001, 100000000)]
[TestCase(4195835, 3145729, 1.3338196)]

... it would be possible to have the five test
cases in the tree directly under the fixture or
to group them in a further level named for the
method as RowTest does.

Which do you prefer for this case?

Charlie
-----Original Message-----
Sent: Sunday, March 16, 2008 6:03 PM
Subject: Data-driven Tests in NUnit 2.5
Hi All,
We've had much discussion on the need for parameterized or
data-driven tests in 2.5. I thought I'd outline here some
thoughts about where I see it going.
[NOTE: This is separate from the discussion of Theories,
which is taking place on NUnitV3 currently. In fact, I'm
writing this to put down my thoughts about data-driven tests
before I jump into the Theories disussion.]
From all that has been said and my own research and
experimentation, I think there is a need for two types of
1. Those whose data arguments are specified right in the
metadata through a custom attribute.
2. Those whose data is provided separately - from a file, an
array or collection or an iterator.
For purposes of this discusssion, I'm calling the first
category "static" data, the second "dynamic." Bear in mind
it's actually all dynamic, but what I mean by "static" in
this context is that the actual values used are contained in
the metadata. For "dynamic"
data, the metadata only contains instructions for how to get
the data - a method name for example.
For static data, NUnit can construct separate test cases at
load time for each set of arguments.
For dynamic data, this would be possible with some changes,
but I'm making a design decision not to do it at this point.
That means that test cases created with static data will show
up in the tree of tests individually.
Those constructed with dynamic data will not show up
separately, but will produce separate results when the tests
are run. Tests are already capable of returning multiple
results in NUnit but the Gui will need to be modified to display them.
Internally, NUnit will support an extension point for
providing data to tests. The extension will return one of two
1) An actual data source, if the data is static.
2) An object for use at runtime to get the actual
data source, if the data is dynamic.
The actual interfaces involved still need to be worked out.
For static test methods, I'm thinking of supporting the
[TestFixture]
public class StaticDataSample
{
[TestCase( 1000, 10, 100.0000)]
[TestCase(-1000, 10, -100.0000)]
[TestCase( 1000, 7, 142.85715)]
[TestCase( 1000, 0.00001, 100000000)]
[TestCase(4195835, 3145729, 1.3338196)]> public void
DivisionTest(double numerator, double
denominator, double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
}
}
If you're familiar with RowTest, you'll see that this is
essentially the same thing, with TestCase substituted for Row.
In fact I took the example from Andreas and I think we can
incorporate his code right into NUnit.
For dynamic data, I'm thinking of supporting something like this...
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double
denominator, double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
}
}
Where MyDataSource is a class provided by the user. I'm still
working on the details of this, so please throw in your
ideas. I'd like to have a typesafe interface for data, rather
than using a string.
More to come on this - your comments are welcome.
Charlie
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Kelly Anderson
2008-03-17 21:24:16 UTC
Permalink
On Sun, Mar 16, 2008 at 7:02 PM, Charlie Poole
Post by Charlie Poole
For dynamic data, this would be possible with some
changes, but I'm making a design decision not to
do it at this point.
Having looked at the code, I think this is the better part of valor...
as it would require a pretty major restructuring of things.
Post by Charlie Poole
That means that test cases created with static data
will show up in the tree of tests individually.
Those constructed with dynamic data will not show
up separately, but will produce separate results
when the tests are run. Tests are already capable
of returning multiple results in NUnit but the
Gui will need to be modified to display them.
I would assume that there will also have to be some sort of
notification that the GUI can pick up...
Post by Charlie Poole
Internally, NUnit will support an extension point
for providing data to tests. The extension will
1) An actual data source, if the data is static.
2) An object for use at runtime to get the actual
data source, if the data is dynamic.
Sure, that could work.
Post by Charlie Poole
The actual interfaces involved still need to be
worked out.
Yeah.. :-)
Post by Charlie Poole
For static test methods, I'm thinking of supporting
[TestFixture]
public class StaticDataSample
{
[TestCase( 1000, 10, 100.0000)]
[TestCase(-1000, 10, -100.0000)]
[TestCase( 1000, 7, 142.85715)]
[TestCase( 1000, 0.00001, 100000000)]
[TestCase(4195835, 3145729, 1.3338196)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator / denominator, 0.00001);
}
}
Why not [Test(1000, 7, 142.85715)] ??? Is there a naming conflict?
Post by Charlie Poole
For dynamic data, I'm thinking of supporting something
like this...
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator / denominator, 0.00001);
}
}
Where MyDataSource is a class provided by the user. I'm
still working on the details of this, so please throw in
your ideas. I'd like to have a typesafe interface for
data, rather than using a string.
I see where you're trying to go with the type safe thing here. That
makes some sense... but don't you still have a problem extracing
numerator, denominator and result from MyDataSource? That part would
want to be type safe too, wouldn't it?
Post by Charlie Poole
More to come on this - your comments are welcome.
I'll give some thought to a type safe way to do this... do you have
anything against using generics? Or would the fact that they aren't
supported in 1.1 keep you off that track? Could the tests be written
in 2.0 code to test against 1.1 code?

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Charlie Poole
2008-03-19 14:37:52 UTC
Permalink
Hi Kelly,
Post by Charlie Poole
That means that test cases created with static data will
show up in
the tree of tests individually.
Those constructed with dynamic data will not show up
separately, but
will produce separate results when the tests are run. Tests are
already capable of returning multiple results in NUnit but
the Gui
will need to be modified to display them.
I would assume that there will also have to be some sort of
notification that the GUI can pick up...
The existing events will work but may need to carry some extra information.
Post by Charlie Poole
For static test methods, I'm thinking of supporting the following
[TestFixture]
public class StaticDataSample
{
[TestCase( 1000, 10, 100.0000)]
[TestCase(-1000, 10, -100.0000)]
[TestCase( 1000, 7, 142.85715)]
[TestCase( 1000, 0.00001, 100000000)]
[TestCase(4195835, 3145729, 1.3338196)]
public void DivisionTest(double numerator, double
denominator, double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
}
}
Why not [Test(1000, 7, 142.85715)] ??? Is there a naming conflict?
1) Test already means a lot of things in NUnit. TestCases, TestSuites
TestFixtures, etc. are all Tests.

2) A new name for a new thing

3) Not breaking compatibility with old apps

4) Test already takes an argument: the description.
Post by Charlie Poole
For dynamic data, I'm thinking of supporting something
like this...
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double
denominator, double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
}
}
Where MyDataSource is a class provided by the user. I'm still
working on the details of this, so please throw in your ideas. I'd
like to have a typesafe interface for data, rather than using a
string.
I see where you're trying to go with the type safe thing
here. That makes some sense... but don't you still have a
problem extracing numerator, denominator and result from
MyDataSource? That part would want to be type safe too, wouldn't it?
The interface can be type safe, the actual returned values need to
be tested for a match to the argument types dynamically. That's no
big deal since large parts of NUnit already work this way.
Post by Charlie Poole
More to come on this - your comments are welcome.
I'll give some thought to a type safe way to do this... do
you have anything against using generics? Or would the fact
that they aren't supported in 1.1 keep you off that track?
No, generics are a graeat approach for the 2.0 version of the runners.
Post by Charlie Poole
Could the tests be written in 2.0 code to test against 1.1 code?
Only to test them under 2.0. That may be OK for some purposes though.


Charlie
Post by Charlie Poole
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by: Microsoft Defy all
challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Kelly Anderson
2008-03-19 17:39:28 UTC
Permalink
On Wed, Mar 19, 2008 at 8:37 AM, Charlie Poole
Post by Charlie Poole
Post by Kelly Anderson
I would assume that there will also have to be some sort of
notification that the GUI can pick up...
The existing events will work but may need to carry some extra information.
Sure. That should work fine.
Post by Charlie Poole
Post by Kelly Anderson
Why not [Test(1000, 7, 142.85715)] ??? Is there a naming conflict?
1) Test already means a lot of things in NUnit. TestCases, TestSuites
TestFixtures, etc. are all Tests.
Ok, but they aren't all TestAttributes, are they? TestFixtureAttribute
doesn't inherit from TestAttribute, right? Or are you just speaking
semantically here...
Post by Charlie Poole
2) A new name for a new thing
Kind of... more like an old thing doing more in my mind.
Post by Charlie Poole
3) Not breaking compatibility with old apps
I'm not sure how it would..
Post by Charlie Poole
4) Test already takes an argument: the description.
This is probably the kicker if the description argument would be
confused with the parameters argument, which I suppose it would in the
case where the test has one string parameter. We don't have tests with
parameters at this point in any other context do we? We could (if we
wanted to) differentiate between

[Test("This test does stuff!")]
public void AnOldTest()
{
}

and

[Test("Kelly Anderson")]
public void ANewTest(string name)
{
}

by looking at the metadata, and seeing that it has a parameter. It's a
bit subtle... but I think I prefer this as every other attribute used
inside of a TestFixture aside from Test doesn't run as a test, but is
rather a special case. This isn't a special case like SetUp or
TearDown, at least it doesn't seem the same to me.

I'll be ok with it either way, but my opinion is that overloading
TestAttribute for this functionality would be just fine. If you can
come up with a specific instance where it would break old code, I'm
listening. If others have opinions, I'd be interested as well. From a
usability standpoint, why add yet another attribute?
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
For dynamic data, I'm thinking of supporting something
like this...
Post by Charlie Poole
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double
denominator, double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
Post by Charlie Poole
}
}
Where MyDataSource is a class provided by the user. I'm still
working on the details of this, so please throw in your ideas. I'd
like to have a typesafe interface for data, rather than using a
string.
I see where you're trying to go with the type safe thing
here. That makes some sense... but don't you still have a
problem extracing numerator, denominator and result from
MyDataSource? That part would want to be type safe too, wouldn't it?
The interface can be type safe, the actual returned values need to
be tested for a match to the argument types dynamically. That's no
big deal since large parts of NUnit already work this way.
Do you have ideas about how to do this? I've been noodling it for a
couple of days now, and nothing really jumps out at me as a solution.
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
More to come on this - your comments are welcome.
I'll give some thought to a type safe way to do this... do
you have anything against using generics? Or would the fact
that they aren't supported in 1.1 keep you off that track?
No, generics are a graeat approach for the 2.0 version of the runners.
Ok, would that help?
Post by Charlie Poole
Post by Kelly Anderson
Could the tests be written in 2.0 code to test against 1.1 code?
Only to test them under 2.0. That may be OK for some purposes though.
Ok. I wasn't sure what the official view towards generics was. I'll
keep noodling, maybe something will pop into my head.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Charlie Poole
2008-03-19 20:59:40 UTC
Permalink
Hi Kelly,
Post by Kelly Anderson
Post by Charlie Poole
Post by Kelly Anderson
Why not [Test(1000, 7, 142.85715)] ??? Is there a naming
conflict?
Post by Charlie Poole
1) Test already means a lot of things in NUnit. TestCases,
TestSuites
Post by Charlie Poole
TestFixtures, etc. are all Tests.
Ok, but they aren't all TestAttributes, are they?
TestFixtureAttribute doesn't inherit from TestAttribute,
right? Or are you just speaking semantically here...
Yes, I always try to speak semantically. :-)

More seriously, what I mean is that because it's used in so
many ways, "TestAttribute" may be something to get away from.
Post by Kelly Anderson
Post by Charlie Poole
2) A new name for a new thing
Kind of... more like an old thing doing more in my mind.
Depends on point of view I guess.
Post by Kelly Anderson
Post by Charlie Poole
3) Not breaking compatibility with old apps
I'm not sure how it would..
See the next item.

Consider a method like this:
[Test][Test] public SomeMethod() { }
It would refuse to compile. To use Test as you suggest,
we would have to allow it.
Post by Kelly Anderson
Post by Charlie Poole
4) Test already takes an argument: the description.
This is probably the kicker if the description argument would
be confused with the parameters argument, which I suppose it
would in the case where the test has one string parameter. We
don't have tests with parameters at this point in any other
context do we? We could (if we wanted to) differentiate between
[Test("This test does stuff!")]
public void AnOldTest()
{
}
and
[Test("Kelly Anderson")]
public void ANewTest(string name)
{
}
by looking at the metadata, and seeing that it has a
parameter. It's a bit subtle... but I think I prefer this as
every other attribute used inside of a TestFixture aside from
Test doesn't run as a test, but is rather a special case.
This isn't a special case like SetUp or TearDown, at least it
doesn't seem the same to me.
Ah - we're working from different value systems here. You're
talking about what it's possible to program. I'm talking about
what will be easiest for users.
Post by Kelly Anderson
I'll be ok with it either way, but my opinion is that
overloading TestAttribute for this functionality would be
just fine. If you can come up with a specific instance where
it would break old code, I'm listening. If others have
opinions, I'd be interested as well. From a usability
standpoint, why add yet another attribute?
Because adding a new form for a new function is the easiest
thing for users to understand. Overloading can sometimes
represent the triumph of cleverness over simplicity.
Post by Kelly Anderson
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
For dynamic data, I'm thinking of supporting
something > like
Post by Charlie Poole
this...
Post by Kelly Anderson
Post by Charlie Poole
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double
denominator, double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
Post by Charlie Poole
}
}
Where MyDataSource is a class provided by the user.
I'm still
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
working on the details of this, so please throw in your ideas.
I'd > > like to have a typesafe interface for data, rather than
using a > > string.
Post by Kelly Anderson
I see where you're trying to go with the type safe thing
here.
That makes some sense... but don't you still have a > problem
extracing numerator, denominator and result from >
MyDataSource? That
Post by Charlie Poole
part would want to be type safe too, wouldn't it?
The interface can be type safe, the actual returned values
need to
Post by Charlie Poole
be tested for a match to the argument types dynamically. That's no
big deal since large parts of NUnit already work this way.
Do you have ideas about how to do this? I've been noodling it
for a couple of days now, and nothing really jumps out at me
as a solution.
Only insofar as I already described it. Since I know that I
know how to do it, it's not something I need to spike. I'll
put out the code when I have it.
Post by Kelly Anderson
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
More to come on this - your comments are welcome.
I'll give some thought to a type safe way to do this...
do > you
Post by Charlie Poole
have anything against using generics? Or would the fact >
that they
Post by Charlie Poole
aren't supported in 1.1 keep you off that track?
No, generics are a graeat approach for the 2.0 version of
the runners.
Ok, would that help?
Post by Charlie Poole
Post by Kelly Anderson
Could the tests be written in 2.0 code to test against 1.1 code?
Only to test them under 2.0. That may be OK for some
purposes though.
Ok. I wasn't sure what the official view towards generics
was. I'll keep noodling, maybe something will pop into my head.
Good.

Charlie
Post by Kelly Anderson
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by: Microsoft Defy all
challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Kelly Anderson
2008-03-19 23:23:35 UTC
Permalink
On Wed, Mar 19, 2008 at 2:59 PM, Charlie Poole
Post by Charlie Poole
Hi Kelly,
Post by Kelly Anderson
Post by Charlie Poole
Post by Kelly Anderson
Why not [Test(1000, 7, 142.85715)] ??? Is there a naming
conflict?
Post by Charlie Poole
1) Test already means a lot of things in NUnit. TestCases,
TestSuites
Post by Charlie Poole
TestFixtures, etc. are all Tests.
Ok, but they aren't all TestAttributes, are they?
TestFixtureAttribute doesn't inherit from TestAttribute,
right? Or are you just speaking semantically here...
Yes, I always try to speak semantically. :-)
More seriously, what I mean is that because it's used in so
many ways, "TestAttribute" may be something to get away from.
You know more than me... but from my mostly user's point of view, I
think it would be great if it were in TestAttribute.
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
2) A new name for a new thing
Kind of... more like an old thing doing more in my mind.
Depends on point of view I guess.
I suppose.
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
3) Not breaking compatibility with old apps
I'm not sure how it would..
See the next item.
[Test][Test] public SomeMethod() { }
It would refuse to compile. To use Test as you suggest,
we would have to allow it.
So what would that do, just run the test twice would be what I would
expect. Since nobody has previously done this (since you can't now)
how is that a backwards compatibility issue?
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
4) Test already takes an argument: the description.
This is probably the kicker if the description argument would
be confused with the parameters argument, which I suppose it
would in the case where the test has one string parameter. We
don't have tests with parameters at this point in any other
context do we? We could (if we wanted to) differentiate between
[Test("This test does stuff!")]
public void AnOldTest()
{
}
and
[Test("Kelly Anderson")]
public void ANewTest(string name)
{
}
by looking at the metadata, and seeing that it has a
parameter. It's a bit subtle... but I think I prefer this as
every other attribute used inside of a TestFixture aside from
Test doesn't run as a test, but is rather a special case.
This isn't a special case like SetUp or TearDown, at least it
doesn't seem the same to me.
Ah - we're working from different value systems here. You're
talking about what it's possible to program. I'm talking about
what will be easiest for users.
So am I. I see that it is possible, but I also see that it's
semantically desirable (from my own point of view). We can just agree
to disagree on that point, of course.
Post by Charlie Poole
Post by Kelly Anderson
I'll be ok with it either way, but my opinion is that
overloading TestAttribute for this functionality would be
just fine. If you can come up with a specific instance where
it would break old code, I'm listening. If others have
opinions, I'd be interested as well. From a usability
standpoint, why add yet another attribute?
Because adding a new form for a new function is the easiest
thing for users to understand. Overloading can sometimes
represent the triumph of cleverness over simplicity.
I suppose.
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
For dynamic data, I'm thinking of supporting
something > like
Post by Charlie Poole
this...
Post by Kelly Anderson
Post by Charlie Poole
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double
denominator, double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
Post by Charlie Poole
}
}
Where MyDataSource is a class provided by the user.
I'm still
Post by Charlie Poole
Post by Kelly Anderson
Post by Charlie Poole
working on the details of this, so please throw in your ideas.
I'd > > like to have a typesafe interface for data, rather than
using a > > string.
Post by Kelly Anderson
I see where you're trying to go with the type safe thing
here.
That makes some sense... but don't you still have a > problem
extracing numerator, denominator and result from >
MyDataSource? That
Post by Charlie Poole
part would want to be type safe too, wouldn't it?
The interface can be type safe, the actual returned values
need to
Post by Charlie Poole
be tested for a match to the argument types dynamically. That's no
big deal since large parts of NUnit already work this way.
Do you have ideas about how to do this? I've been noodling it
for a couple of days now, and nothing really jumps out at me
as a solution.
Only insofar as I already described it. Since I know that I
know how to do it, it's not something I need to spike. I'll
put out the code when I have it.
I'm interested in/curious about what DataSource will look like.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Andreas Schlapsi
2008-03-19 22:47:31 UTC
Permalink
Post by Kelly Anderson
Post by Charlie Poole
4) Test already takes an argument: the description.
This is probably the kicker if the description argument would be
confused with the parameters argument, which I suppose it would in the
case where the test has one string parameter. We don't have tests with
parameters at this point in any other context do we? We could (if we
wanted to) differentiate between
[Test("This test does stuff!")]
public void AnOldTest()
{
}
and
[Test("Kelly Anderson")]
public void ANewTest(string name)
{
}
by looking at the metadata, and seeing that it has a parameter. It's a
bit subtle... but I think I prefer this as every other attribute used
inside of a TestFixture aside from Test doesn't run as a test, but is
rather a special case. This isn't a special case like SetUp or
TearDown, at least it doesn't seem the same to me.
I think this is hard to understand for users. I prefer the
TestCaseAttribute.


Andreas


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Andreas Schlapsi
2008-03-17 23:49:45 UTC
Permalink
Hi Charlie,
Post by Charlie Poole
Internally, NUnit will support an extension point
for providing data to tests. The extension will
1) An actual data source, if the data is static.
2) An object for use at runtime to get the actual
data source, if the data is dynamic.
Will there be two separate extension points or just one?
Post by Charlie Poole
For static test methods, I'm thinking of supporting
[TestFixture]
public class StaticDataSample
{
[TestCase( 1000, 10, 100.0000)]
[TestCase(-1000, 10, -100.0000)]
[TestCase( 1000, 7, 142.85715)]
[TestCase( 1000, 0.00001, 100000000)]
[TestCase(4195835, 3145729, 1.3338196)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator / denominator, 0.00001);
}
}
If you're familiar with RowTest, you'll see that this is
essentially the same thing, with TestCase substituted for Row.
In fact I took the example from Andreas and I think we can
incorporate his code right into NUnit.
I'll help you with this.
Post by Charlie Poole
For dynamic data, I'm thinking of supporting something
like this...
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator / denominator, 0.00001);
}
}
Where MyDataSource is a class provided by the user. I'm
still working on the details of this, so please throw in
your ideas. I'd like to have a typesafe interface for
data, rather than using a string.
This looks interesting. How would the typesafe interface look like? Will
an extension be able to provide another way of generating dynamic data
by using the extension point? Otherwise we won't need the extension
point for dynamic data (or to cope with the one returning an object
which returns the data at runtime), because the data source is specified
as type.


Andreas



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Charlie Poole
2008-03-19 14:33:40 UTC
Permalink
Hi Andreas,
Post by Charlie Poole
Internally, NUnit will support an extension point for
providing data
1) An actual data source, if the data is static.
2) An object for use at runtime to get the actual
data source, if the data is dynamic.
Will there be two separate extension points or just one?
I think we need one if some future extensions are going to support
both types of data, as shown by some of the Theory examples. I think
such extensions would have to treat all the data dynamically for
consistency however.
Post by Charlie Poole
For static test methods, I'm thinking of supporting the following
[TestFixture]
public class StaticDataSample
{
[TestCase( 1000, 10, 100.0000)]
[TestCase(-1000, 10, -100.0000)]
[TestCase( 1000, 7, 142.85715)]
[TestCase( 1000, 0.00001, 100000000)]
[TestCase(4195835, 3145729, 1.3338196)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
}
}
If you're familiar with RowTest, you'll see that this is
essentially
the same thing, with TestCase substituted for Row.
In fact I took the example from Andreas and I think we can
incorporate
his code right into NUnit.
I'll help you with this.
Great. Lets get our heads together offline.
Post by Charlie Poole
For dynamic data, I'm thinking of supporting something like this...
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
}
}
Where MyDataSource is a class provided by the user. I'm
still working
on the details of this, so please throw in your ideas. I'd like to
have a typesafe interface for data, rather than using a string.
This looks interesting. How would the typesafe interface look
like? Will an extension be able to provide another way of
generating dynamic data by using the extension point?
Otherwise we won't need the extension point for dynamic data
(or to cope with the one returning an object which returns
the data at runtime), because the data source is specified as type.
I'm not sure about the interface but I thought I'd throw the idea
out to let others think about it as well. I'll work on an example.

I think the extension would simply implement the interface - how
it created the data would be an internal matter.

Charlie
Post by Charlie Poole
Andreas
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Charlie Poole
2008-03-21 00:09:19 UTC
Permalink
Hi Andreas,
Post by Charlie Poole
Internally, NUnit will support an extension point for
providing data
1) An actual data source, if the data is static.
2) An object for use at runtime to get the actual
data source, if the data is dynamic.
Will there be two separate extension points or just one?
It's a good question. It would probably be easier to implement
separate interfaces but we may want to mix static with
dynamic data at some point, which leads me to think that we
should at least have a single interface - if not a single
extension point. I guess we'll let the code tell us. :-)
Post by Charlie Poole
For static test methods, I'm thinking of supporting the following
[TestFixture]
public class StaticDataSample
{
[TestCase( 1000, 10, 100.0000)]
[TestCase(-1000, 10, -100.0000)]
[TestCase( 1000, 7, 142.85715)]
[TestCase( 1000, 0.00001, 100000000)]
[TestCase(4195835, 3145729, 1.3338196)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
}
}
If you're familiar with RowTest, you'll see that this is
essentially
the same thing, with TestCase substituted for Row.
In fact I took the example from Andreas and I think we can
incorporate
his code right into NUnit.
I'll help you with this.
Good, I was hoping you would. How do you want to start?
Post by Charlie Poole
For dynamic data, I'm thinking of supporting something like this...
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
}
}
Where MyDataSource is a class provided by the user. I'm
still working
on the details of this, so please throw in your ideas. I'd like to
have a typesafe interface for data, rather than using a string.
This looks interesting. How would the typesafe interface look
like? Will an extension be able to provide another way of
generating dynamic data by using the extension point?
Otherwise we won't need the extension point for dynamic data
(or to cope with the one returning an object which returns
the data at runtime), because the data source is specified as type.
No answers on how it will look, but yes the idea is that there should
be an extension point in it somewhere.

Charlie
Post by Charlie Poole
Andreas
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Kelly Anderson
2008-03-21 16:31:08 UTC
Permalink
On Sun, Mar 16, 2008 at 7:02 PM, Charlie Poole
Post by Charlie Poole
Hi All,
We've had much discussion on the need for parameterized
or data-driven tests in 2.5. I thought I'd outline here
some thoughts about where I see it going.
<snip>
For dynamic data, I'm thinking of supporting something
like this...
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double denominator,
double result)
{
Assert.AreEqual(result, numerator / denominator, 0.00001);
}
}
I was trying to get my mind around the issue of data-driven and Theory
testing, and I ran across this little tidbit in the C# spec about what
types could be parameters to Attributes.

C# Language Specification
17.1.3 Attribute parameter types

The types of positional and named parameters for an attribute class
are limited to the attribute parameter types, which are:

* One of the following types: bool, byte, char, double, float,
int, long, short, string.
* The type object.
* The type System.Type.
* An enum type, provided it has public accessibility and the types
in which it is nested (if any) also have public accessibility (Section
17.2).
* Single-dimensional arrays of the above types.

What this seems to mean is that for the TestCase Attribute, all of the
parameters have to be primitives, or type unsafe object's... It limits
the flexibility of how we can approach this problem.

I'm still trying to come up with something that will produce a compile
time error/warning, but nothing comes to mind yet.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Charlie Poole
2008-03-21 17:17:35 UTC
Permalink
Hi Kelly,
-----Original Message-----
Behalf Of Kelly Anderson
Sent: Friday, March 21, 2008 9:31 AM
Subject: Re: [nunit-developer] Data-driven Tests in NUnit 2.5
On Sun, Mar 16, 2008 at 7:02 PM, Charlie Poole
Post by Charlie Poole
Hi All,
We've had much discussion on the need for parameterized or
data-driven tests in 2.5. I thought I'd outline here some thoughts
about where I see it going.
<snip>
For dynamic data, I'm thinking of supporting something
like this...
Post by Charlie Poole
[TestFixture]
public class DynamicDataSample
{
[Test, DataSource( typeof(MyDataSource)]
public void DivisionTest(double numerator, double
denominator, double result)
{
Assert.AreEqual(result, numerator /
denominator, 0.00001);
Post by Charlie Poole
}
}
I was trying to get my mind around the issue of data-driven
and Theory testing, and I ran across this little tidbit in
the C# spec about what types could be parameters to Attributes.
C# Language Specification
17.1.3 Attribute parameter types
The types of positional and named parameters for an attribute
* One of the following types: bool, byte, char, double,
float, int, long, short, string.
* The type object.
* The type System.Type.
* An enum type, provided it has public accessibility and
the types in which it is nested (if any) also have public
accessibility (Section 17.2).
* Single-dimensional arrays of the above types.
What this seems to mean is that for the TestCase Attribute,
all of the parameters have to be primitives, or type unsafe
object's... It limits the flexibility of how we can approach
this problem.
I'm still trying to come up with something that will produce
a compile time error/warning, but nothing comes to mind yet.
Yes, that's why my hypothetical attribute takes a type. NUnit
would have to check that the provided type actually implements
the necessary interface.

It's awkward, but not a show-stopper.

Charlie
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by: Microsoft Defy all
challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Kelly Anderson
2008-03-22 00:05:19 UTC
Permalink
On Fri, Mar 21, 2008 at 11:17 AM, Charlie Poole
Post by Charlie Poole
Hi Kelly,
Post by Kelly Anderson
I'm still trying to come up with something that will produce
a compile time error/warning, but nothing comes to mind yet.
Yes, that's why my hypothetical attribute takes a type. NUnit
would have to check that the provided type actually implements
the necessary interface.
It's awkward, but not a show-stopper.
The key benefit in my mind of type safety is compile time vs. runtime
notification of the problem. If you have to live with runtime, why not
use the simpler "string" approach? Or, do you see some other advantage
aside from compile time notification of type safety?

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Charlie Poole
2008-03-22 00:16:06 UTC
Permalink
Hi Kelly,
Post by Kelly Anderson
Post by Charlie Poole
It's awkward, but not a show-stopper.
The key benefit in my mind of type safety is compile time vs.
runtime notification of the problem. If you have to live with
runtime, why not use the simpler "string" approach? Or, do
you see some other advantage aside from compile time
notification of type safety?
Requiring a Type automatically catches things like spelling
errors and allows the code to be refactored by tools like R#.

It should cut out most of the low-level errors, but will
lead to a runtime error if your type doesn't implement
a required interface - just for one example.

However, the proof will be in the code. We're just
talking here.

Charlie
Post by Kelly Anderson
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by: Microsoft Defy all
challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Kelly Anderson
2008-03-24 15:58:24 UTC
Permalink
On Fri, Mar 21, 2008 at 8:40 PM, Charlie Poole
Post by Charlie Poole
Hi Kelly,
This isn't about Theory, it's about Data-driven tests in
NUnit 2.5.
Ok, but the point is still that if the data generation is far
away from the data test, it makes the code harder to read. If
it's a function in the TestFixture class, that ensures that
it's fairly close to the TestCase that's being run. That's
the only point I was trying to make. Putting the data
generation code in another class may mean you have to read
more code to understand the nature of the Test.
Point taken. However, the issue is more important IMO with Theory
and less so with standard data-driven tests, where all the data
is presumed to be correct - and may come from a file anyway.
After the original Red run, I would think that you would presume your
Theory would always be correct too... :-)

It's clearly very important to have the data "close" to your Theory
for readability.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Kelly Anderson
2008-03-24 16:01:42 UTC
Permalink
On Fri, Mar 21, 2008 at 9:28 PM, Charlie Poole
Post by Charlie Poole
Hi Kelly,
That ship has sailed. :-) Internally, most of what NUnit does
involves objects because we implement features that can't be
done in a static manner. For example, a great deal of type
casting is involved when we compare numeric types - everything
that normally happens automatically in a numeric expression
has to be done explicitly by NUnit at run time.
So I'm not minimizing the problems, just saying that we
already solve them internally without the user knowing
about it. What you're talking about will just me more
of the same old same old. :-)
I can see that this is the case. The "requirement" that you proposed
was to have type safety. If we don't have type safety now, why require
it for a future feature? Or do you achieve type safety on some level
now that you want to maintain?

I'm trying to understand the motivation behind the request for type
safety in the implementation of Theory data sources.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Charlie Poole
2008-03-24 16:33:00 UTC
Permalink
Hi Kelly,
Post by Charlie Poole
That ship has sailed. :-) Internally, most of what NUnit does
involves objects because we implement features that can't
be > done in a static manner. For example, a great deal of
type > casting is involved when we compare numeric types -
everything > that normally happens automatically in a
numeric expression > has to be done explicitly by NUnit at run time.
Post by Charlie Poole
So I'm not minimizing the problems, just saying that we
already solve them internally without the user knowing >
about it. What you're talking about will just me more > of
the same old same old. :-)
I can see that this is the case. The "requirement" that you
proposed was to have type safety. If we don't have type
safety now, why require it for a future feature? Or do you
achieve type safety on some level now that you want to maintain?
Yes.
I'm trying to understand the motivation behind the request
for type safety in the implementation of Theory data sources.
I think I see where this discussion has gotten off track.

I wasn't intending to establish a "requirement" or even make a
"request." I merely made a statement of how I intended to
proceed with my own programming experiments. I really didn't
anticipate debating about it.

And again: I'm talking about data-driven tests not Theory.

Charlie



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Kelly Anderson
2008-03-24 17:31:10 UTC
Permalink
On Mon, Mar 24, 2008 at 10:33 AM, Charlie Poole
Post by Charlie Poole
Hi Kelly,
Post by Kelly Anderson
I can see that this is the case. The "requirement" that you
proposed was to have type safety. If we don't have type
safety now, why require it for a future feature? Or do you
achieve type safety on some level now that you want to maintain?
Yes.
Ok.
Post by Charlie Poole
Post by Kelly Anderson
I'm trying to understand the motivation behind the request
for type safety in the implementation of Theory data sources.
I think I see where this discussion has gotten off track.
I wasn't intending to establish a "requirement" or even make a
"request." I merely made a statement of how I intended to
proceed with my own programming experiments. I really didn't
anticipate debating about it.
And again: I'm talking about data-driven tests not Theory.
This isn't a debate. I just don't know how to achieve it. I suppose I
should just wait and see what you come up with. I was simply trying to
noodle it out for myself, but with no luck thus far.

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Loading...