Modificatons to NUnit 2.5 Data-Driven tests

[Charlie Poole]

Hi All,

Kelly has pointed out several issues with DataSourceAttribute, all of
which revolve around the fact that the property runs at load time,
which is before the tests are ever executed. This leads to some
potential user confusion, which is not helped by the fact that
the attribute name /sounds like/ data that is input to a test, rather
than a set of parameters used to construct the test.

MbUnit has a long history of using "Factory" in the names of attributes
that create tests. This may be a better naming convention, even though
they seem to have had one or two users confused by the issue as well.
I'm inclined to using it for us as well.

I also think now that limiting the data source (or factory) to be
a static property is too limiting. We should allow properties, methods
or fields and let them be either static or instance. Users will just
need to be aware that the instance creating the test cases is not
necessarily the same as the instance that executes them.

That's the general way I'm looking at modifying this stuff. Any thoughts?

Charlie
* Factory attribute on the property or method providing test cases
*




-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Jeff Brown]

It seems kind of pointless to me to have an instance for evaluating data
sources if it's not the one being used to execute the tests.

Jeff.

-----Original Message-----
From: nunit-developer-***@lists.sourceforge.net
[mailto:nunit-developer-***@lists.sourceforge.net] On Behalf Of Charlie
Poole
Sent: Saturday, May 31, 2008 2:10 AM
To: nunit-***@lists.sourceforge.net
Subject: [nunit-developer] Modificatons to NUnit 2.5 Data-Driven tests

Hi All,

Kelly has pointed out several issues with DataSourceAttribute, all of which
revolve around the fact that the property runs at load time, which is before
the tests are ever executed. This leads to some potential user confusion,
which is not helped by the fact that the attribute name /sounds like/ data
that is input to a test, rather than a set of parameters used to construct
the test.

MbUnit has a long history of using "Factory" in the names of attributes that
create tests. This may be a better naming convention, even though they seem
to have had one or two users confused by the issue as well.
I'm inclined to using it for us as well.

I also think now that limiting the data source (or factory) to be a static
property is too limiting. We should allow properties, methods or fields and
let them be either static or instance. Users will just need to be aware that
the instance creating the test cases is not necessarily the same as the
instance that executes them.

That's the general way I'm looking at modifying this stuff. Any thoughts?

Charlie
* Factory attribute on the property or method providing test cases
*




-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges.
Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
nunit-developer mailing list
nunit-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nunit-developer


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Charlie Poole]

If the test cases are being created as the test is loaded,
and - for example - displayed in the gui, then it doesn't seem
possible to use the same instance for execution unless we
use that instance for every execution of the test. Failure
to destroy instances properly at the end of each run was a source
of bugs in the past, so now we clean up properly.

To be honest, I don't really see why this type of test factory -
one that creates a test at load time - should ever need to be an
instance member of the fixture. Most of my own use for a test
factory would be to allow reuse of parameters, which would
otherwise be expressed as metadata, and static works perfectly
well in that case, since the values are constant.

Maybe I shouldn't allow use of instance members until/unless
the need for it makes sense to me. :-)

Charlie

Post by Jeff Brown
-----Original Message-----
Behalf Of Jeff Brown
Sent: Saturday, May 31, 2008 12:35 PM
Subject: Re: [nunit-developer] Modificatons to NUnit 2.5
Data-Driven tests
It seems kind of pointless to me to have an instance for
evaluating data sources if it's not the one being used to
execute the tests.
Jeff.
-----Original Message-----
Behalf Of Charlie Poole
Sent: Saturday, May 31, 2008 2:10 AM
Subject: [nunit-developer] Modificatons to NUnit 2.5 Data-Driven tests
Hi All,
Kelly has pointed out several issues with
DataSourceAttribute, all of which revolve around the fact
that the property runs at load time, which is before the
tests are ever executed. This leads to some potential user
confusion, which is not helped by the fact that the attribute
name /sounds like/ data that is input to a test, rather than
a set of parameters used to construct the test.
MbUnit has a long history of using "Factory" in the names of
attributes that create tests. This may be a better naming
convention, even though they seem to have had one or two
users confused by the issue as well.
I'm inclined to using it for us as well.
I also think now that limiting the data source (or factory)
to be a static property is too limiting. We should allow
properties, methods or fields and let them be either static
or instance. Users will just need to be aware that the
instance creating the test cases is not necessarily the same
as the instance that executes them.
That's the general way I'm looking at modifying this stuff.
Any thoughts?
Charlie
* Factory attribute on the property or method providing test cases
*
--------------------------------------------------------------
-----------
This SF.net email is sponsored by: Microsoft Defy all challenges.
Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer
--------------------------------------------------------------
-----------
This SF.net email is sponsored by: Microsoft Defy all
challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Kelly Anderson]

Post by Charlie Poole
Hi All,
Kelly has pointed out several issues with DataSourceAttribute, all of
which revolve around the fact that the property runs at load time,
which is before the tests are ever executed. This leads to some
potential user confusion, which is not helped by the fact that
the attribute name /sounds like/ data that is input to a test, rather
than a set of parameters used to construct the test.
MbUnit has a long history of using "Factory" in the names of attributes
that create tests. This may be a better naming convention, even though
they seem to have had one or two users confused by the issue as well.
I'm inclined to using it for us as well.

I suppose that could help to some extent. The part that makes me stop
and think is that the only real solution is more/better documentation,
which leads me to wonder if there isn't a somewhat better way. Then I
think perhaps there isn't... :-)

Post by Charlie Poole
I also think now that limiting the data source (or factory) to be
a static property is too limiting. We should allow properties, methods
or fields and let them be either static or instance. Users will just
need to be aware that the instance creating the test cases is not
necessarily the same as the instance that executes them.

I think both instances should be the same. I can think of no reason
that they wouldn't be. The reason you would want them to be the same
is if you set a member variable to a value in the IEnumerator field,
you would be confused by the fact that it wasn't set when your test
was entered. Now, that makes sense to us who are really quite familiar
with the structure, but it doesn't make sense to a casual user.

Post by Charlie Poole
That's the general way I'm looking at modifying this stuff. Any thoughts?

I'd still really like to see the case simplified such that when you
have only one parameter to your test function, such that the one
parameter could be an object, rather than having to be put into an
object array. Stuffing one object into an object array of length one
in your enumerator makes the code less readable and harder to write
too IMHO. That's my only really strong thought at this point. Seems
that it would be pretty easy to figure out from the metadata if the
items in the IEnumerator were of type object[] or of type object...

The point where you could disagree is that it makes it inconsistent
when you have to go to two or more parameters, but then it really IS
an array... :-)

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Charlie Poole]

Hi Kelly,

Post by Charlie Poole

Post by Charlie Poole
MbUnit has a long history of using "Factory" in the names of
attributes that create tests. This may be a better naming

convention,

Post by Charlie Poole
even though they seem to have had one or two users confused

by the issue as well.

Post by Charlie Poole
I'm inclined to using it for us as well.

I suppose that could help to some extent. The part that makes
me stop and think is that the only real solution is
more/better documentation,

There is currently zero documentation, so it's evident we need
more and better. We agreed on this many cycles back, so I think
you're beating a dead horse.

Post by Charlie Poole
which leads me to wonder if there isn't a somewhat better way.
Then I think perhaps there isn't... :-)

If, as I believe, most use cases will involve constants, then I
think you overstate the problem of understandability.

Post by Charlie Poole
I think both instances should be the same. I can think of no
reason that they wouldn't be.

We have discussed this one or two times and I just restated
the issue for Jeff, who wasn't part of the discussion. I thought
you had ended by agreeing with me that it isn't possible to do
what you are asking in THIS type of data source, as opposed to
what we were calling dynamic.

Here is the logic:
1. Tests may be loaded once but executed many times.
2. Each execution gets a new set of fixture objects, which
are created before the run and destroyed afterward.
3. Since we use a different instance for each run, it is
not possible to also use the same instance for the
one-time load.
4. If you want to use the same instance to read the data
(as opposed to load the tests) that is possible. That is
what we were (I was?) calling dynamic tests. Such tests
cannot appear in the initial load, but only in the result
tree.

Post by Charlie Poole
The reason you would want them
to be the same is if you set a member variable to a value in
the IEnumerator field, you would be confused by the fact that
it wasn't set when your test was entered. Now, that makes
sense to us who are really quite familiar with the structure,
but it doesn't make sense to a casual user.

What you are describing seems to me as either an anti-pattern
or (giving the benefit of the doubt) an advanced technique. :-)
In either case, the casual user would not need to understand.

Can you give a use case for this technique?

In any case, I think this is a job for the (I thought) agreed
upon /separate/ dynamic test feature. (Which, btw, I want to
work on next)

Charlie

Post by Charlie Poole

Post by Charlie Poole
That's the general way I'm looking at modifying this stuff.

Any thoughts?
I'd still really like to see the case simplified such that
when you have only one parameter to your test function, such
that the one parameter could be an object, rather than having
to be put into an object array. Stuffing one object into an
object array of length one in your enumerator makes the code
less readable and harder to write too IMHO. That's my only
really strong thought at this point. Seems that it would be
pretty easy to figure out from the metadata if the items in
the IEnumerator were of type object[] or of type object...
The point where you could disagree is that it makes it
inconsistent when you have to go to two or more parameters,
but then it really IS an array... :-)

I believe I suggested - and you filed - a bug. In case it
wasn't clear, that means I was agreeing with you. :-)

Post by Charlie Poole
-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by: Microsoft Defy all
challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Kelly Anderson]

On Sat, May 31, 2008 at 7:53 PM, Charlie Poole

Post by Charlie Poole
Hi Kelly,

Post by Charlie Poole

Post by Charlie Poole
MbUnit has a long history of using "Factory" in the names of
attributes that create tests. This may be a better naming

convention,

Post by Charlie Poole
even though they seem to have had one or two users confused

by the issue as well.

Post by Charlie Poole
I'm inclined to using it for us as well.

I suppose that could help to some extent. The part that makes
me stop and think is that the only real solution is
more/better documentation,

There is currently zero documentation, so it's evident we need
more and better. We agreed on this many cycles back, so I think
you're beating a dead horse.

Sorry, didn't mean to do that.

Post by Charlie Poole

Post by Charlie Poole
which leads me to wonder if there isn't a somewhat better way.
Then I think perhaps there isn't... :-)

If, as I believe, most use cases will involve constants, then I
think you overstate the problem of understandability.

Possibly. Many of my current data driven tests do not involve
constants, but rather file sets. So I may have a different mind set.
You've always said you would optimize for programming tests, and that
data tests would be considered second. So I think your thoughts on the
matter are consistent. I can send you my current set of data-driven
tests off line if you think it would be at all helpful to look at a
real example.

Post by Charlie Poole

Post by Charlie Poole
I think both instances should be the same. I can think of no
reason that they wouldn't be.

We have discussed this one or two times and I just restated
the issue for Jeff, who wasn't part of the discussion. I thought
you had ended by agreeing with me that it isn't possible to do
what you are asking in THIS type of data source, as opposed to
what we were calling dynamic.

Sorry, I wasn't remembering about the two types. I'll reserve further
comment until after I see your implementation of the second type as
well.

Post by Charlie Poole
1. Tests may be loaded once but executed many times.

That makes a pretty big difference.

Post by Charlie Poole
2. Each execution gets a new set of fixture objects, which
are created before the run and destroyed afterward.
3. Since we use a different instance for each run, it is
not possible to also use the same instance for the
one-time load.
4. If you want to use the same instance to read the data
(as opposed to load the tests) that is possible. That is
what we were (I was?) calling dynamic tests. Such tests
cannot appear in the initial load, but only in the result
tree.

Ok, this all makes sense. Now that you say it, I even remember it.
Kinda tired lately... :-) Too many irons in too many fires at once.

Post by Charlie Poole

Post by Charlie Poole
The reason you would want them
to be the same is if you set a member variable to a value in
the IEnumerator field, you would be confused by the fact that
it wasn't set when your test was entered. Now, that makes
sense to us who are really quite familiar with the structure,
but it doesn't make sense to a casual user.

What you are describing seems to me as either an anti-pattern
or (giving the benefit of the doubt) an advanced technique. :-)
In either case, the casual user would not need to understand.

If it is an anti-pattern, then perhaps in the documentation we should
promote that the IEnumerator field should be stateless. At least I
think that's what you're getting at.

Post by Charlie Poole
Can you give a use case for this technique?

At the moment, I can't... but I'll keep thinking about it. I had one
the other day. I think I'm just exhausted today. Sorry. Can't seem to
make my brain function properly at the moment.

Post by Charlie Poole
In any case, I think this is a job for the (I thought) agreed
upon /separate/ dynamic test feature. (Which, btw, I want to
work on next)

This thread is getting too chopped up in my head... That's my problem
not yours of course... :-)

-Kelly

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Charlie Poole]

Hi Kelly,

Post by Charlie Poole

Post by Kelly Anderson
which leads me to wonder if there isn't a somewhat better way.
Then I think perhaps there isn't... :-)

If, as I believe, most use cases will involve constants,

then I think

Post by Charlie Poole
you overstate the problem of understandability.

Possibly. Many of my current data driven tests do not involve
constants, but rather file sets. So I may have a different mind set.
You've always said you would optimize for programming tests,
and that data tests would be considered second. So I think
your thoughts on the matter are consistent. I can send you my
current set of data-driven tests off line if you think it
would be at all helpful to look at a real example.

Yes, that would be useful.

Post by Charlie Poole
What you are describing seems to me as either an anti-pattern or
(giving the benefit of the doubt) an advanced technique.

:-) In either

Post by Charlie Poole
case, the casual user would not need to understand.

If it is an anti-pattern, then perhaps in the documentation
we should promote that the IEnumerator field should be
stateless. At least I think that's what you're getting at.

Yes. It seems to me that a test should not know where it's
data comes from any more than it should know who ( console,
gui, nant, etc. ) is calling it.

Post by Charlie Poole
Can you give a use case for this technique?

At the moment, I can't... but I'll keep thinking about it. I
had one the other day. I think I'm just exhausted today.
Sorry. Can't seem to make my brain function properly at the moment.

I'm sure it will come back. :-)

Post by Charlie Poole
In any case, I think this is a job for the (I thought) agreed upon
/separate/ dynamic test feature. (Which, btw, I want to

work on next)
This thread is getting too chopped up in my head... That's my
problem not yours of course... :-)

I'm feeling I need to provide a higher level summary of where
I think we are going with this rather than just talk about small
parts of it as I have been doing. I'll try to find some time.

Charlie

-Kelly
--------------------------------------------------------------
-----------
This SF.net email is sponsored by: Microsoft Defy all
challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
nunit-developer mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-developer

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/