Discussion:
Best practices for sharing state between story steps?
(too old to reply)
Sam Stokes
2008-09-07 23:58:00 UTC
Permalink
Hi all,

I'm just getting into RSpec stories and liking them (especially with
webrat), but I'm finding it tricky to write steps that are
self-contained and reusable, particularly where features intersect.
What approaches do people use to achieve this?

(Maybe a better question is, do people bother? One of the things I like
about story-runner is the way I'm building a DSL for integration testing
my application, but should I just write the scenario I need, write the
steps to make it run and forget about reusing steps?)

As an example of where I'm having trouble, say I'm writing a blog (since
it's the Web 2.0 version of Hello World), so I have posts and comments.
I want to write a scenario something like

----
Given a post
And some comments for the post
When I view the post
Then I should see the comments
----

How do I tell the "Given some comments" step which post to attach the
comments to?

I can do it by having "Given a post" set a @post instance variable and
having "Given some comments" use that, but it feels like global
variables all over again. My steps (probably in different files) are
coupled via the @post instance variable, and other steps can clobber it,
and if I forget to clear it I might pollute later steps, and if I want
to refer to more than one post (@post1, @post2) I have to rewrite all my
steps...

I noticed a little note on the Cucumber wiki
(http://github.com/aslakhellesoy/cucumber/wikis/home) explicitly
advising against using @variables for this, I'm guessing for these
reasons.

I could avoid the first issue by combining the top two steps into "Given
a post with some comments" but then I still have the problem of which
post "When I view the post" should GET. Maybe "When I view a post with
comments"...?

Any advice, criticism or sympathy appreciated :)
--
Sam
--
Posted via http://www.ruby-forum.com/.
Zach Dennis
2008-09-08 01:57:36 UTC
Permalink
This thread may help give you some insight...

http://www.mail-archive.com/rspec-users-***@public.gmane.org/msg05382.html

Zach
Post by Sam Stokes
Hi all,
I'm just getting into RSpec stories and liking them (especially with
webrat), but I'm finding it tricky to write steps that are
self-contained and reusable, particularly where features intersect.
What approaches do people use to achieve this?
(Maybe a better question is, do people bother? One of the things I like
about story-runner is the way I'm building a DSL for integration testing
my application, but should I just write the scenario I need, write the
steps to make it run and forget about reusing steps?)
As an example of where I'm having trouble, say I'm writing a blog (since
it's the Web 2.0 version of Hello World), so I have posts and comments.
I want to write a scenario something like
----
Given a post
And some comments for the post
When I view the post
Then I should see the comments
----
How do I tell the "Given some comments" step which post to attach the
comments to?
having "Given some comments" use that, but it feels like global
variables all over again. My steps (probably in different files) are
and if I forget to clear it I might pollute later steps, and if I want
steps...
I noticed a little note on the Cucumber wiki
(http://github.com/aslakhellesoy/cucumber/wikis/home) explicitly
reasons.
I could avoid the first issue by combining the top two steps into "Given
a post with some comments" but then I still have the problem of which
post "When I view the post" should GET. Maybe "When I view a post with
comments"...?
Any advice, criticism or sympathy appreciated :)
--
Sam
--
Posted via http://www.ruby-forum.com/.
_______________________________________________
rspec-users mailing list
http://rubyforge.org/mailman/listinfo/rspec-users
--
Zach Dennis
http://www.continuousthinking.com
http://www.mutuallyhuman.com
Sam Stokes
2008-09-08 15:44:55 UTC
Permalink
Post by Zach Dennis
This thread may help give you some insight...
Thanks for pointing me there. The first insight it's given me is that
I'm actually asking two different (though related) questions:
1) What's the best way to build reusable steps, and is it a good idea?
2) What's the best way to share state between the steps in a single
scenario?

That thread addresses 1) directly, and sounds like there's some evidence
that reusable steps aren't such a good idea. In which case 2) pretty
much drops out - if I don't try so hard to reuse my steps then there are
fewer disadvantages to the obvious global @variable tactic.

If you take the view that reusable steps may be unhelpful, what do you
find is a good way to organise them? I can see an extreme case - each
plain-text foo.story gets its own steps defined in foo.rb, so there are
no cross-file @variable conventions to worry about. In this way foo.rb
is basically a clarification in code of foo.story, rather than part of
an app-specific language (ASL?) for integration testing. Does anyone do
this, and do they find it a useful way of looking at the problem?
--
Posted via http://www.ruby-forum.com/.
Jonathan Linowes
2008-09-08 13:58:36 UTC
Permalink
Post by Sam Stokes
What approaches do people use to achieve this?
Perhaps I'm bucking what others have advised against, so take it for
what it's worth.

I make some limited use of global (instance) variables that
correspond to english language pronouns. I have things like
@current_user (corresponds to "I"), @current_project (corresponds to
"the project"), etc. I am careful to be consistent. There's only a
handful of these, but I find it extremely convenient. Note, these
also tend correspond to 'states' in my app, which might be in the
session or part of a nested URL.

With regard to reusable steps, I have some steps that are generic,
some are app specific, and some are (group of) story/feature specific:

1. The generic ones are very reusable, for example

When "I click the $link link" do |link|
clicks_link link.strip_quotes
end

Then "the browser should show $a_or_an $tag tag with $contents" do
|_, tag, contents|
response.should have_tag(tag, contents.strip_quotes)
end


2. The app specific ones are intended to be reusable in any story but
only my app, may do some database initialization (like fixtures),
logging in, for example:

Given "a $pname project with $settings"
Given "I am logged in as a $role member"

Note, $settings is in the format
Given a Test project with foo: 1, bar: "bar value", and baz: yadda
yadda

creates a default project overridden with the specified settings
(using to_hash_from_story, http://www.mail-archive.com/rspec-
users-***@public.gmane.org/msg05771.html )


3. And the feature-set specific steps are not very reusable at all.
As it turns out, there's relatively few of these, and they tend to
reflect aggregates of more granular steps already tested in a
different story or scenario.

hope this helps,

linoj
Sam Stokes
2008-09-08 16:17:42 UTC
Permalink
Post by Jonathan Linowes
I make some limited use of global (instance) variables that
correspond to english language pronouns. I have things like
@current_user (corresponds to "I"), @current_project (corresponds to
"the project"), etc. I am careful to be consistent. There's only a
handful of these, but I find it extremely convenient. Note, these
also tend correspond to 'states' in my app, which might be in the
session or part of a nested URL.
So could I summarise this approach as, "use @variables, with good names
and consistent principles for use"?

One problem I've had is that some steps implicitly have to be preceded
by other steps, so that the @variables get set up right (e.g. "Given
some comments" refers to @post, which has to be set beforehand by "Given
a post"). I can make the wording of the steps more explicit, but then I
end up with clunky steps like "Given some existing comments by the user
for the post".

Generally I've found with this sort of approach I end up writing fairly
imperative-style scenarios (as in
http://www.benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories/),
and they're quite brittle with respect to small changes to steps.

Do you encounter these problems at all?
--
Posted via http://www.ruby-forum.com/.
Jim Morris
2008-09-08 20:01:13 UTC
Permalink
Post by Sam Stokes
Post by Jonathan Linowes
I make some limited use of global (instance) variables that
correspond to english language pronouns. I have things like
@current_user (corresponds to "I"), @current_project (corresponds to
"the project"), etc. I am careful to be consistent. There's only a
handful of these, but I find it extremely convenient. Note, these
also tend correspond to 'states' in my app, which might be in the
session or part of a nested URL.
and consistent principles for use"?
I'm of the opinion "do what works". I know there are purists that say don't do this and don't do
that, but when you come down to it you have to use variables between steps, look at the rails
examples and it sets a response variable between steps.

What I do is horrible :) but "it works", I set GLOBAL variables (yup $current_state), because I
found that sometimes @variable didn't get setup properly in some cases.

Then I use the listeners to clear those variables between scenarios, so every scenario I have a
listener (effectively before_scenario), that clears all the global variables I use between steps,
this avoids errors where I don't set up something in a step but the scenario passes because it just
happens to have the right value in a global set by a previous test.

This does require some maintenance, however I could use a global hash for my
inter-step-dependencies, and just $hash.clear in the before_scenario listener.

Ahh I remember why I had to use $variable and not @variable, for some reason the before_scenario
listener does not have access to the same scope as the scenario so I have to use $variables.

So most of my scenarios look like this....

before_scenario do
$v1= nil
$v2= nil
end

Scenario "xxx" do
Given "something..."
When "I do something" # this will set $v1 and $v2
Then "check it was done" # this will check $v1 and $v2
end

I know purists will hate this ;) But "it works for me"

BTW I use stories entirely for integration testing, testing the entire stack by using the same
inputs (in my case a WEB based REST API) to the stack as a client would.

I also do nasty things like delve into the database directly in a Then to check the database
actually got setup the way I expected by a previous When, as well as checking the returned XML from
the Rest-API call.

As always YMMV
--
Jim Morris, http://blog.wolfman.com
Dan North
2008-09-09 12:21:59 UTC
Permalink
Hi Jim.

I guess I'm not a purist then - that looks fine to me, and it's probably
something I would consider doing too.

The thing to bear in mind is that there is magic going on when you run
steps. Each step in a scenario is run in the context of the same object
instance (which you don't get to see explicitly) by pixies, which means any
@variables you set in one step should be visible to any other steps in the
same scenario. I don't think of this as global state - it's simply a
sensible way to share state across steps. You can think of the object the
steps run in as a "world" (it might even still be called that - it was at
one point) where you can set and verify state.

There are bound to be pathological cases though, such as the before_scenario
and after_scenario listeners. Perhaps there's a bug there, in that they
should be running in the same object instance as the scenario steps
themselves. Perhaps not. In the latter case I would definitely go with
$globals to communicate state into your scenario, as long as you promise
never to use them in your application code. Never, you hear me?

Cheers,
Dan
Post by Jim Morris
Post by Sam Stokes
Post by Jonathan Linowes
I make some limited use of global (instance) variables that
correspond to english language pronouns. I have things like
@current_user (corresponds to "I"), @current_project (corresponds to
"the project"), etc. I am careful to be consistent. There's only a
handful of these, but I find it extremely convenient. Note, these
also tend correspond to 'states' in my app, which might be in the
session or part of a nested URL.
and consistent principles for use"?
I'm of the opinion "do what works". I know there are purists that say don't
do this and don't do that, but when you come down to it you have to use
variables between steps, look at the rails examples and it sets a response
variable between steps.
What I do is horrible :) but "it works", I set GLOBAL variables (yup
properly in some cases.
Then I use the listeners to clear those variables between scenarios, so
every scenario I have a listener (effectively before_scenario), that clears
all the global variables I use between steps, this avoids errors where I
don't set up something in a step but the scenario passes because it just
happens to have the right value in a global set by a previous test.
This does require some maintenance, however I could use a global hash for
my inter-step-dependencies, and just $hash.clear in the before_scenario
listener.
reason the before_scenario listener does not have access to the same scope
as the scenario so I have to use $variables.
So most of my scenarios look like this....
before_scenario do
$v1= nil
$v2= nil
end
Scenario "xxx" do
Given "something..."
When "I do something" # this will set $v1 and $v2
Then "check it was done" # this will check $v1 and $v2
end
I know purists will hate this ;) But "it works for me"
BTW I use stories entirely for integration testing, testing the entire
stack by using the same inputs (in my case a WEB based REST API) to the
stack as a client would.
I also do nasty things like delve into the database directly in a Then to
check the database actually got setup the way I expected by a previous When,
as well as checking the returned XML from the Rest-API call.
As always YMMV
--
Jim Morris, http://blog.wolfman.com
_______________________________________________
rspec-users mailing list
http://rubyforge.org/mailman/listinfo/rspec-users
aslak hellesoy
2008-09-09 12:42:28 UTC
Permalink
Post by Dan North
Hi Jim.
I guess I'm not a purist then - that looks fine to me, and it's probably
something I would consider doing too.
The thing to bear in mind is that there is magic going on when you run
steps. Each step in a scenario is run in the context of the same object
instance (which you don't get to see explicitly) by pixies, which means any
@variables you set in one step should be visible to any other steps in the
same scenario. I don't think of this as global state - it's simply a
sensible way to share state across steps. You can think of the object the
steps run in as a "world" (it might even still be called that - it was at
one point) where you can set and verify state.
The debate seems to be whether step definitions should be stateful or not.
In practice this is achieved by setting one or more @variables in a
step and reusing them in a different step - all within a scenario.

Both are fine, but beware that as your codebase grows and you have
hundred or so step definitions, things can become *really* hard to
maintain.
It won't bite you when you start on a new codebase, but in my
experience stateful steps will bite you later.

Having experienced this pain in one project, I decided to try out
stateless steps exclusively on the next project. I've found stateless
steps to be a tad more verbose than stateful ones, both in the text
and in the implementation (because you have to pass "identifiers"
around). Still, the improved maintainability of my features and
scenarios using this approach outweighs this slight increase in
verbosity.

Here is an example of a stateless scenario: (As a convention I'm
"quoting" variables)

http://github.com/aslakhellesoy/ba/tree/master/features/submit_proposal.feature

You can see the extra verbosity I'm talking about in the repetition of
"Aslak", "Beerfest" etc.

I'm still pragmatic about this of course, but now you know some of my
experience.

Aslak
Post by Dan North
There are bound to be pathological cases though, such as the before_scenario
and after_scenario listeners. Perhaps there's a bug there, in that they
should be running in the same object instance as the scenario steps
themselves. Perhaps not. In the latter case I would definitely go with
$globals to communicate state into your scenario, as long as you promise
never to use them in your application code. Never, you hear me?
Cheers,
Dan
Post by Jim Morris
Post by Sam Stokes
Post by Jonathan Linowes
I make some limited use of global (instance) variables that
correspond to english language pronouns. I have things like
@current_user (corresponds to "I"), @current_project (corresponds to
"the project"), etc. I am careful to be consistent. There's only a
handful of these, but I find it extremely convenient. Note, these
also tend correspond to 'states' in my app, which might be in the
session or part of a nested URL.
and consistent principles for use"?
I'm of the opinion "do what works". I know there are purists that say
don't do this and don't do that, but when you come down to it you have to
use variables between steps, look at the rails examples and it sets a
response variable between steps.
What I do is horrible :) but "it works", I set GLOBAL variables (yup
properly in some cases.
Then I use the listeners to clear those variables between scenarios, so
every scenario I have a listener (effectively before_scenario), that clears
all the global variables I use between steps, this avoids errors where I
don't set up something in a step but the scenario passes because it just
happens to have the right value in a global set by a previous test.
This does require some maintenance, however I could use a global hash for
my inter-step-dependencies, and just $hash.clear in the before_scenario
listener.
reason the before_scenario listener does not have access to the same scope
as the scenario so I have to use $variables.
So most of my scenarios look like this....
before_scenario do
$v1= nil
$v2= nil
end
Scenario "xxx" do
Given "something..."
When "I do something" # this will set $v1 and $v2
Then "check it was done" # this will check $v1 and $v2
end
I know purists will hate this ;) But "it works for me"
BTW I use stories entirely for integration testing, testing the entire
stack by using the same inputs (in my case a WEB based REST API) to the
stack as a client would.
I also do nasty things like delve into the database directly in a Then to
check the database actually got setup the way I expected by a previous When,
as well as checking the returned XML from the Rest-API call.
As always YMMV
--
Jim Morris, http://blog.wolfman.com
_______________________________________________
rspec-users mailing list
http://rubyforge.org/mailman/listinfo/rspec-users
_______________________________________________
rspec-users mailing list
http://rubyforge.org/mailman/listinfo/rspec-users
Jim Morris
2008-09-09 18:52:10 UTC
Permalink
Post by aslak hellesoy
The debate seems to be whether step definitions should be stateful or not.
step and reusing them in a different step - all within a scenario.
I think that is the debate, but I'd like to point out that there is always state between steps, it
is just a matter of where it is being kept. In your example for instance most of the state is being
kept in the database between steps (ie between Given an "active" site_user names "aslak" and he
following steps.

(As a side question how do you clean up the database between tests, so the "state" from the previous
Scenario doesn't affect the next Scenario?) In some cases I use a randomly generated record each time.
Post by aslak hellesoy
Here is an example of a stateless scenario: (As a convention I'm
"quoting" variables)
That is quite interesting, however as I pointed out above there is state between steps, in the
database and in the response.

I actually do something very similar, with named resources in each step, however I find I still need
to use variables between steps, here is an example... (Not rails)

Scenario: testing a pet
Given I have a pet named "my dog"
When I stroke "my dog"
Then "my dog"'s health goes up

In the Given I create a pet named "my dog" in the database either directly or through the REST-ful
API to my Web app, however in both cases I need the returned id of the created pet to make any
further calls on it.

In the When clause I need to make an API call that pets "my dog", the API is POST stroke/345
So what I do is in the Given I assign the returned id of the new record to $petid, then in the When
clause I do something like, $http_req.post("stroke", id => $petid).

Now I needed to carry that variable $petid around otherwise how would I be able to make the call in
When.

Similarly when I want to check my pets health in the Then clause I still need that $petid, whether I
check the value directly in the database or make an API call that returns my pets health.

I don't see anyway around keeping state in these cases.

Now I do clear that state ($petid= nil) in the before_scenario hook, so another scenario which may
be written badly and not assign the $petid, won't succeeed due to a previously successful scenario.

I'd definitely be interested in better ways to do this though, as I hate passing global variables
around (as I said in an earlier post I can't use @variable because the before_scenario does not seem
to have access to the same scope as the scenario that is about to run). Although I think someone
said each scenario has its own variables? I didn't notice that behavior.
--
Jim Morris, http://blog.wolfman.com
Matt Wynne
2008-09-09 19:52:45 UTC
Permalink
Post by Jim Morris
Post by aslak hellesoy
The debate seems to be whether step definitions should be stateful or not.
step and reusing them in a different step - all within a scenario.
I think that is the debate, but I'd like to point out that there is
always state between steps, it is just a matter of where it is
being kept. In your example for instance most of the state is being
kept in the database between steps (ie between Given an "active"
site_user names "aslak" and he following steps.
(As a side question how do you clean up the database between tests,
so the "state" from the previous Scenario doesn't affect the next
Scenario?) In some cases I use a randomly generated record each time.
I think the scenarios are each wrapped in a transaction, so (as long
as you're using the right type of database / tables) the slate should
be wiped clean between each one.
Post by Jim Morris
Post by aslak hellesoy
Here is an example of a stateless scenario: (As a convention I'm
"quoting" variables)
That is quite interesting, however as I pointed out above there is
state between steps, in the database and in the response.
I actually do something very similar, with named resources in each
step, however I find I still need to use variables between steps,
here is an example... (Not rails)
Scenario: testing a pet
Given I have a pet named "my dog"
When I stroke "my dog"
Then "my dog"'s health goes up
In the Given I create a pet named "my dog" in the database either
directly or through the REST-ful API to my Web app, however in both
cases I need the returned id of the created pet to make any further
calls on it.
In the When clause I need to make an API call that pets "my dog",
the API is POST stroke/345
So what I do is in the Given I assign the returned id of the new
record to $petid, then in the When clause I do something like,
$http_req.post("stroke", id => $petid).
Now I needed to carry that variable $petid around otherwise how
would I be able to make the call in When.
Similarly when I want to check my pets health in the Then clause I
still need that $petid, whether I check the value directly in the
database or make an API call that returns my pets health.
I don't see anyway around keeping state in these cases.
Now I do clear that state ($petid= nil) in the before_scenario
hook, so another scenario which may be written badly and not assign
the $petid, won't succeeed due to a previously successful scenario.
I'd definitely be interested in better ways to do this though, as I
hate passing global variables around (as I said in an earlier post
have access to the same scope as the scenario that is about to
run). Although I think someone said each scenario has its own
variables? I didn't notice that behavior.
I've been thinking about this a bit, after reading Aslak's advice
which went contrary to my instincts. It seems like you could probably
do something like this instead:

Scenario: testing a pet
Given I have a pet
When I stroke the pet
Then the pet's health goes up

Because your Given step creates just one Pet instance, you can safely
refer to it as "the pet" in the other steps, and simply call Pet.find
(:first) in the step matcher.

See?


In a more complicated scenario:

Scenario: testing a pet
Given I have two pets named "my dog" and "my cat"
When I stroke "my dog"
Then "my cat" should be pissed off

Now your step matcher needs to refer to a particular pet, but can't
it just use the name as a key, since that will be unique within the
set of two Pet records you created in your Given step?

When /I stroke "(.*)"/ |pet_name| do
Pet.find_by_name(pet_name).stroke
end


Does that make sense to you?


cheers,
Matt
----
http://blog.mattwynne.net
http://songkick.com

In case you wondered: The opinions expressed in this email are my own
and do not necessarily reflect the views of any former, current or
future employers of mine.
Jim Morris
2008-09-09 21:18:54 UTC
Permalink
Thanks for the hints, but I think I did point out I am not using Rails, I write Integration tests,
that talk directly to the web application via HTTP running the full stack.

So Active record is not available neither are transactions for the database.

Sorry for the confusion.

If I were using Rails I would undoubtedly do it the way you suggest :)
Post by Jim Morris
Post by aslak hellesoy
The debate seems to be whether step definitions should be stateful or not.
step and reusing them in a different step - all within a scenario.
I think that is the debate, but I'd like to point out that there is
always state between steps, it is just a matter of where it is being
kept. In your example for instance most of the state is being kept in
the database between steps (ie between Given an "active" site_user
names "aslak" and he following steps.
(As a side question how do you clean up the database between tests, so
the "state" from the previous Scenario doesn't affect the next
Scenario?) In some cases I use a randomly generated record each time.
I think the scenarios are each wrapped in a transaction, so (as long as
you're using the right type of database / tables) the slate should be
wiped clean between each one.
Post by Jim Morris
Post by aslak hellesoy
Here is an example of a stateless scenario: (As a convention I'm
"quoting" variables)
That is quite interesting, however as I pointed out above there is
state between steps, in the database and in the response.
I actually do something very similar, with named resources in each
step, however I find I still need to use variables between steps, here
is an example... (Not rails)
Scenario: testing a pet
Given I have a pet named "my dog"
When I stroke "my dog"
Then "my dog"'s health goes up
In the Given I create a pet named "my dog" in the database either
directly or through the REST-ful API to my Web app, however in both
cases I need the returned id of the created pet to make any further
calls on it.
In the When clause I need to make an API call that pets "my dog", the
API is POST stroke/345
So what I do is in the Given I assign the returned id of the new
record to $petid, then in the When clause I do something like,
$http_req.post("stroke", id => $petid).
Now I needed to carry that variable $petid around otherwise how would
I be able to make the call in When.
Similarly when I want to check my pets health in the Then clause I
still need that $petid, whether I check the value directly in the
database or make an API call that returns my pets health.
I don't see anyway around keeping state in these cases.
Now I do clear that state ($petid= nil) in the before_scenario hook,
so another scenario which may be written badly and not assign the
$petid, won't succeeed due to a previously successful scenario.
I'd definitely be interested in better ways to do this though, as I
hate passing global variables around (as I said in an earlier post I
access to the same scope as the scenario that is about to run).
Although I think someone said each scenario has its own variables? I
didn't notice that behavior.
I've been thinking about this a bit, after reading Aslak's advice which
went contrary to my instincts. It seems like you could probably do
Scenario: testing a pet
Given I have a pet
When I stroke the pet
Then the pet's health goes up
Because your Given step creates just one Pet instance, you can safely
refer to it as "the pet" in the other steps, and simply call
Pet.find(:first) in the step matcher.
See?
Scenario: testing a pet
Given I have two pets named "my dog" and "my cat"
When I stroke "my dog"
Then "my cat" should be pissed off
Now your step matcher needs to refer to a particular pet, but can't it
just use the name as a key, since that will be unique within the set of
two Pet records you created in your Given step?
When /I stroke "(.*)"/ |pet_name| do
Pet.find_by_name(pet_name).stroke
end
Does that make sense to you?
cheers,
Matt
----
http://blog.mattwynne.net
http://songkick.com
In case you wondered: The opinions expressed in this email are my own
and do not necessarily reflect the views of any former, current or
future employers of mine.
--
Jim Morris, http://blog.wolfman.com
Jonathan Linowes
2008-09-10 20:17:21 UTC
Permalink
of course, there's one 'global' shared between steps that we cant
live without: response
:)
Zach Dennis
2008-09-09 21:29:04 UTC
Permalink
Post by Jim Morris
Post by aslak hellesoy
The debate seems to be whether step definitions should be stateful or not.
step and reusing them in a different step - all within a scenario.
I think that is the debate, but I'd like to point out that there is always
state between steps, it is just a matter of where it is being kept. In your
example for instance most of the state is being kept in the database between
steps (ie between Given an "active" site_user names "aslak" and he following
steps.
(As a side question how do you clean up the database between tests, so the
"state" from the previous Scenario doesn't affect the next Scenario?) In
some cases I use a randomly generated record each time.
I think the scenarios are each wrapped in a transaction, so (as long as
you're using the right type of database / tables) the slate should be wiped
clean between each one.
I don't think this works as you may expect since My
SQL and PostgreSQL don't support nested transactions, which is what
would happen if your test environment wrapped stories/scenarios in
transactions and your application utilized transactions.

Granted there has been hopes of getting savepoints fix the problem,
but the ticket has not been resolved:

http://rails.lighthouseapp.com/projects/8994/tickets/383-activerecord-should-use-savepoints-for-nested-transactions
--
Zach Dennis
http://www.continuousthinking.com
http://www.mutuallyhuman.com
Jarkko Laine
2008-09-10 05:35:37 UTC
Permalink
Post by Zach Dennis
I don't think this works as you may expect since My
SQL and PostgreSQL don't support nested transactions, which is what
would happen if your test environment wrapped stories/scenarios in
transactions and your application utilized transactions.
I've been bitten by this a couple of times. A common idiom in the
Rails world is (was at least) to use save! to raise an exception when
an object couldn't be saved. In a rescue block you could then do the
things you wanted to do when saving failed. Raising the exception also
rolled back a transaction, so it was a nice way to ensure some
transactional logic:

MyModel.transaction do
MyModel.save!
OtherModel.save!
end

If either line in the transaction failed, it was ensured that neither
of them was persisted in the db. This helped in many cases where you
wanted either all or none of your updates to get into the db. However,
with stories these cases always fail, because of the reason Zach
mentions.

The question whether a case where someone posts an empty email address
is an exceptional state in strict PragProg sense might be a bit
questionable, of course, but it's beyond the point here...

//jarkko

--
Jarkko Laine
http://jlaine.net
http://dotherightthing.com
http://www.railsecommerce.com
http://odesign.fi
aslak hellesoy
2008-09-09 20:21:41 UTC
Permalink
Post by Jim Morris
Post by aslak hellesoy
The debate seems to be whether step definitions should be stateful or not.
step and reusing them in a different step - all within a scenario.
I think that is the debate, but I'd like to point out that there is always
state between steps, it is just a matter of where it is being kept. In your
example for instance most of the state is being kept in the database between
steps (ie between Given an "active" site_user names "aslak" and he following
steps.
There is persistent state (database) and object state (the object that
serves as a context for a scenario).

I'm not saying that state in and of itself is bad. However, *coupling*
is bad - for maintenance reasons. If the steps share object state (in
@variables) they become coupled.

So why is coupling bad for maintenance? When your step library grows
to several dozens or like in my case, over a hundred, you want to be
able to cherry-pick steps to build new scenarios. If you have coupled
them with object state, you can't do that. These steps will simply not
work unless they're used alongside the steps they are @coupled to.

That's when you start to add new step definitions with a similar
semantic and wording, but with a different implementation - to work
around the other steps that you realised you can't reuse. Now things
get confusing. You have Given /there is a contract named (.*)/ and a
Given /there is an existing contract named (.*)/ etc. etc.
Post by Jim Morris
(As a side question how do you clean up the database between tests, so the
"state" from the previous Scenario doesn't affect the next Scenario?) In
some cases I use a randomly generated record each time.
Cucumber starts a transaction when a scenario starts and rolls it back
when it ends (when on Rails).
Post by Jim Morris
Post by aslak hellesoy
Here is an example of a stateless scenario: (As a convention I'm
"quoting" variables)
That is quite interesting, however as I pointed out above there is state
between steps, in the database and in the response.
That's not the coupling kind of state I'm talking about. That's ok.
Post by Jim Morris
I actually do something very similar, with named resources in each step,
however I find I still need to use variables between steps, here is an
example... (Not rails)
Scenario: testing a pet
Given I have a pet named "my dog"
When I stroke "my dog"
Then "my dog"'s health goes up
In the Given I create a pet named "my dog" in the database either directly
or through the REST-ful API to my Web app, however in both cases I need the
returned id of the created pet to make any further calls on it.
In the When clause I need to make an API call that pets "my dog", the API is
POST stroke/345
So what I do is in the Given I assign the returned id of the new record to
$petid, then in the When clause I do something like,
$http_req.post("stroke", id => $petid).
Now I needed to carry that variable $petid around otherwise how would I be
able to make the call in When.
Similarly when I want to check my pets health in the Then clause I still
need that $petid, whether I check the value directly in the database or make
an API call that returns my pets health.
I don't see anyway around keeping state in these cases.
Now I do clear that state ($petid= nil) in the before_scenario hook, so
another scenario which may be written badly and not assign the $petid, won't
succeeed due to a previously successful scenario.
I'd definitely be interested in better ways to do this though, as I hate
passing global variables around (as I said in an earlier post I can't use
@variable because the before_scenario does not seem to have access to the
same scope as the scenario that is about to run). Although I think someone
said each scenario has its own variables? I didn't notice that behavior.
In Cucumber a Before block has access to the same scope as the steps.
Post by Jim Morris
--
Jim Morris, http://blog.wolfman.com
_______________________________________________
rspec-users mailing list
http://rubyforge.org/mailman/listinfo/rspec-users
Jim Morris
2008-09-09 21:31:56 UTC
Permalink
Post by aslak hellesoy
There is persistent state (database) and object state (the object that
serves as a context for a scenario).
I'm not saying that state in and of itself is bad. However, *coupling*
is bad - for maintenance reasons. If the steps share object state (in
@variables) they become coupled.
So why is coupling bad for maintenance? When your step library grows
to several dozens or like in my case, over a hundred, you want to be
able to cherry-pick steps to build new scenarios. If you have coupled
them with object state, you can't do that. These steps will simply not
I agree and have run into this problem, but I have not found a better way in the environment I am using.
Post by aslak hellesoy
Cucumber starts a transaction when a scenario starts and rolls it back
when it ends (when on Rails).
Nice if you are using Rails, which I am not :) I was curious if anyone had run into the same
problems I do, because I don't use rails, I test directly against the HTTP API, so the database gets
all this test data, you can't use transactions, so the database potentially has this state from
previous Scenarios.

Right now in my Before_Scenario I generally try to clear the database, but that gets hard when it is
relational with a ton of foreign key constraints, and you can't simply delete or truncate a table.
As I said one work around is I create randomly named records every run, so as to avoid collision
with previous data. I wonder if anyone has a better idea?

BTW my Web service is written in Java/Jetty/Spring so I have no Rails/Ruby niceties on that side, so
it is nice to be able to use Ruby in the integration tests.
Post by aslak hellesoy
Post by Jim Morris
That is quite interesting, however as I pointed out above there is state
between steps, in the database and in the response.
That's not the coupling kind of state I'm talking about. That's ok.
Ok I think I see the difference, by the state being in the Database or the response variable it does
not implicitly couple steps like an @variable would. But it still requires the database to have been
setup in a certain state by certain preceding Givens or Whens, which is a kind of coupling. But
maybe I am stretching too far on that argument? I still feel coupling is coupling and state is state :)
Post by aslak hellesoy
In Cucumber a Before block has access to the same scope as the steps.
Good that'll make the transition easier I hope.
--
Jim Morris, http://blog.wolfman.com
Sam Stokes
2008-09-10 00:50:23 UTC
Permalink
Thanks everyone for your responses - this is a really helpful discussion
and every post has added to my understanding. Thanks also for
disagreeing with each other, as it makes me feel less stupid for asking
the question :)
Post by Jim Morris
Ok I think I see the difference, by the state being in the Database or
the response variable it does
requires the database to have been
setup in a certain state by certain preceding Givens or Whens, which is
a kind of coupling. But
maybe I am stretching too far on that argument? I still feel coupling is
coupling and state is state :)
It seems to me that with

"Given an existing post 'lol internet'" # creates a post with that title
"Then the page should include 'lol internet'" # assumes a post with that
title

the assumptions are in line with what you'd naturally understand from
the English, whereas

"Given an existing post 'lol internet'" # creates and stores in
@the_post
"Then I should see the post title" # verifies @the_post matches what's
in the db

seems to make more assumptions than are obvious from the English.

In fact, couldn't one argue that the second example is redundantly
storing the post in two places (the database and @the_post)? I think
this is what Aslak means by distinguishing "stateful" and "stateless" -
@variables are *extra* state, on top of the application state which must
change either way or you're not testing anything.
--
Posted via http://www.ruby-forum.com/.
Jim Morris
2008-09-10 03:54:06 UTC
Permalink
Post by Sam Stokes
It seems to me that with
"Given an existing post 'lol internet'" # creates a post with that title
"Then the page should include 'lol internet'" # assumes a post with that
title
the assumptions are in line with what you'd naturally understand from
the English, whereas
"Given an existing post 'lol internet'" # creates and stores in
@the_post
in the db
seems to make more assumptions than are obvious from the English.
In fact, couldn't one argue that the second example is redundantly
this is what Aslak means by distinguishing "stateful" and "stateless" -
@variables are *extra* state, on top of the application state which must
change either way or you're not testing anything.
Thats true in that case. I also wouldn't do what you show in the second case. If I can get the
result from the API or the database I would.

In my case I store data between states that I can't get anywhere else, or that tells me where to get
the results from.
--
Jim Morris, http://blog.wolfman.com
aslak hellesoy
2008-09-10 07:34:04 UTC
Permalink
Post by Jim Morris
Post by aslak hellesoy
There is persistent state (database) and object state (the object that
serves as a context for a scenario).
I'm not saying that state in and of itself is bad. However, *coupling*
is bad - for maintenance reasons. If the steps share object state (in
@variables) they become coupled.
So why is coupling bad for maintenance? When your step library grows
to several dozens or like in my case, over a hundred, you want to be
able to cherry-pick steps to build new scenarios. If you have coupled
them with object state, you can't do that. These steps will simply not
I agree and have run into this problem, but I have not found a better way in
the environment I am using.
Post by aslak hellesoy
Cucumber starts a transaction when a scenario starts and rolls it back
when it ends (when on Rails).
Nice if you are using Rails, which I am not :) I was curious if anyone had
run into the same problems I do, because I don't use rails, I test directly
against the HTTP API, so the database gets all this test data, you can't use
transactions, so the database potentially has this state from previous
Scenarios.
Sorry for not paying attention to this earlier.
Post by Jim Morris
Right now in my Before_Scenario I generally try to clear the database, but
that gets hard when it is relational with a ton of foreign key constraints,
and you can't simply delete or truncate a table.
As I said one work around is I create randomly named records every run, so
as to avoid collision with previous data. I wonder if anyone has a better
idea?
BTW my Web service is written in Java/Jetty/Spring so I have no Rails/Ruby
niceties on that side, so it is nice to be able to use Ruby in the
integration tests.
I've done that on several projects (using selenium/watir mostly). Some
things you could try:

1) HTTP database endpoint
A couple of years ago I was using Selenium against a Java web app. We
implemented a "special" HTTP endpoint for testing only that we would
access before a test. It would put the database in a pristine state.
It's clunky, but it works.

2) Ruby database endpoint
Another way to do it is to access the database directly from Ruby
before the test runs and clear it there.

3) Turn off constraints
Approaches 1 and 2 may be hard to do if you have complex constraints.
One strategy is to turn off constraints in your test database and just
do brute force deletes along with 1 or 2.

4) Incremental, semi-random data
This is the approach you are describing. The problem with this is risk
of false positives and intermittent failures. How risky it is depends
on your app of course.

5) Nested transactions
Using 1 or 2, start a transaction in the beginning of a scenario and
roll it back at the end. Only works if your database supports nested
transactions.

I don't know if there is any good literature or discussion groups
about these topics, but there ought to be.

Aslak
Post by Jim Morris
Post by aslak hellesoy
Post by Jim Morris
That is quite interesting, however as I pointed out above there is state
between steps, in the database and in the response.
That's not the coupling kind of state I'm talking about. That's ok.
Ok I think I see the difference, by the state being in the Database or the
would. But it still requires the database to have been setup in a certain
state by certain preceding Givens or Whens, which is a kind of coupling. But
maybe I am stretching too far on that argument? I still feel coupling is
coupling and state is state :)
Post by aslak hellesoy
In Cucumber a Before block has access to the same scope as the steps.
Good that'll make the transition easier I hope.
--
Jim Morris, http://blog.wolfman.com
_______________________________________________
rspec-users mailing list
http://rubyforge.org/mailman/listinfo/rspec-users
Rick DeNatale
2008-09-10 12:24:36 UTC
Permalink
Post by aslak hellesoy
Post by Dan North
Hi Jim.
I guess I'm not a purist then - that looks fine to me, and it's probably
something I would consider doing too.
The debate seems to be whether step definitions should be stateful or not.
step and reusing them in a different step - all within a scenario.
Both are fine, but beware that as your codebase grows and you have
hundred or so step definitions, things can become *really* hard to
maintain.
It won't bite you when you start on a new codebase, but in my
experience stateful steps will bite you later.
Having experienced this pain in one project, I decided to try out
stateless steps exclusively on the next project. I've found stateless
steps to be a tad more verbose than stateful ones, both in the text
and in the implementation (because you have to pass "identifiers"
around). Still, the improved maintainability of my features and
scenarios using this approach outweighs this slight increase in
verbosity.
Here is an example of a stateless scenario: (As a convention I'm
"quoting" variables)
http://github.com/aslakhellesoy/ba/tree/master/features/submit_proposal.feature
You can see the extra verbosity I'm talking about in the repetition of
"Aslak", "Beerfest" etc.
I'm still pragmatic about this of course, but now you know some of my
experience.
I've done something similar, but I might have handled the steps
implementation a bit differently in that I'm not sure my approach would be
called stateless.

For example say I had (in a very abstract form)

Given 'Rick' has admin privileges
Then 'Rick' can do admin function X

In the step for the given, I'd get or create an admin user named rick, and
set an instance variable say @admin
Then in the step for the then, I'd have an expectation that the name of
@admin matched the name given in the step.
This was a way of catching cases where the story mixed up actors.

Although the short example doesn't really emphasize the problem I was trying
to solve, I'm not sure in retrospect whether or not this was a valuable
thing to do.
--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/
Jim Morris
2008-09-10 19:43:28 UTC
Permalink
Post by Rick DeNatale
I've done something similar, but I might have handled the steps
implementation a bit differently in that I'm not sure my approach would
be called stateless.
For example say I had (in a very abstract form)
Given 'Rick' has admin privileges
Then 'Rick' can do admin function X
In the step for the given, I'd get or create an admin user named rick,
Then in the step for the then, I'd have an expectation that the name of
@admin matched the name given in the step.
This was a way of catching cases where the story mixed up actors.
Although the short example doesn't really emphasize the problem I was
trying to solve, I'm not sure in retrospect whether or not this was a
valuable thing to do.
I like this discussion because it shows the ways different people have solved a very practical problem.

I don't think what you describe is "bad", I do something very similar with my integration tests that
test a RESTFUL API based web service that returns XML to requests.

I use Hpricot to test the responses, but as this is not rails, I have to store the response between
steps.

A contrived Example...

Scenario "test get resource list"
Given "database has 10 resources
When "request made to list resources"
Then "10 resources are returned"

In my case the Given actually clears the resource table and populates it with 10 resources, I access
the database directly using the Sequel Gem, however if the database is remote this is problematic.

The When makes the RESTful request via http. and captures the result as an Hpricot object.
The When step looks something like this...

When "..." do
resp = @req_http.post(path, params, headers)

if resp.code != "200"
raise "#{resp.code} #{resp.message}"
end

@hdoc = Hpricot.XML(resp.body)
end

The Then step looks something like this...

Then "$cnt resources are returned" do |cnt|
@hdoc.should have_xpath("/resources", :count => cnt)
@hdoc.should have_xpath("/resources/resource[@id='1']")
@hdoc.should have_xpath("/resources/resource[@id='2']")
...etc...
end

I have a custom hpricot matcher which provides the have_xpath matcher

Yes this is stateful, but the @hdoc variable is central to the tests and I don't see any other way
to carry the response from the When to the Then, unless you combine the step in a then and skip the
when, so you don't have to carry state across the steps, but I prefer the first method as I can run
more tests on the returned document.

e.g.
Then "requesting a list of resources returns 10 items"
--
Jim Morris, http://blog.wolfman.com
Jim Morris
2008-09-11 16:54:53 UTC
Permalink
I've done something similar, but I might have handled the steps implementation a bit differently
in that I'm not sure my approach would be called stateless.
For example say I had (in a very abstract form)
Given 'Rick' has admin privileges
Then 'Rick' can do admin function X
In the step for the given, I'd get or create an admin user named rick, and set an instance
given in the step.
This was a way of catching cases where the story mixed up actors.
Although the short example doesn't really emphasize the problem I was trying to solve, I'm not
sure in retrospect whether or not this was a valuable thing to do.

I like this discussion because it shows the ways different people have solved a very practical problem.

I don't think what you describe is "bad", I do something very similar with my integration tests that
test a RESTFUL API based web service that returns XML to requests.

I use Hpricot to test the responses, but as this is not rails, I have to store the response between
steps.

A contrived Example...

Scenario "test get resource list"
Given "database has 10 resources
When "request made to list resources"
Then "10 resources are returned"

In my case the Given actually clears the resource table and populates it with 10 resources, I access
the database directly using the Sequel Gem, however if the database is remote this is problematic.

The When makes the RESTful request via http. and captures the result as an Hpricot object.
The When step looks something like this...

When "..." do
resp = @req_http.post(path, params, headers)

if resp.code != "200"
raise "#{resp.code} #{resp.message}"
end

@hdoc = Hpricot.XML(resp.body)
end

The Then step looks something like this...

Then "$cnt resources are returned" do |cnt|
@hdoc.should have_xpath("/resources", :count => cnt)
@hdoc.should have_xpath("/resources/resource[@id='1']")
@hdoc.should have_xpath("/resources/resource[@id='2']")
...etc...
end

I have a custom hpricot matcher which provides the have_xpath matcher

Yes this is stateful, but the @hdoc variable is central to the tests and I don't see any other way
to carry the response from the When to the Then, unless you combine the step in a then and skip the
when, so you don't have to carry state across the steps, but I prefer the first method as I can run
more tests on the returned document.

e.g.
Then "requesting a list of resources returns 10 items"
--
Jim Morris, http://blog.wolfman.com Rick DeNatale wrote:
Jim Morris
2008-09-09 18:28:27 UTC
Permalink
Post by Dan North
Hi Jim.
I guess I'm not a purist then - that looks fine to me, and it's probably
something I would consider doing too.
I'd never call you a purist Dan ;) But I do feel less dirty now, although after reading Aslaks post
I wonder what exactly is meant by stateless steps? It seems to me that you have to set some kind of
variable between steps to communicate what was done in one and what needs to be checked in another.

Having been guilty of writing unit tests that were not stateless (ie one unit test depended on the
result of a previous one, which BTW I really avoid doing), I did not consider setting variables
between steps to be stateful.
Post by Dan North
scenario steps themselves. Perhaps not. In the latter case I would
definitely go with $globals to communicate state into your scenario, as
long as you promise never to use them in your application code. Never,
you hear me?
Hmmm well OK I'll try to never use them ;)
--
Jim Morris, http://blog.wolfman.com
aslak hellesoy
2008-09-09 18:40:41 UTC
Permalink
Post by Jim Morris
Post by Dan North
Hi Jim.
I guess I'm not a purist then - that looks fine to me, and it's probably
something I would consider doing too.
I'd never call you a purist Dan ;) But I do feel less dirty now, although
after reading Aslaks post
I wonder what exactly is meant by stateless steps? It seems to me that you
What I mean by a stateless step is a step that doesn't assign or use
Post by Jim Morris
have to set some kind of
variable between steps to communicate what was done in one and what needs to
be checked in another.
Having been guilty of writing unit tests that were not stateless (ie one
unit test depended on the
result of a previous one, which BTW I really avoid doing), I did not
consider setting variables
between steps to be stateful.
Post by Dan North
scenario steps themselves. Perhaps not. In the latter case I would
definitely go with $globals to communicate state into your scenario, as long
as you promise never to use them in your application code. Never, you hear
me?
Hmmm well OK I'll try to never use them ;)
--
Jim Morris, http://blog.wolfman.com
_______________________________________________
rspec-users mailing list
http://rubyforge.org/mailman/listinfo/rspec-users
Continue reading on narkive:
Loading...