Help running tests


#1

Hey all,

Forgive me if this is documented somewhere, but I was unable to find it.

I have had a hard time getting the tests to run cleanly, wondering if there is something I am missing here.

The first issue I ran into was needing a couple of ENV vars set for integration tests. After setting them I got some errors from WebMock about sending actual HTTP requests. So I just disabled the integration tests all together.

Next issues had to do with a couple of missing gems (ruby-prof and rails-perftest), so I installed those and moved on.

At that point I got some errors about a class being overloaded. AuthTest exists in both the browser tests and unit tests. So I disabled the browser tests temporarily.

Now I am seeing a ton of failures, even on master.

I am sure I am doing something wrong here, any guidance would be appreciated.

Thanks!


#2

Maybe @thorsteneckel can help at this point, when he finds some spare time.


#3

Our test suite is currently a bit cluttered. We’re actively working on refactoring it. Here are a list on how to execute the tests:

RSpec

New tests should be written in RSpec. Existing tests will get migrated over time. To execute the RSpec test suite run:

  • export RAILS_ENV=test
  • script/bootstrap.sh
  • bundle exec rspec

Unit and Controller tests

UnitTests currently cover most of the application. These will get replaced by RSpec on the long run. Additionally the Controller tests will get replaced with RSpec Request tests as recommended by the Rails core team. To execute the Unit and Controller tests run:

  • export RAILS_ENV=test
  • script/bootstrap.sh
  • rake test:units
  • rake test:controllers

(Browser) Integration tests

There are different integration tests. Some of them run out of the box, some need special ENV vars and some of them are currently failing because of third party provider limitations. I’ll skip them for now because describing and providing the required ENVs for each of them will take too much time for now and will probably change (soon). Let me know if you have any questions in particular - I’m happy to answer them.

Brower tests

Running browser tests is currently pretty complex and takes the most effort. We’re working on introducing Capybara feature tests which will ease the process a lot. ETA is about 3-4 weeks from now.

  • export RAILS_ENV=test
  • script/bootstrap.sh
  • export BROWSER=chrome
  • export BROWSER_URL=http://localhost:3000
  • cp contrib/auto_wizard_test.json auto_wizard.json
  • start all Zammad services (Application server, Websocket server, Scheduler)
  • ruby -Itest test/integration/aaa_auto_wizard_base_setup_test.rb
  • ruby -Itest test/browser/your.rb

Let me know if you have questions :rocket: Happy hacking!


#4

Thanks for the detailed instructions, those did the trick!


#5

I’m currently trying to write an RSpec test for Zammad, but the slow startup time makes it rather annoying:

$ bundle exec rspec ./spec/models/user_spec.rb:40
[...]
Finished in 0.46001 seconds (files took 5.94 seconds to load)

I’ve found that using spring-commands-rspec makes it much faster. Zammad in development/test mode already depends on it, you just need to enable it:

bundle exec spring binstub rspec

Then start a spring instance in a separate window:

spring server

and then run the tests with bundle exec bin/rspec:

$ bundle exec bin/rspec ./spec/models/user_spec.rb:40
[...]
Finished in 0.34802 seconds (files took 0.56761 seconds to load)

#6

I haven’t been notified for your last reply but stumbled across it now. Spring is still in a beta phase because it causes some unexpected side effects* (e.g. when switching branches or changing dependencies). Anyhow, you can easily activate it by setting the ENV ENABLE_SPRING without having to run the stubs command or starting the server by hand. All you need to do is set the ENV and run any command (rspec, rails etc.) without any further doing. I have it in my .envrc to set it automatically when entering the Zammad directory and it’s pretty stable after some fixes in the Zammad codebase.

Would love to hear your feedback so we can mark it as stable and have spring active by default in development and test Rails env.

*Best practice: When in doubt about an error: Run spring stop and try again. Afaik this is a common issue with Spring and not related to Zammad.