Actions

Difference between revisions of "Testing Area/Behat Testing/Error & Solutions"

From Mahara Wiki

< Testing Area‎ | Behat Testing
 
(27 intermediate revisions by 3 users not shown)
Line 25: Line 25:
  
 
== Behat tests cannot run unless $cfg->behat_wwwroot, $cfg->behat_dataroot, and $cfg->behat_dbprefix are defined in config.php ==
 
== Behat tests cannot run unless $cfg->behat_wwwroot, $cfg->behat_dataroot, and $cfg->behat_dbprefix are defined in config.php ==
 
 
Make sure the following following lines of code are in your config.php file
 
Make sure the following following lines of code are in your config.php file
 
+
‎<syntaxhighlight lang="php">
  $cfg->behat_dbprefix = 'behat_'; // must not empty
+
$cfg->behat_dbprefix = 'behat_'; // must not empty
  $cfg->behat_dataroot = "/var/lib/maharadata/master_behat"; // Behat's copy of maharadata
+
$cfg->behat_dataroot = "/var/lib/maharadata/master_behat"; // Behat's copy of maharadata
  $cfg->behat_wwwroot = 'http://localhost:8000'; // Must be this
+
$cfg->behat_wwwroot = 'http://localhost:8000'; // Must be this
  $cfg->behat_selenium2 = "http://127.0.0.1:4444/wd/hub"; // Must be this
+
$cfg->behat_selenium2 = "http://127.0.0.1:4444/wd/hub"; // Must be this
 
+
‎</syntaxhighlight>
  
 
== Selenium issues ==
 
== Selenium issues ==
Line 42: Line 41:
 
If you run into issues where the script says it has started Selenium, but that Behat throws an exception because it thinks Selenium has not started, specifically an error thrown in BehatHooks before_scenario saying 'Selenium server is not running, you need to start it to run tests that involve Javascript.', check the JRE installed:
 
If you run into issues where the script says it has started Selenium, but that Behat throws an exception because it thinks Selenium has not started, specifically an error thrown in BehatHooks before_scenario saying 'Selenium server is not running, you need to start it to run tests that involve Javascript.', check the JRE installed:
  
<source>java -version</source>
+
<syntaxhighlight lang="Bash">java -version</syntaxhighlight>
  
 
If it says you're using OpenJDK 10 or 11 (which can ship with Ubuntu 18.04 depending on which service pack is used), this needs to be uninstalled first:
 
If it says you're using OpenJDK 10 or 11 (which can ship with Ubuntu 18.04 depending on which service pack is used), this needs to be uninstalled first:
  
<source>sudo apt remove --purge openjdk*</source>
+
<syntaxhighlight lang="Bash">sudo apt remove --purge openjdk*</syntaxhighlight>
  
 
Once this is removed, you will need to re-run the OpenJDK 8 installation:
 
Once this is removed, you will need to re-run the OpenJDK 8 installation:
  
<source lang="bash" enclose="div">
+
<syntaxhighlight lang="Bash">
 
$ sudo apt install openjdk-8-jre-headless
 
$ sudo apt install openjdk-8-jre-headless
</source>
+
</syntaxhighlight>
  
 
=== Server not started ===
 
=== Server not started ===
When Selenium upgrades causing firefox to try run headless and not open firefox. It can also throw an error when you try run the test. It will mention something about the Selenium server not being able to start up.
+
When Selenium upgrades causing firefox to try run headless and not open firefox. It can also throw an error when you try to run the test. It will mention something about the Selenium server not being able to start up.
  
 
TO FIX:  
 
TO FIX:  
  
 
* First try the 'Shut down incorrectly' steps below - if you have had behat running successfully in the past
 
* First try the 'Shut down incorrectly' steps below - if you have had behat running successfully in the past
 
+
* Go to the script: <syntaxhighlight lang="shell" inline>mahara/test/behat/mahara_behat.sh</syntaxhighlight> script.  
* Go to the script: mahara/test/behat/mahara_behat.sh script.  
+
* Update the version
 
 
* Update the version (around lines number 95 and 96 of the sh file):
 
<pre>
 
* SELENIUM_VERSION_MAJOR=x.xx
 
* SELENIUM_VERSION_MINOR=x
 
</pre>
 
 
 
 
* Download the latest jar file from: http://www.seleniumhq.org/download/  
 
* Download the latest jar file from: http://www.seleniumhq.org/download/  
 +
* Save the jar file to <syntaxhighlight lang="bash" inline>/mahara/test/behat/</syntaxhighlight>
 +
* Auto update by running: <syntaxhighlight lang="bash" inline>sudo -u www-data ./test/behat/mahara_behat.sh rundebug @tag</syntaxhighlight>
  
* Save the jar file to /mahara/test/behat/
+
To find out if Selenium has updated to a new version, go to http://www.seleniumhq.org/download/ and see the version of the Selenium Server is available.
 
 
* Auto update by running:
 
<pre>
 
sudo -u www-data ./test/behat/mahara_behat.sh rundebug @tag
 
</pre>
 
 
 
To find out if Selenium has updated to a new version, go to http://www.seleniumhq.org/download/ and see version of the Selenium Server is available.
 
  
 
You might need to do some more debugging. But this information should point you in the right direction.
 
You might need to do some more debugging. But this information should point you in the right direction.
 +
==== How to update the version ====
 +
Locate and update the following variables in <syntaxhighlight lang="shell" inline>mahara/test/behat/mahara_behat.sh</syntaxhighlight>.
 +
<syntaxhighlight lang="shell" line start="140">
 +
SELENIUM_VERSION_MAJOR=X.XX
 +
SELENIUM_VERSION_MINOR=X
 +
</syntaxhighlight>
  
 
=== Shut down incorrectly ===
 
=== Shut down incorrectly ===
  
Another bug with Selenium is if you are using Moodle or running Behat from two diff versions of Mahara is sometimes the Selenium Server doesn't shut down properly. You will know this is happening if you've run the tests and it says at the end "Selenium server not not shut down" or it just wont say that's it has shut down. It will be written the same way it does when Selenium starts up when you run the tests.
+
Another bug with Selenium is if you are using Moodle or running Behat from two diff versions of Mahara is sometimes the Selenium Server doesn't shut down properly. You will know this is happening if you've run the tests and it says at the end "Selenium server not not shut down" or it just won't say that's it has shut down. It will be written the same way it does when Selenium starts up when you run the tests.
  
 
Eg: Composer installing... PHP server running, Selenium running.
 
Eg: Composer installing... PHP server running, Selenium running.
At the end of the tests running it says the same thing.. PHP server shut down... Selenium shut down.  
+
At the end of the tests running, it says the same thing.. the PHP server shut down... Selenium shut down.  
  
 
When you try to run the tests it may also say "Selenium server is not running or could not be run" or something similar. This is because there are 2 running and it's getting confused
 
When you try to run the tests it may also say "Selenium server is not running or could not be run" or something similar. This is because there are 2 running and it's getting confused
Line 107: Line 100:
 
   kill 6590
 
   kill 6590
  
However don't kill the one that say's --color=auto selenium you should kill the other one.
+
However, don't kill the one that says --color=auto selenium you should kill the other one.
  
 
Then run the command again to make sure that the server was killed
 
Then run the command again to make sure that the server was killed
Line 115: Line 108:
 
This will just kill selenium running so you can finish what you are working on and run the test properly.
 
This will just kill selenium running so you can finish what you are working on and run the test properly.
  
Once you have done that, if you are a dev take a look into the /test/behat/mahara_behat.sh script because there will be something up with the way that selenium is shutting down. Also check that this issue isn't happening with Jenkins. However, keep in mind that Jenkins runs headless.
+
Once you have done that, if you are a dev take a look into the /test/behat/mahara_behat.sh script because there will be something up with the way that selenium is shutting down. Also, check that this issue isn't happening with Jenkins. However, keep in mind that Jenkins runs headless.
  
 
If you aren't a dev then go tell one of the devs that it's not shutting down properly. Or you can do this by writing a bug on Launchpad here: https://bugs.launchpad.net/mahara/+filebug
 
If you aren't a dev then go tell one of the devs that it's not shutting down properly. Or you can do this by writing a bug on Launchpad here: https://bugs.launchpad.net/mahara/+filebug
Line 125: Line 118:
 
=== Can't log in ===
 
=== Can't log in ===
  
Firefox upgrades and when you run the tests Firefox wont pop up with the Mahara instance in it. Another issue is the window opening but Behat not being able to run the test properly and it will say that you cant log in. It will run the first log in steps (say as an admin) but it wont pass the next step (to verify that the admin can be logged in) and if you actually look at Behat running (requires you to not be running it headless) then you will see it's not actually logging in.
+
Firefox upgrades and when you run the tests Firefox won't pop up with the Mahara instance in it. Another issue is the window opening but Behat not being able to run the test properly and it will say that you cant log in. It will run the first log in steps (say as an admin) but it won't pass the next step (to verify that the admin can be logged in) and if you actually look at Behat running (requires you to not be running it headless) then you will see it's not actually logging in.
  
 
LINK TO ROLLBACK FIREFOX VERSION 28
 
LINK TO ROLLBACK FIREFOX VERSION 28
 
   http://askubuntu.com/questions/500644/how-to-downgrade-firefox-from-30-to-28
 
   http://askubuntu.com/questions/500644/how-to-downgrade-firefox-from-30-to-28
  
ROLL BACK TO VERSION 42
+
ROLLBACK TO VERSION 42
 
+
<syntaxhighlight lang="bash">
  sudo apt-get purge firefox
+
sudo apt-get purge firefox
 
+
wget https://launchpad.net/~ubuntu-mozilla-security/+archive/ubuntu/ppa/+build/8220812/+files/firefox_42.0+build2-0ubuntu0.14.04.1_amd64.deb
  wget https://launchpad.net/~ubuntu-mozilla-security/+archive/ubuntu/ppa/+build/8220812/+files/firefox_42.0+build2-0ubuntu0.14.04.1_amd64.deb
+
sudo dpkg -i firefox_42.0+build2-0ubuntu0.14.04.1_amd64.deb
 
+
sudo apt-mark hold firefox
  sudo dpkg -i firefox_42.0+build2-0ubuntu0.14.04.1_amd64.deb
+
</syntaxhighlight>
 
+
Follow instructions in the link above to unhold and update.
  sudo apt-mark hold firefox
 
 
 
Follow instructions in link above to unhold and update.
 
 
 
 
 
 
 
 
  
 
TO FIX:
 
TO FIX:
  
Generally Firefox and Selenium are heavily entwined. When Firefox breaks, go to http://www.seleniumhq.org/download/ and download the newer version of Selenium. When one upgarades it's usually best to upgrade the other at the same time so they are compatible.
+
Generally, Firefox and Selenium are heavily entwined. When Firefox breaks, go to http://www.seleniumhq.org/download/ and download the newer version of Selenium. When one upgrades it's usually best to upgrade the other at the same time so they are compatible.
  
 
If all else fails then revert back to the previous stable version.   
 
If all else fails then revert back to the previous stable version.   
  
I change directory to downloads and run this in a separate terminal. This requires you to have the previous version of Firefox downloaded and still sitting in downloads.
+
I change the directory to downloads and run this in a separate terminal. This requires you to have the previous version of Firefox downloaded and still sitting in downloads.
 +
<syntaxhighlight lang="bash">
 
java -jar selenium-server-standalone-2.44.0.jar -Dwebdriver.firefox.bin=/home/jinelleb/Downloads/firefox/firefox
 
java -jar selenium-server-standalone-2.44.0.jar -Dwebdriver.firefox.bin=/home/jinelleb/Downloads/firefox/firefox
 
+
</syntaxhighlight>
  
 
== PHP server issues ==
 
== PHP server issues ==
Line 160: Line 147:
 
=== Server not closing ===
 
=== Server not closing ===
  
Same as the Selenium server issues, you can get errors with the PHP server not closing or running multiple servers. You can tell this because it will give a error message when you try run the tests. It will generally throw an error with the key words "localhost 8000" or "PHP server".
+
Same as the Selenium server issues, you can get errors with the PHP server not closing or running multiple servers. You can tell this because it will give an error message when you try to run the tests. It will generally throw an error with the keywords "localhost 8000" or "PHP server".
  
There could be multiple things causing this issue including the config file not point to the right host name.
+
There could be multiple things causing this issue including the config file not point to the right hostname.
  
  
 
TO FIX:
 
TO FIX:
  
Grep search for php server and kill the process ID.
+
Grep search for PHP server and kill the process ID.
  
Go to your mahara config.php and read the Behat section and make sure that everything is pointing to the right place.  
+
Go to your Mahara config.php and read the Behat section and make sure that everything is pointing to the right place.  
  
If these fail then if you aren't a dev file a bug here https://bugs.launchpad.net/mahara/+filebug with the summary Behat: description of what's happening when you try run tests.
+
If these fail then if you aren't a dev file a bug here https://bugs.launchpad.net/mahara/+filebug with the summary Behat: description of what's happening when you try to run tests.
  
If you are a dev then you need to look into the mahara_behat script and check the php server is closing properly. A way to debug is to run a test locally and check if the php server is starting up and shutting down properly. It will say in the terminal if you run "rundebug". I think it runs with just run as well.
+
If you are a dev then you need to look into the mahara_behat script and check the PHP server is closing properly. A way to debug is to run a test locally and check if the PHP server is starting up and shutting down properly. It will say in the terminal if you run "rundebug". I think it runs with just run as well.
  
 
== Composer issue ==
 
== Composer issue ==
Line 189: Line 176:
 
* Delete the lock file (mahara/external/composerlock.php) and then run Behat again.
 
* Delete the lock file (mahara/external/composerlock.php) and then run Behat again.
  
== Debugg steps that are failing ==
+
== Debug steps that are failing ==
  
 
=== Test keeps failing ===
 
=== Test keeps failing ===
Line 197: Line 184:
 
TO FIX:
 
TO FIX:
  
For a start when you run the test run using the debug option instead of just "run" this will give you a more human readable output. The step that is failing will go red.
+
For a start when you run the test run using the debug option instead of just "run" this will give you a more human-readable output. The step that is failing will go red.
 
EG:
 
EG:
  
Line 213: Line 200:
  
 
== Dropping only the behat_ tables ==
 
== Dropping only the behat_ tables ==
 
+
=== PostgreSQL ===
 
If you want to drop your behat_ tables but not the whole database, first connect to the db:<br />
 
If you want to drop your behat_ tables but not the whole database, first connect to the db:<br />
  
('''NB:''' if the database you're using is not called mahara-master, replace with your db's name)
+
('''NB:''' if the database you're using is not called mahara-main, replace with your DB's name)
   psql -U maharauser -h localhost mahara-master
+
   psql -U maharauser -h localhost mahara-main
  
 
When you see the db prompt
 
When you see the db prompt
   mahara-master=#
+
   mahara-main=#
 
you can run this command:
 
you can run this command:
 
+
<syntaxhighlight lang="sql">
  DO
+
DO
  $do$
+
$do$
  DECLARE
+
DECLARE
  _tbl text;
+
    _tbl text;
  BEGIN
+
BEGIN
  FOR _tbl  IN
+
FOR _tbl  IN
 
     SELECT quote_ident(table_schema) || '.'
 
     SELECT quote_ident(table_schema) || '.'
         || quote_ident(table_name)     -- escape identifier and schema-qualify!
+
         || quote_ident(table_name)         -- escape identifier and schema-qualify!
 
     FROM  information_schema.tables
 
     FROM  information_schema.tables
 
     WHERE  table_name LIKE 'behat_' || '%'  -- your table name prefix
 
     WHERE  table_name LIKE 'behat_' || '%'  -- your table name prefix
 
     AND    table_schema NOT LIKE 'pg_%'    -- exclude system schemas
 
     AND    table_schema NOT LIKE 'pg_%'    -- exclude system schemas
  LOOP
+
LOOP
  -- RAISE NOTICE '%',
+
    -- RAISE NOTICE '%',                      
 
     EXECUTE
 
     EXECUTE
  'DROP TABLE ' || _tbl || ' CASCADE';
+
    'DROP TABLE ' || _tbl || ' CASCADE';
  END LOOP;
+
END LOOP;
  END
+
END
  $do$;
+
$do$;
 
+
</syntaxhighlight>
 
If you just want to see what it will delete before running it change these two lines
 
If you just want to see what it will delete before running it change these two lines
 +
<syntaxhighlight lang="sql">
 
   -- RAISE NOTICE '%',
 
   -- RAISE NOTICE '%',
 
     EXECUTE
 
     EXECUTE
 
+
</syntaxhighlight>
 
to
 
to
 
+
<syntaxhighlight lang="sql">
 
   RAISE NOTICE '%',
 
   RAISE NOTICE '%',
 
   --  EXECUTE
 
   --  EXECUTE
 +
</syntaxhighlight>
  
Once you've run the commands, use Ctrl D to exit the db.
+
Once you've run the commands, use Ctrl D to exit the DB.
  
== Having trouble getting you Behat testing started up again ==
+
=== MySQL/MariaDB ===
 +
 
 +
Connect to the DB as a user that can drop tables and then run the following:
 +
<syntaxhighlight lang="sql">
 +
SET SESSION group_concat_max_len = 1000000;
 +
SELECT CONCAT(
 +
  'SET FOREIGN_KEY_CHECKS = 0; DROP TABLE IF EXISTS ',
 +
  GROUP_CONCAT(table_name),
 +
  '; SET FOREIGN_KEY_CHECKS = 1;'
 +
) AS statement
 +
FROM information_schema.tables
 +
WHERE table_schema = 'maharatest2'
 +
AND table_name LIKE 'behat_%';
 +
</syntaxhighlight>
 +
This will generate an SQL statement.  Copy and paste that into your db session and this will clean up the tables.
 +
 
 +
== Having trouble getting your Behat testing started up again ==
  
 
Sometimes if Behat crashing in an unexpected way it can be a bit of a bother to get it started up again.
 
Sometimes if Behat crashing in an unexpected way it can be a bit of a bother to get it started up again.
Line 350: Line 355:
  
 
===Stopping Chrome/Chromium auto-updating and undoing all your hard work===
 
===Stopping Chrome/Chromium auto-updating and undoing all your hard work===
 +
You'll need to lock the version you just installed, to save having to do this all again soon. I can help with Ubuntu:
 +
'''For Chromium:'''
 +
<syntaxhighlight lang="bash">sudo apt-mark hold chromium-browser</syntaxhighlight>
 +
and, depending on the codecs you installed,
 +
<syntaxhighlight lang="bash">sudo apt-mark hold chromium-codecs-ffmpeg-extra</syntaxhighlight>
 +
or
 +
<syntaxhighlight lang="bash">sudo apt-mark hold chromium-codecs-ffmpeg</syntaxhighlight>
 +
'''For Chrome:'''
 +
<syntaxhighlight lang="bash">sudo apt-mark hold google-chrome</syntaxhighlight>
 +
 +
== Other ==
 +
 +
=== Requested config file does not exist ===
 +
Drop the behat tables
 +
 +
sudo -u postgres dropdb behat_master && sudo -u postgres createdb -Omaharauser behat_master
 +
 +
weirdly enough it seems connected to the master-behat db too, so if that doens't work
 +
 +
sudo -u postgres dropdb mahara-master & sudo -u postgres createdb -Omaharauser mahara-master
 +
 +
and this time when you ./test/behat/mahara_behat.sh rundebugheadless test.feature --format=pretty
  
You'll need to lock the version you just installed, to save having to do this all again soon. I can help with Ubuntu:
+
The site should install at the start of your behat -- this is a good sign ;)
 +
 
 +
=== window.isMaharaPageReady is not a function ===
 +
Check folders (/var/lib/maharadata/master_behat) exist and give permissions to the folders again.
  
'''For Chromium:'''
+
=== If the site stops working saying it's unavailable ===
  sudo apt-mark hold chromium-browser
+
It might mean that it didn't shut down properly (accidentally or deliberate interruption), before trying to hack your way around this problem, try to run a basic test to let the normal process shut down behat and give you your local site back
  
and, depending on the codecs you installed,
+
=== Try running a test to shut it down normally ===
  
sudo apt-mark hold chromium-codecs-ffmpeg-extra
+
An example of a basic test: <syntaxhighlight lang="shell" inline>run.feature</syntaxhighlight>
  
or
+
<syntaxhighlight lang="Gherkin">
 +
Feature: Test
 +
    Shut down behat normal process by finishing a test
 +
Background:
 +
    Given the following site settings are set:
 +
    | field | value |
 +
    | licensemetadata | 0 |
 +
 +
    Given the following "users" exist:
 +
    | username | password | email            | firstname | lastname | institution | authname | role |
 +
    | UserA    | Kupuh1pa!| [email protected] | Painterio | Mahara  | mahara      | internal | member |
 +
 +
    And the following "pages" exist:
 +
    # Available fields: title*, description, ownertype*, ownername*, layout, tags
 +
    | title        | description  | ownertype | ownername |
 +
    | Page One A    | UserA Page 1 | user      | UserA    |
 +
 
 +
Scenario: login as different users to see blocktypes and interaction
 +
    Given I log in as "UserA" with password "Kupuh1pa!"
 +
</syntaxhighlight>
  
sudo apt-mark hold chromium-codecs-ffmpeg
+
Then run <syntaxhighlight lang="shell" inline>./test/behat/mahara_behat.sh rundebug run.feature --format=pretty</syntaxhighlight>
  
 +
=== If that doesn't work ===
 +
try sudo chmod -R a+wrX /var/lib/maharadata/master_behat
  
'''For Chrome:'''
+
This happens when you interrupt behat 🤭 It doesn't like interruptions -_-
  sudo apt-mark hold google-chrome
+
If it's still not fixed, try...
 +
* running a test again and *let it finish 😉* It might shut behat down properly and give you your local Mahara back to you 🌈
 +
* deleting your local mahara database (not the behat one) and installing a new site - something about the set up of Behat looks at your local instance before doing behat stuff... can't explain it but it fixes most problems when the error message is not clear.
  
== Other ==
 
  The requested config file does not exist -> drop the behat tables
 
 
 
  unknown error: window.isMaharaPageReady is not a function -> check folders (/var/lib/maharadata/master_behat) exist give permissions to folders again with behat steps
 
 
 
  
 
[[Category: Behat]]
 
[[Category: Behat]]

Latest revision as of 14:09, 14 February 2022

Errors when using Behat

This is a place where you can log errors and come to fix yours. If you log an error please be sure to put down how you fixed it

Permission issues

If you are getting a permission denied:

  • [WAR] 7e (/var/www/maharacode/mahara/external/vendor/behat/gherkin/src/Behat/Gherkin/Cache/FileCache.php:88) file_put_contents(/tmp/behat_gherkin_cache/412/7852eac8b52e46380d27aa7d2702426c.feature.cache): failed to open stream: Permission denied

Make sure the following files are owned by the apache user:

  • sitedata (directory in your $cfg->behat_dataroot)
  • /tmp/behat_rerun_cache
  • /tmp/behat_gherkin_cache

If not, update their permissions:

  • sudo chown -R www-data.www-data /tmp/behat_gherkin_cache
  • sudo chown -R www-data.www-data /tmp/behat_rerun_cache
  • sudo chown -R www-data.www-data sitedata (directory in your $cfg->behat_dataroot)
  • For Ubuntu, apache runs with user www-data
  • For Centos, apache runs with user apache

Behat tests cannot run unless $cfg->behat_wwwroot, $cfg->behat_dataroot, and $cfg->behat_dbprefix are defined in config.php

Make sure the following following lines of code are in your config.php file

$cfg->behat_dbprefix = 'behat_'; // must not empty
$cfg->behat_dataroot = "/var/lib/maharadata/master_behat"; // Behat's copy of maharadata
$cfg->behat_wwwroot = 'http://localhost:8000'; // Must be this
$cfg->behat_selenium2 = "http://127.0.0.1:4444/wd/hub"; // Must be this

Selenium issues

Version of OpenJDK is too new

In some cases, depending on the version of Ubuntu you have, the wrong version of OpenJDK will be installed. Selenium only runs with OpenJDK 8, but Ubuntu Bionic tends to install OpenJDK 11 (especially if 'default-jre' was installed)

If you run into issues where the script says it has started Selenium, but that Behat throws an exception because it thinks Selenium has not started, specifically an error thrown in BehatHooks before_scenario saying 'Selenium server is not running, you need to start it to run tests that involve Javascript.', check the JRE installed:

java -version

If it says you're using OpenJDK 10 or 11 (which can ship with Ubuntu 18.04 depending on which service pack is used), this needs to be uninstalled first:

sudo apt remove --purge openjdk*

Once this is removed, you will need to re-run the OpenJDK 8 installation:

$ sudo apt install openjdk-8-jre-headless

Server not started

When Selenium upgrades causing firefox to try run headless and not open firefox. It can also throw an error when you try to run the test. It will mention something about the Selenium server not being able to start up.

TO FIX:

  • First try the 'Shut down incorrectly' steps below - if you have had behat running successfully in the past
  • Go to the script: mahara/test/behat/mahara_behat.sh script.
  • Update the version
  • Download the latest jar file from: http://www.seleniumhq.org/download/
  • Save the jar file to /mahara/test/behat/
  • Auto update by running: sudo -u www-data ./test/behat/mahara_behat.sh rundebug @tag

To find out if Selenium has updated to a new version, go to http://www.seleniumhq.org/download/ and see the version of the Selenium Server is available.

You might need to do some more debugging. But this information should point you in the right direction.

How to update the version

Locate and update the following variables in mahara/test/behat/mahara_behat.sh.

140SELENIUM_VERSION_MAJOR=X.XX
141SELENIUM_VERSION_MINOR=X

Shut down incorrectly

Another bug with Selenium is if you are using Moodle or running Behat from two diff versions of Mahara is sometimes the Selenium Server doesn't shut down properly. You will know this is happening if you've run the tests and it says at the end "Selenium server not not shut down" or it just won't say that's it has shut down. It will be written the same way it does when Selenium starts up when you run the tests.

Eg: Composer installing... PHP server running, Selenium running. At the end of the tests running, it says the same thing.. the PHP server shut down... Selenium shut down.

When you try to run the tests it may also say "Selenium server is not running or could not be run" or something similar. This is because there are 2 running and it's getting confused


TO FIX:

Temporary fix, go to the terminal and type

ps aux | grep selenium

And then kill the process ID. This is the first number after your username.

Eg:

<username>@<computername>:~$ ps aux | grep selenium
<username> 6590 0.0 0.0 12960 936 pts/26 S+ 10:31 0:00 grep --color=auto selenium

So the command will be

 kill 6590

However, don't kill the one that says --color=auto selenium you should kill the other one.

Then run the command again to make sure that the server was killed

 ps aux | grep selenium 


This will just kill selenium running so you can finish what you are working on and run the test properly.

Once you have done that, if you are a dev take a look into the /test/behat/mahara_behat.sh script because there will be something up with the way that selenium is shutting down. Also, check that this issue isn't happening with Jenkins. However, keep in mind that Jenkins runs headless.

If you aren't a dev then go tell one of the devs that it's not shutting down properly. Or you can do this by writing a bug on Launchpad here: https://bugs.launchpad.net/mahara/+filebug In the summary write Behat: tell them what the issue is here. Just so they can tell instantly it's a Behat issue you are having.

Firefox issues

Can't log in

Firefox upgrades and when you run the tests Firefox won't pop up with the Mahara instance in it. Another issue is the window opening but Behat not being able to run the test properly and it will say that you cant log in. It will run the first log in steps (say as an admin) but it won't pass the next step (to verify that the admin can be logged in) and if you actually look at Behat running (requires you to not be running it headless) then you will see it's not actually logging in.

LINK TO ROLLBACK FIREFOX VERSION 28

 http://askubuntu.com/questions/500644/how-to-downgrade-firefox-from-30-to-28

ROLLBACK TO VERSION 42

sudo apt-get purge firefox
wget https://launchpad.net/~ubuntu-mozilla-security/+archive/ubuntu/ppa/+build/8220812/+files/firefox_42.0+build2-0ubuntu0.14.04.1_amd64.deb
sudo dpkg -i firefox_42.0+build2-0ubuntu0.14.04.1_amd64.deb
sudo apt-mark hold firefox

Follow instructions in the link above to unhold and update.

TO FIX:

Generally, Firefox and Selenium are heavily entwined. When Firefox breaks, go to http://www.seleniumhq.org/download/ and download the newer version of Selenium. When one upgrades it's usually best to upgrade the other at the same time so they are compatible.

If all else fails then revert back to the previous stable version.

I change the directory to downloads and run this in a separate terminal. This requires you to have the previous version of Firefox downloaded and still sitting in downloads.

java -jar selenium-server-standalone-2.44.0.jar -Dwebdriver.firefox.bin=/home/jinelleb/Downloads/firefox/firefox

PHP server issues

Server not closing

Same as the Selenium server issues, you can get errors with the PHP server not closing or running multiple servers. You can tell this because it will give an error message when you try to run the tests. It will generally throw an error with the keywords "localhost 8000" or "PHP server".

There could be multiple things causing this issue including the config file not point to the right hostname.


TO FIX:

Grep search for PHP server and kill the process ID.

Go to your Mahara config.php and read the Behat section and make sure that everything is pointing to the right place.

If these fail then if you aren't a dev file a bug here https://bugs.launchpad.net/mahara/+filebug with the summary Behat: description of what's happening when you try to run tests.

If you are a dev then you need to look into the mahara_behat script and check the PHP server is closing properly. A way to debug is to run a test locally and check if the PHP server is starting up and shutting down properly. It will say in the terminal if you run "rundebug". I think it runs with just run as well.

Composer issue

Dependencies

Originally composer wasn't pegged. Now it has been you can often get a yellow error when you run Behat that mentions composer dependencies being out of date. It will say "Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies".


TO FIX:

  • Run update to update them.

Or

  • Delete the lock file (mahara/external/composerlock.php) and then run Behat again.

Debug steps that are failing

Test keeps failing

When you are trying to write a test but the test keeps failing on a step and you don't know why it's failing. You need to be able to find a way to hold the window open so you can see what's happening.

TO FIX:

For a start when you run the test run using the debug option instead of just "run" this will give you a more human-readable output. The step that is failing will go red. EG:

./test/behat/mahara_behat.sh rundebug @youtagthatyouwanttorun

To hold the window open so you can add a step right before the one breaking saying And I wait "30" seconds. Another way of holding the window open for a longer period of time is to click on the terminal in the and hit ctrl+z this will pause the test from running right where it is. You then need to type fg to bring the test back into the foreground and let it keep running till the end.

Stopping a test

A test has started running and you need to stop it in the middle of it.

TO FIX:

Press ctrl+c this will stop it from running

Dropping only the behat_ tables

PostgreSQL

If you want to drop your behat_ tables but not the whole database, first connect to the db:

(NB: if the database you're using is not called mahara-main, replace with your DB's name)

 psql -U maharauser -h localhost mahara-main

When you see the db prompt

 mahara-main=#

you can run this command:

DO
$do$
DECLARE
    _tbl text;
BEGIN
FOR _tbl  IN
    SELECT quote_ident(table_schema) || '.'
        || quote_ident(table_name)          -- escape identifier and schema-qualify!
    FROM   information_schema.tables
    WHERE  table_name LIKE 'behat_' || '%'  -- your table name prefix
    AND    table_schema NOT LIKE 'pg_%'     -- exclude system schemas
LOOP
    -- RAISE NOTICE '%',                        
    EXECUTE
    'DROP TABLE ' || _tbl || ' CASCADE';
END LOOP;
END
$do$;

If you just want to see what it will delete before running it change these two lines

  -- RAISE NOTICE '%',
    EXECUTE

to

  RAISE NOTICE '%',
  --  EXECUTE

Once you've run the commands, use Ctrl D to exit the DB.

MySQL/MariaDB

Connect to the DB as a user that can drop tables and then run the following:

SET SESSION group_concat_max_len = 1000000;
SELECT CONCAT(
  'SET FOREIGN_KEY_CHECKS = 0; DROP TABLE IF EXISTS ',
  GROUP_CONCAT(table_name),
  '; SET FOREIGN_KEY_CHECKS = 1;'
) AS statement
FROM information_schema.tables 
WHERE table_schema = 'maharatest2'
AND table_name LIKE 'behat_%';

This will generate an SQL statement. Copy and paste that into your db session and this will clean up the tables.

Having trouble getting your Behat testing started up again

Sometimes if Behat crashing in an unexpected way it can be a bit of a bother to get it started up again.

If you follow these 3 steps it should fire up again.

  1. Kill any still running Selenium processes
  2. Drop the behat prefixed tables from the database
  3. Delete the 'behat' subdirectory of the $cfg->behat_dataroot directory

Doing things in this order should make sure that localhost ports in use are released, that the behat site is not installed and that the behat dataroot flag that indicates the behat system is setup is also removed - so your system is set back to the point before you ever ran behat for the first time.

Tests failing erratically since Chrome/Chromium upgrade to version 61

Note: Behat and selenium were upgraded in May 2018 and so now latest version of Chrome/Chromium should work fine again

If you have suddenly noticed tests are failing to find visible items on the page sometime in September 2017, you're not alone! The latest stable version of Chrome/Chromium, 61.0.3163.100, is playing havoc with webdriver's ability to find stuff. Hopefully this issue will be fixed in a subsequent release, but in the meantime you can work around it by installing the previous stable version.

First of all confirm whether the problem you are having is related to this issue.

For Chromium, run:

 chromium-browser --version

or for chrome, run:

 google-chrome --version

If the answer is:

Chromium 61.0.3163.100

or

Google Chrome 61.0.3163.100


it looks like this is your problem. The previous stable version of Chrome/Chromium is 60.0.3112.113.

If you don't have a copy of the old version saved already, you will need to search for it online. Try searching for your OS name with Chrome/Chromium 60.0.3112.113.

For Chromium on Ubuntu, I can help you.

For 14.04, go to:

https://launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/13295237

And for 16.04, go to:

https://launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/13295233

And for 17.04, go to:

https://launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/13295225

You'll need to download two packages:

chromium-browser_<version><your OS>_amd64.deb

and

chromium-codecs-ffmpeg_<version><your OS>_amd64.deb

or

chromium-codecs-ffmpeg-extra_<version><your OS>_amd64.deb

(If you don't know which codecs you need, just grab the 'extra' version.)

You've downloaded the file/s. Now what?

Back on your system, you'll need to uninstall the current version before you try to install the older version. In Ubuntu:

 sudo apt-get remove chromium-browser
 sudo apt-get remove chromium-codecs-ffmpeg-extra
 sudo apt-get remove chromium-codecs-ffmpeg

(One of the codec packages will tell you it wasn't installed in the first place. Don't worry about that.)

Chrome doesn't have a separate codecs package, so for Chrome:

 sudo apt-get remove google-chrome

The easiest way to install is to go to the directory you downloaded the packages in and double click to install. For Chromium, you will need to install the codec package before the browser package or you'll get an error.

Installing old packages

Go to the directory where you downloaded them and install the codecs first:

sudo dpkg -i chromium-codecs-ffmpeg-extra_60.0.3112.113-0ubuntu0.16.04.1298_amd64.deb

Then the browser:

sudo dpkg -i chromium-browser_60.0.3112.113-0ubuntu0.16.04.1298_amd64.deb

Now you should be good to run your tests again without things breaking... (fingers crossed).

Stopping Chrome/Chromium auto-updating and undoing all your hard work

You'll need to lock the version you just installed, to save having to do this all again soon. I can help with Ubuntu: For Chromium:

sudo apt-mark hold chromium-browser

and, depending on the codecs you installed,

sudo apt-mark hold chromium-codecs-ffmpeg-extra

or

sudo apt-mark hold chromium-codecs-ffmpeg

For Chrome:

sudo apt-mark hold google-chrome

Other

Requested config file does not exist

Drop the behat tables

sudo -u postgres dropdb behat_master && sudo -u postgres createdb -Omaharauser behat_master

weirdly enough it seems connected to the master-behat db too, so if that doens't work

sudo -u postgres dropdb mahara-master & sudo -u postgres createdb -Omaharauser mahara-master

and this time when you ./test/behat/mahara_behat.sh rundebugheadless test.feature --format=pretty

The site should install at the start of your behat -- this is a good sign ;)

window.isMaharaPageReady is not a function

Check folders (/var/lib/maharadata/master_behat) exist and give permissions to the folders again.

If the site stops working saying it's unavailable

It might mean that it didn't shut down properly (accidentally or deliberate interruption), before trying to hack your way around this problem, try to run a basic test to let the normal process shut down behat and give you your local site back

Try running a test to shut it down normally

An example of a basic test: run.feature

 Feature: Test
     Shut down behat normal process by finishing a test
 Background:
     Given the following site settings are set:
     | field | value |
     | licensemetadata | 0 |
 
     Given the following "users" exist:
     | username | password | email             | firstname | lastname | institution | authname | role |
     | UserA    | Kupuh1pa!| [email protected] | Painterio | Mahara   | mahara      | internal | member |
 
     And the following "pages" exist:
     # Available fields: title*, description, ownertype*, ownername*, layout, tags
     | title         | description   | ownertype | ownername |
     | Page One A    | UserA Page 1 | user      | UserA     |

 Scenario: login as different users to see blocktypes and interaction
     Given I log in as "UserA" with password "Kupuh1pa!"

Then run ./test/behat/mahara_behat.sh rundebug run.feature --format=pretty

If that doesn't work

try sudo chmod -R a+wrX /var/lib/maharadata/master_behat

This happens when you interrupt behat 🤭 It doesn't like interruptions -_- If it's still not fixed, try...

  • running a test again and *let it finish 😉* It might shut behat down properly and give you your local Mahara back to you 🌈
  • deleting your local mahara database (not the behat one) and installing a new site - something about the set up of Behat looks at your local instance before doing behat stuff... can't explain it but it fixes most problems when the error message is not clear.