Developer Area/Contributing Code/Troubleshooting your Gerrit connection: Difference between revisions

Latest revision as of 07:49, 10 February 2023

Troubleshooting your Gerrit connection

This page contains instructions to help you troubleshoot your connection to the Mahara gerrit site, https://reviews.mahara.org . This page may be helpful if you've been following the instructions on these pages but receiving errors:

These instructions assume you are using Ubuntu Linux. They will probably be almost equally applicable to any type of Linux, somewhat helpful in Unix or Mac OS X, and of limited help in Windows.

Issue 1: Problems with "git fetch gerrit"

The first step to figuring out what's wrong with your connection to gerrit is to run this command. It should be run in your terminal, while in the directory where your Mahara git repository is located.

git fetch gerrit

If this is successful, you will see output similar to this, fetching the list of branches from Gerrit.

From ssh://reviews.mahara.org:29418/mahara
 * [new branch]      1.0_STABLE -> gerrit/1.0_STABLE
 * [new branch]      1.10_STABLE -> gerrit/1.10_STABLE
 * [new branch]      1.1_STABLE -> gerrit/1.1_STABLE
 * [new branch]      1.2_STABLE -> gerrit/1.2_STABLE
 * [new branch]      1.3_STABLE -> gerrit/1.3_STABLE
 * [new branch]      1.4_STABLE -> gerrit/1.4_STABLE
 * [new branch]      1.5_STABLE -> gerrit/1.5_STABLE
 * [new branch]      1.6_STABLE -> gerrit/1.6_STABLE
 * [new branch]      1.7_STABLE -> gerrit/1.7_STABLE
 * [new branch]      1.8_STABLE -> gerrit/1.8_STABLE
 * [new branch]      1.9_STABLE -> gerrit/1.9_STABLE
 * [new branch]      15.04_STABLE -> gerrit/15.04_STABLE
 * [new branch]      15.10_STABLE -> gerrit/15.10_STABLE
 * [new branch]      master     -> gerrit/master

If you see that, then you're able to pull successfully. If you see an error, see if it's one of these:

Error 1.1: "fatal: not a git repository"

Do you see this error when you do git fetch gerrit?

fatal: Not a git repository (or any parent up to mount point /var)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

This indicates that you're not running the git command in the location where your mahara git repository is located. The mahara repo will be a directory sitting inside the directory where you did the "git clone" command earlier. It will be named "mahara" unless you renamed it. It will have an "invisible" directory called ".git" in it, which you can see if you run the command "ls -a" while in the directory.

If you are following the instructions on the Developer Environment setup page, then your Mahara git repository will be the directory ~/code/mahara. The ~ is a linux shortcut for your home directory; e.g. /home/YOURUSERNAME.

If you are following the Developer Area/Contributing Code instructions, we don't specifically tell you where to put your Mahara codebase, so you'll need to:

- 1.1.1. Remember where you put your local mahara git repo.
- 1.1.2. Locate your local mahara git repo
- 1.1.3. Just clone a new local mahara git repo

Option 1.1.1: Remembering where you put your local mahara git repo.

I don't really have any good advice for this one. ;)

Option 1.1.2: Finding your local Mahara repo

(If you are very new to working with the command-line, it may be helpful to you to know that you can use the cd command to move around into different directories, and cd .. to move into the parent directory of your current directory, and pwd to print your current directory.)

The easiest way to search for Mahara's location on your computer is to use the find command, which will search the files on your computer in real-time. This can take a long time if you search your entire hard drive, so it is best to limit the search to a few specific likely locations. Mahara contains two files called "mahara.php"; searching for those should show you where it's located. (I add "2> /dev/null" to the end of the command to avoid warning messages about being unable to search private directories.)

find ~ /var/www /Library/WebServer /srv/www -name "mahara.php" -type f 2> /dev/null

If this is successful, then you'll see your Mahara directory. It'll be the directory which contains the "htdocs/lib/mahara.php" and "htdocs/lang/en.utf8/mahara.php"

aaronw@vegas:~$ find ~ /var/www /Library/WebServer /srv/www -name "mahara.php" -type f 2> /dev/null
/home/aaronw/code/mahara/htdocs/lib/mahara.php
/home/aaronw/code/mahara/htdocs/lang/en.utf8/mahara.php
# My mahara repo is /home/aaronw/code/mahara

If that doesn't work, then you may have put your Mahara repo somewhere unusual. You can try running find / -name "mahara.php" -type f 2> /dev/null to search your entire filesystem, but it may take several minutes. If you have root access, you can also try updating your "locate" index and using the locate command.

aaronw@vegas:~$ sudo updatedb
[sudo] password for aaronw: <enter your password>
# ... wait several minutes
aaronw@vegas:~$ locate mahara.php

Option 1.1.3. Clone a new local copy of mahara

It doesn't take long to clone a new copy of Mahara, so sometimes it's best just to do that. (The only downside is you might stumble across your lost copy later and confuse the two.) Just follow the directions you used earlier, on one of the links listed at the top of this page.

Error 1.2: Permission denied (publickey).

When you run git fetch, do you see this error?

(master)aaronw@vegas:~/code/mahara$ git fetch gerrit
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This means you are failing to connect to Mahara via SSH. There can be several causes for this:

Step 1. Verify you have the right URL and username

In the terminal, run the "git remote-v" command from within your Mahara repo directory, and make sure the listing for the gerrit repo looks like this:

aaron@vegas:~/code/mahara$ git remote -v
gerrit	ssh://[email protected]:29418/mahara (fetch)
gerrit	ssh://[email protected]:29418/mahara (push)
origin	[email protected]:MaharaProject/mahara.git (fetch)
origin	[email protected]:MaharaProject/mahara.git (push)

Importantly, make sure that the part that reads "YOURUSERNAME" is replaced with the username you have registered in your reviews.mahara.org account settings, under the "Profile" section. This may be different from your username on your local filesystem! Note: if your username on your local system is different to your username in gerrit/launchpad you may need to do the connect via HTTPS. Also make sure that the URL itself doesn't have any typoes in it.

If there was a problem, then the easiest thing is to use the git remote rm command to remove the incorrect remote listing, and do git remote add again to fix it.

git remote rm gerrit
git remote add gerrit ssh://[email protected]:29418/mahara
git fetch gerrit

Step 2. Verify you've correctly loaded your SSH public key into Gerrit

If the URL is correct, then you'll want to verify your SSH public key is correctly in Gerrit. First, check to see if you have an SSH public key at all. In the terminal, run this command to see whether you have an SSH public key generated:

ls ~/.ssh

You should see (among other things) a file called id_rsa.pub there. If not, then follow the directions here to generate an SSH public/private key pair: Developer Area/Developer Tools#Secure Shell (SSH) Key

If you do have an id_rsa.pub file, then you'll need to make sure you have copied its contents into your reviews.mahara.org account, as described here: Developer Area/Developer Tools#Gerrit code review (reviews.mahara.org). Specifically, you'll need to view the contents of id_rsa.pub (perhaps by running this command in the terminal: cat ~/.ssh/id_rsa.pub, then copy-paste those contents into the "SSH Public Keys" section of your reviews.mahara.org account settings. Note that if you named your key pair something other than id_rsa/id_rsa.pub, it won't work. You'll need to rename it.

Once you've done that, try doing a "git fetch gerrit" again, as described above.

Step 3. Generate a new SSH key

If none of the two steps above resolve the issue, you may wish to generate a new SSH key using the following option as suggested:

 ssh-keygen -t ed25519 -C "[email protected]"

Error 1.3: "Connection timed out."

When you run git fetch gerrit do you see this?

(master)aaronw@vegas:~/code/mahara$ git fetch gerrit
ssh: connect to host reviews.mahara.org port 29418: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This error indicates that you were not able to connect to reviews.mahara.org on port 29418, as required by its Gerrit git+ssh URL. If you're seeing this problem, you should:

Step 1: Verify you have the right URL and username. See the directions for this in the previous "Error 1.2" section. In particular, make sure you have the port number "29418" correct.

Step 2: Check your network connection. Can you connect to the Internet generally? Can you load up https://reviews.mahara.org in your browser?

Step 3: If you can connect to https://reviews.mahara.org in your browser, is it possible you're behind a firewall that is blocking your access to port 29418? You can check using the command "telnet reviews.mahara.org 29418". If you are successful, you will see output like below. Note that to quit telnet you'll need to press the "Control" key and the "]" key, and then type "quit" and press return.

(master)aaronw@vegas:~/code/mahara$ telnet reviews.mahara.org 29418
Trying 202.78.242.17...
Connected to reviews.mahara.org.
Escape character is '^]'.
SSH-2.0-GerritCodeReview_2.11 (SSHD-CORE-0.14.0)
^]

telnet> quit
Connection closed.

If you are unsuccessful, you will see Telnet say "Trying 202.78.242.17..." for a long time with no further output. (It may go on to give a message about the connection timing out). You can press Control-C to quit this state.

Step 4: If you can't access reviews.mahara.org on port 29418, but you can access https://reviews.mahara.org, then you can try connecting via HTTPS intead of SSH. See the section at the bottom of this page.

Issue 2: Problems with "make push"

Does "git fetch gerrit" work fine for you, but you can't push to gerrit? The underlying issue will depend on the error message you're seeing. As described on the Developer Area/Contributing Code page, usually we push to gerrit using the Makefile macro "make push". Here's what a successful push will look like. (And feel free to push a test commit if you want to. The devs can always mark it "abandoned" in Gerrit later.)

(master)aaronw@vegas:~/www/mahara$ make push
Running minimum acceptance test...
All good!
htdocs/auth/webservice/db/install.xml validates
htdocs/artefact/internal/db/install.xml validates
htdocs/artefact/comment/db/install.xml validates
htdocs/artefact/file/db/install.xml validates
htdocs/artefact/annotation/db/install.xml validates
htdocs/artefact/blog/db/install.xml validates
htdocs/artefact/resume/db/install.xml validates
htdocs/artefact/blog/blocktype/taggedposts/db/install.xml validates
htdocs/artefact/plans/db/install.xml validates
htdocs/blocktype/wall/db/install.xml validates
htdocs/lib/db/install.xml validates
htdocs/blocktype/externalfeed/db/install.xml validates
htdocs/notification/internal/db/install.xml validates
htdocs/interaction/forum/db/install.xml validates
htdocs/notification/emaildigest/db/install.xml validates
htdocs/search/elasticsearch/db/install.xml validates
htdocs/module/multirecipientnotification/db/install.xml validates
Pushing the change upstream...
Counting objects: 15, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 895 bytes | 0 bytes/s, done.
Total 5 (delta 1), reused 1 (delta 0)
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: new: 1, refs: 1, done    
remote: 
remote: New Changes:
remote:   https://reviews.mahara.org/5710 Test commit. Abandon me please.
remote: 
To ssh://[email protected]:29418/mahara
 * [new branch]      HEAD -> refs/publish/master

Error 2.1: "No rule to make target push"

Do you see this when you do make push?

(master)aaronw@vegas:~/www/code/htdocs$ make push
make: *** No rule to make target `push'.  Stop.

This message happens when you try to run "make push" while not in the base level of your Mahara local git repo. This is a limitation of the "make" command itself. It will only work when run in the same directory where a "Makefile" is present. In Mahara, this is the bottom level of the Mahara source code. (The same place where the "htdocs", ".git", and "Makefile" are present). In the example above, I would want to go up one level to ~/www/code, and then "make push" will work.

Error 2.2: "(no new changes)"

Do you see this error from Gerrit?

(master)aaronw@vegas:~/www/mahara$ make push
Running minimum acceptance test...
All good!
htdocs/auth/webservice/db/install.xml validates
htdocs/artefact/internal/db/install.xml validates
htdocs/artefact/comment/db/install.xml validates
htdocs/artefact/file/db/install.xml validates
htdocs/artefact/annotation/db/install.xml validates
htdocs/artefact/blog/db/install.xml validates
htdocs/artefact/resume/db/install.xml validates
htdocs/artefact/blog/blocktype/taggedposts/db/install.xml validates
htdocs/artefact/plans/db/install.xml validates
htdocs/lib/db/install.xml validates
htdocs/blocktype/wall/db/install.xml validates
htdocs/blocktype/externalfeed/db/install.xml validates
htdocs/notification/internal/db/install.xml validates
htdocs/interaction/forum/db/install.xml validates
htdocs/notification/emaildigest/db/install.xml validates
htdocs/search/elasticsearch/db/install.xml validates
htdocs/module/multirecipientnotification/db/install.xml validates
Pushing the change upstream...
Total 0 (delta 0), reused 0 (delta 0)
remote: Processing changes: refs: 1, done    
To ssh://[email protected]:29418/mahara
 ! [remote rejected] HEAD -> refs/publish/master (no new changes)
error: failed to push some refs to 'ssh://[email protected]:29418/mahara'
make: *** [push] Error 1

This happens if you have not actually made any local commits to push yet! Or they are in a different git branch from the one you have currently checked out. You can use the "git log" command to see the commits in the current branch. You can also do "git status" to see if you have any locally uncommitted changes.

Check out the Developer Area/Contributing Code and Developer Area/Developer Environment pages for some cheat sheets on creating git commits.

Error 2.3: "Missing Change-Id in commit message footer"

Do you see this when you do make push? The local validation steps pass, but then Gerrit complains that you are missing the Change-Id in your commit message footer?

(master)aaronw@vegas:~/code/mahara$ make push
Running minimum acceptance test...
All good!
htdocs/auth/webservice/db/install.xml validates
htdocs/artefact/internal/db/install.xml validates
htdocs/artefact/comment/db/install.xml validates
htdocs/artefact/file/db/install.xml validates
htdocs/artefact/blog/blocktype/taggedposts/db/install.xml validates
htdocs/artefact/blog/db/install.xml validates
htdocs/artefact/annotation/db/install.xml validates
htdocs/artefact/resume/db/install.xml validates
htdocs/artefact/plans/db/install.xml validates
htdocs/lib/db/install.xml validates
htdocs/blocktype/wall/db/install.xml validates
htdocs/blocktype/externalfeed/db/install.xml validates
htdocs/notification/internal/db/install.xml validates
htdocs/notification/emaildigest/db/install.xml validates
htdocs/search/elasticsearch/db/install.xml validates
htdocs/interaction/forum/db/install.xml validates
htdocs/module/multirecipientnotification/db/install.xml validates
Pushing the change upstream...
Counting objects: 13, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 295 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: refs: 1, done    
remote: ERROR: missing Change-Id in commit message footer
remote: 
remote: Hint: To automatically insert Change-Id, install the hook:
remote:   gitdir=$(git rev-parse --git-dir); scp -p -P 29418 [email protected]:hooks/commit-msg ${gitdir}/hooks/
remote: And then amend the commit:
remote:   git commit --amend
remote: 
To ssh://[email protected]:29418/mahara
 ! [remote rejected] HEAD -> refs/publish/master (missing Change-Id in commit message footer)
error: failed to push some refs to 'ssh://[email protected]:29418/mahara'
make: *** [push] Error 1

If this happens, it means you are missing the post-commit hook that adds Gerrit's special "Change-Id" line to your commit. The error itself actually tells you how to fix it. Just run the commands listed under the "Hint":

gitdir=$(git rev-parse --git-dir); scp -p -P 29418 [email protected]:hooks/commit-msg ${gitdir}/hooks/
git commit --amend

Running "git commit --amend" will open up an editor asking you whether you want to change your commit message. You can just save and close the editor without making any changes. Once this is done, do "make push" again and it should work.

Error 2.4: Change-Id in the wrong place.

This error looks very similar to the previous one:

(master)aaronw@vegas:~/code/mahara$ make push
(master)aaronw@vegas:~/code/mahara$ make push
Running minimum acceptance test...
All good!
htdocs/artefact/internal/db/install.xml validates
htdocs/auth/webservice/db/install.xml validates
htdocs/artefact/file/db/install.xml validates
htdocs/artefact/comment/db/install.xml validates
htdocs/artefact/annotation/db/install.xml validates
htdocs/artefact/blog/blocktype/taggedposts/db/install.xml validates
htdocs/artefact/blog/db/install.xml validates
htdocs/artefact/resume/db/install.xml validates
htdocs/artefact/plans/db/install.xml validates
htdocs/blocktype/wall/db/install.xml validates
htdocs/blocktype/externalfeed/db/install.xml validates
htdocs/lib/db/install.xml validates
htdocs/notification/internal/db/install.xml validates
htdocs/notification/emaildigest/db/install.xml validates
htdocs/interaction/forum/db/install.xml validates
htdocs/search/elasticsearch/db/install.xml validates
htdocs/module/multirecipientnotification/db/install.xml validates
Pushing the change upstream...
Counting objects: 13, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 349 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: refs: 1, done    
remote: ERROR: missing Change-Id in commit message footer
remote: 
remote: Hint: A potential Change-Id was found, but it was not in the footer (last paragraph) of the commit message.
remote: 
remote: Hint: To automatically insert Change-Id, install the hook:
remote:   gitdir=$(git rev-parse --git-dir); scp -p -P 29418 [email protected]:hooks/commit-msg ${gitdir}/hooks/
remote: And then amend the commit:
remote:   git commit --amend
remote: 
To ssh://[email protected]:29418/mahara
 ! [remote rejected] HEAD -> refs/publish/master (missing Change-Id in commit message footer)
error: failed to push some refs to 'ssh://[email protected]:29418/mahara'
make: *** [push] Error 1

The difference is in the hint that Gerrit gives you. Notice that it says "Hint: A potential Change-Id was found, but it was not in the footer (last paragraph) of the commit message." Gerrit requires its Change-Id to be the last line (or at least in the last paragraph) of the commit message. If it isn't then it won't be able to find it.

You can check whether this is the case by doing a "git log -1" (that's the numeral 1) to see the last log message. Does it look something like this?

(master)aaronw@vegas:~/code/mahara$ git log -1
commit a60cfb21b313421b024875dd8c6e8d9aeccb406d
Author: Aaron Wells 
Date:   Fri Nov 13 16:05:30 2015 +1300

    test commit. Please abandon. :)
    
    Change-Id: Ia68ca2b5f77ae664902bd350b9aabc0298bb52b2
    
    wrong order!

Notice the paragraph that reads "wrong order!" underneath the Change-Id. If this is the case, you can fix it by doing a "git commit --amend". This will open the git commit message for editing. Then cut the Change-Id line and paste it into the bottom of your commit message, save, and quit. Do "git log -1" again to see if it's correct, and if so, do a "make push".

Error 2.5: "Invalid author" or "You are not committer X"

Does gerrit tell you "invalid author" or "You are not committer [email protected]"? This can happen when the git commits you're generating locally, have authorship information that doesn't match up with your account details on reviews.mahara.org.

To check if this is the case, first log in to https://reviews.mahara.org. Look at the "Full Name" and "Email Address" lines in the "Profile" section of your account settings.

Now, run "git log -1" (that's the numeral one) to look at your most recent commit.

(master)aaronw@vegas:~/code/mahara$ git log -1
commit fd10f31839414840267793206b51afbec014c148
Author: Aaron Wells 
Date:   Fri Nov 13 16:23:52 2015 +1300

    test commit. Please abandon. :)
    
    Change-Id: Ia68ca2b5f77ae664902bd350b9aabc0298bb52b2

Notice the "Author" line. Does the name and email address there match the name and email address in your reviews.mahara.org account? If you have newly configured git, then it's quite possible that the email address "@" portion will be the name of your current computer, not your actual email address.

You can fix this information by setting it in your global git config, like so. Just make sure to put in your real name and email!

git config --global user.name "FirstName LastName"
git config --global user.email "[email protected]"

Error 2.6: "xargs: xargs: xmllint: No such file or directoryxmllint"

Do you see this when you try to do "make push"?

(master)aaronw@vegas:~/code/mahara$ make push
ssh: Could not resolve hostname reviews.mahara.org/mahara: Name or service not known
Running minimum acceptance test...
All good!
xargs: xargs: xmllint: No such file or directoryxmllint
: No such file or directory
make: *** [minaccept] Error 127

This means that you didn't install the libxml2-utils package, which is used by Mahara's pre-commit syntax checker. This is an easy step to miss when you're going through the instructions on the Developer Tools page. To fix it, do this:

sudo apt-get install libxml2-utils

If you are not using Ubuntu, then you may need a different package, or may need to install it differently. Search through your system's software repositories for the correct package to get the xmllint utility.

Error 2.7: "error: unpack failed: error Missing tree ff0a6265a118f09e5c2e0f28088e13b85e537e22"

Do you see this when you try to do "make push"?

maharacode/mahara$ make push
Running minimum acceptance test...
No syntax errors detected in htdocs/lang/en.utf8/auth.php
No syntax errors detected in htdocs/api/xmlrpc/lib.php
Acceptance test passed. :)
Pushing the change upstream...
Counting objects: 1, done.
Writing objects: 100% (1/1), 321 bytes | 0 bytes/s, done.
Total 1 (delta 0), reused 0 (delta 0)
error: unpack failed: error Missing tree ff0a6265a118f09e5c2e0f28088e13b85e537e22
fatal: Unpack error, check server log
To ssh://[email protected]:29418/mahara
 ! [remote rejected] HEAD -> refs/publish/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://[email protected]:29418/mahara'
make: *** [push] Error 1

This is a known issue with Gerrit: https://code.google.com/p/gerrit/issues/detail?id=1582

To get around it, use the --no-thin flag:

git push --no-thin gerrit HEAD:refs/publish/master

instead of 'make push'.

Still not working?

If nothing helped, please check with your IT support if port 29418 to which you need to connect in order to reach Gerrit is blocked at your end.

Addendum: Setting up Gerrit to work in HTTPS

If you are absolutely unable to get Gerrit to work over SSH, then you can try switching your Gerrit to go over HTTPS instead.

Step 1: Log in to your account on https://reviews.mahara.org. Go to the "Settings" section, and look in the "HTTP Password" tab. Note the username here. Click the Generate Password button and copy down the password.

Step 2: Set your gerrit remote to https

In the terminal, go into your Mahara directory, and use these commands to remove the SSH url for Gerrit and replace it with the HTTPS url.

git remote rm gerrit
git remote add gerrit https://[email protected]/mahara
git fetch gerrit

The next time you try to push a commit to this repository, you will be prompted for the password from https://reviews.mahara.org. Paste it in as required, and your commit will go through. Git will prompt you for your password every time you push to Gerrit. This can get annoying quickly. ;)

But there are ways to cache your password so that you don't have to enter it over and over again. GitHub has a good tutorial about it: https://help.github.com/articles/caching-your-github-password-in-git/