Stratus3D

Software Engineering, Web Development and 3D Graphics

Migrating From BitBucket to Local GitLab Server

GitLab CE

I decided to move away from BitBucket Git repository hosting to a self-hosted Git solution. Since I am familiar with Ruby on Rails I decided use GitLab to host my repositories since GitLab version 8 is a Rails 4.1 application. GitLab originally used Gitolite to handle the Git repository hosting but they have moved away from Gitolite to plain old Ruby code. I already had a server running Ubuntu 14.04 and I figured it would be perfect for running GitLab.

Installation

It turns out the GitLab installation instructions for Ubuntu are straightforward.

First we need to install OpenSSH server, Postfix, and a few other things:

$ sudo apt-get install curl openssh-server ca-certificates postfix

Since I didn’t need email I chose the default options when I went through the Postfix configuration process. Now we can install GitLab:

$ curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
$ sudo apt-get install gitlab-ce

The curl command configures apt-get with the information it needs to fetch and install the GitLab package. If you want to see the contents of the script before you run it you can pipe it to cat or less instead of sudo bash. apt-get then installs GitLab.

Configuration

After installation run:

$ sudo gitlab-ctl reconfigure

After installing check the status of GitLab:

$ sudo gitlab-ctl status

Everything should be running. If GitLab is not running try starting it manually by running:

$ sudo gitlab-ctl start

Now that GitLab is running you should be able to navigate to your server’s IP address or hostname in your browser and see the login page.

GitLab CE

Sign in as root with the default password (5iveL!ve) and then change your password. After changing the password you should be able to navigate web interface to customize your profile, setup permissions and create repositories.

Repository Migration

After getting GitLab installed and the permissions setup the next step is to migrate the BitBucket repositories to the GitLab server. Unfortunately importing BitBucket repositories is a little confusing and it took me quite a while to figure out how to import them. The difficulties are mostly due to the way the BitBucket API works.

Create an OAuth Key

We first need to create an OAuth consumer key so that GitLab can access private repositories. To create an OAuth key sign into your BitBucket account and navigate to settings > Oauth. Then click “Add consumer” and create a consumer for GitLab. The callback URL should be the URL to your GitLab instance. The URL could be anything, but it makes sense to also make it the URL to your GitLab instance. Under the Permissions heading make sure the OAuth consumer has read and admin permissions to repositories. It may seem a little odd to giving GitLab admin permissions to repositories, but this is needed in order to GitLab to be able to clone the repositories. The BitBucket import script will use the BitBucket API to add the public key as a deploy key for each repository your import, then clone down the repository via SSH and remove the deploy key once the import is complete. This is necessary because the BitBucket API doesn’t allow repositories to be cloned by OAuth applications via HTTPS. In order to add a deploy key to a repository GitLab must have admin access to the repositories being imported, since adding keys is an admin-level action.

After creating the OAuth consumer make a note of the key and secret values. You will need them when you configure GitLab to use the OAuth consumer.

Configure GitLab with the OAuth Consumer

Update gitlab.rb configuration:

$ sudo vi /etc/gitlab/gitlab.rb

Add an omniauth_providers section to the file:

1
2
3
4
5
6
7
8
gitlab_rails['omniauth_providers'] = [
   {
     "name" => "bitbucket",
     "app_id" => "YOUR_KEY",
     "app_secret" => "YOUR_APP_SECRET",
     "url" => "https://bitbucket.org/"
   }
]

Use the key and secret values you got from the BitBucket interface in place of YOUR_KEY and YOUR_APP_SECRET in the BitBucket section of the file.

Create an SSH Public Key for the GitLab Import Script

Next, you need to create a SSH public key on the server you have GitLab installed on. This key must NOT be associated with ANY existing BitBucket accounts. If it is the import will fail with an “Access denied! Please verify you can add deploy keys to this repository.” error. The BitBucket import script will use the BitBucket API to add this public key as a deploy key for each repository you choose to import. If the key is already being used by a BitBucket account, the API call to add the key will fail since the key is already being used. To be safe, generate a brand new key. The key should be generated for the git user:

1
2
3
$ sudo -u git -H ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/var/opt/gitlab/.ssh/id_rsa): bitbucket_rsa

When prompted for the name of the file to store the key in, enter bitbucket_rsa. Mine ended up being stored at /var/opt/gitlab/.ssh/bitbucket_rsa. The documentation says that the location should be /home/git/.ssh/bitbucket_rsa. Apparently my git user’s home directory was /var/opt/gitlab/ instead of /home/git/. It seemed to work just fine in /var/opt/gitlab/. I believe the only thing that is important is that the key is placed in the git user’s home directory. You can verify the home directory of the git user by running echo ~git. This is a known issue with the documentation and apparently /var/opt/gitlab/ is correct.

Your SSH client will also need to be configured to use this key when connecting to BitBucket. You can do that by adding this to the SSH config (mine was located at /var/opt/gitlab/.ssh/config, you may need to create the file):

1
2
3
Host bitbucket.org
  IdentityFile ~/.ssh/bitbucket_rsa
  User git

Next, to verify you have the ssh key setup correctly, run:

$ sudo -u git -H ssh -Tv bitbucket.org

You should see output along the lines of what is shown below and be prompted continue connecting:

1
2
3
4
5
6
7
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /var/opt/gitlab/.ssh/config
...
debug1: Server host key: RSA 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40
The authenticity of host 'bitbucket.org (131.103.20.167)' can't be established.
RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
Are you sure you want to continue connecting (yes/no)?

Type yes to confirm that you want to connect to bitbucket.org. After confirming you should see a permission denied message. If you see a authentication successful message you have done something wrong. The key you are using has already been added to a BitBucket account and will cause the import script to fail. Ensure the key you are using CANNOT authenticate with BitBucket.

Update GitLab with the New Configuration

Next all you need to do is tell GitLab to reconfigure itself with the changes made in gitlab.rb. To reload the configuration run:

$ sudo gitlab-ctl reconfigure

Import Repositories

You should now be able to go to the web interface and import a repository. Go to your GitLab server’s homepage and click the “New Project” button. On the page the third row in the form should say “Import project from” and have a list of third party sites. Click on the BitBucket button. You should be taken to the BitBucket OAuth authorization page (You will need to be logged into your BitBucket account for this to work). Allow the GitLab consumer access to your account. After you allow access to your account you should be taken back to your GitLab website. The page should have a list of all your BitBucket repositories with an import button for each one.

BitBucket Import

Try clicking the import button for an individual repository, or click “Import All Projects”. If everything is working a spinner icon should appear and after a few seconds the import should complete. Once the import is complete the repository will be just like any other repository in GitLab. You can edit the repository and customize it in any way you see fit.

If something goes wrong during import and you get the “Access denied! Please verify you can add deploy keys to this repository.” you can alter the add_deploy_key method in the importer so that more readable error messages are logged to the production log.

$ sudo vi /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/bitbucket_import/client.rb

Then update the add_deploy_key method so it looks like this:

1
2
3
4
5
6
7
8
9
10
def add_deploy_key(project_identifier, key)
  deploy_key = find_deploy_key(project_identifier, key)
  Rails.logger.info("=== add_deploy_key #{project_identifier}, #{key}, #{deploy_key}")
  return if deploy_key

  response = api.post("/api/1.0/repositories/#{project_identifier}/deploy-keys", key: key, label: "GitLab import key")
  Rails.logger.info("=== add_deploy_key: #{response.inspect}")
  Rails.logger.info("=== add_deploy_key response body: #{response.body}")
  JSON.parse(response.body)
end

The additional logging calls will make it easy to diagnose the problem. Restart GitLab and try to import the repositories again. Then check the production log (sudo tail -f /var/log/gitlab/gitlab-rails/production.log).

Migrating GitHub Repositories

Importing GitHub repositories is easier than importing repositories stored on BitBucket.

First you need to do is register a new application on GitHub. Go to Settings > Applications > Developer Applications > Register new application. The homepage URL should be the URL of your GitLab instance. Fill in the form and submit it. Make a note of the Client ID and the Client Secret after you create the application.

Add An OmniAuth Provider for GitHub

Next you need to add another OmniAuth provider for GitHub. All you need to do is add another hash to the gitlab_rails['omniauth_providers'] array. Edit gitlab.rb again:

$ sudo vi /etc/gitlab/gitlab.rb

The file should like this:

1
2
3
4
5
6
7
8
9
10
11
12
gitlab_rails['omniauth_providers'] = [
    {
    # ... BitBucket OmniAuth settings
    },
    {
      "name" => "github",
      "app_id" => "YOUR_APP_ID",
      "app_secret" => "YOUR_APP_SECRET",
      "url" => "https://github.com/",
      "args" => { "scope" => "user:email" }
    }
]

YOUR_APP_ID and YOUR_APP_SECRET should be set to the Client ID and Client Secret values shown in the GitHub interface. Next reconfigure GitLab:

$ sudo gitlab-ctl reconfigure

Import Repositories

Now go to GitLab and click “New Project”. Under the “Import project from” heading click “GitHub”. You will be taken to GitHub and prompted to give the application you created permission to you account. You will need to be logged in to your GitHub account for this to work. After granting permission to the application you will be taken back to GitLab and shown a list of all GitHub repositories. Choose “Import All Projects” or import individual repositories.

Backing Up GitLab Itself

When I was setting up my GitLab server I came across a script that backs up GitLab data to a remote server - Auto GitLab Backup. Since I don’t have another server or service to backup the data on my GitLab server I wasn’t able to use this script. If you want another copy of your Git repositories all you would need to do would be to setup another server (maybe an EC2 instance?) and then add the Auto GitLab Backup script to your crontab on your GitLab server.

Conclusion

GitLab is a good alternative to BitBucket or GitHub. It’s great to have a completely open source piece of software that I can run on my own servers. The BitBucket import tool is definitely be tricky to setup but it’s great that GitLab provides the tools for migrating repositories. Since it does lack some of the features present in BitBucket and GitHub it may be not be suitable for large teams. But it’s perfect for individuals or small teams that want to be independent of BitBucket and GitHub.

Resources