GitLab HA on-premise
GitLab's omnibus install package ships "out of the box" with everything needed to create and maintain a highly available architecture for each component of the GitLab architecture. In this section, we'll present a high-level overview of each. However - configuring those components is much more detailed, and each section will link to more detailed documentation to review.
As part of this certification program, you should review and become familiar with that documentation as well.
Overview
Order-of-operations
The following components will each be made highly available separately, and as some of those components depend on the other, this is the recommended order for instantiation.
- Configure the database
- Configure Redis
- Configure NFS
- Configure the GitLab application servers
- Configure the load balancers
Database
You can choose to install and manage a database server (PostgreSQL/MySQL) yourself, or you can use GitLab Omnibus packages to help. GitLab recommends PostgreSQL. This is the database that will be installed if you use the Omnibus package to manage your database.
Requirements
- A minimum of three database nodes Each node will run the following services:
-
PostgreSQL
- The database itself -
repmgrd
- A service to monitor, and handle failover in case of a failure -
Consul
agent - Used for service discovery, to alert other nodes when failover occurs
-
- A minimum of three
Consul
server nodes - A minimum of one
pgbouncer
service node
Architecture
Connection flow
Each service in the package comes with a set of default ports. You may need to make specific firewall rules for the connections listed below:
- Application servers connect to PgBouncer default port
- PgBouncer connects to the primary database servers PostgreSQL default port
- Repmgr connects to the database servers PostgreSQL default port
- Postgres secondaries connect to the primary database servers PostgreSQL default port
- Consul servers and agents connect to each others Consul default ports
Setup
For an example configuration, see the example configuration here.
For a more detailed walkthrough, see the GitLab documentation
Redis
High Availability with Redis is possible using a Master x Slave topology with a Redis Sentinel service to watch and automatically start the failover procedure.
You can choose to install and manage Redis and Sentinel yourself, use a hosted cloud solution or you can use the one that comes bundled with Omnibus GitLab packages.
Notes:
- Redis requires authentication for High Availability. See Redis Security documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service.
- You are highly encouraged to read the [Redis Sentinel][sentinel] documentation before configuring Redis HA with GitLab to fully understand the topology and architecture.
- This is the documentation for the Omnibus GitLab packages. For installations from source, follow the Redis HA source installation guide.
- Redis Sentinel daemon is bundled with Omnibus GitLab Enterprise Edition only. For configuring Sentinel with the Omnibus GitLab Community Edition and installations from source, read the Available configuration setups section below.
Requirements
You need at least 3
independent machines: physical, or VMs running into
distinct physical machines. It is essential that all master and slaves Redis
instances run in different machines. If you fail to provision the machines in
that specific way, any issue with the shared environment can bring your entire
setup down.
It is OK to run a Sentinel alongside of a master or slave Redis instance. There should be no more than one Sentinel on the same machine though.
You also need to take into consideration the underlying network topology, making sure you have redundant connectivity between Redis / Sentinel and GitLab instances, otherwise the networks will become a single point of failure.
For a more detailed walkthrough, see the GitLab documentation
NFS
NFS should be configured using your corporate standards for NFS / file stoarge. There probably is already a highly-available file system architecture in your environment, and assuming it can meet the requirements below and is highly performant, you should start by testing that solution with your GitLab application servers.
Required features
File locking: GitLab requires advisory file locking, which is only supported natively in NFS version 4. NFSv3 also supports locking as long as Linux Kernel 2.6.5+ is used. We recommend using version 4 and do not specifically test NFSv3.
Recommended options
When you define your NFS exports, we recommend you also add the following options:
-
no_root_squash
- NFS normally changes theroot
user tonobody
. This is a good security measure when NFS shares will be accessed by many different users. However, in this case only GitLab will use the NFS share so it is safe. GitLab recommends theno_root_squash
setting because we need to manage file permissions automatically. Without the setting you may receive errors when the Omnibus package tries to alter permissions. Note that GitLab and other bundled components do not run asroot
but as non-privileged users. The recommendation forno_root_squash
is to allow the Omnibus package to set ownership and permissions on files, as needed. In some cases where theno_root_squash
option is not available, theroot
flag can achieve the same result. -
sync
- Force synchronous behavior. Default is asynchronous and under certain circumstances it could lead to data loss if a failure occurs before data has synced.
For a more detailed walkthrough, see the GitLab documentation
Note
GitLab strongly recommends against using AWS Elastic File System (EFS). Our support team will not be able to assist on performance issues related to file system access.
Application Servers
Assuming you have already configured a database, Redis, and NFS, you can configure the GitLab application server(s) now.
Note: There is some additional configuration near the bottom for additional GitLab application servers. It's important to read and understand these additional steps before proceeding with GitLab installation.
-
If necessary, install the NFS client utility packages using the following commands:
# Ubuntu/Debian apt-get install nfs-common # CentOS/Red Hat yum install nfs-utils nfs-utils-lib
-
Specify the necessary NFS shares. Mounts are specified in
/etc/fstab
. The exact contents of/etc/fstab
will depend on how you chose to configure your NFS server. See NFS documentation for the various options. Here is an example snippet to add to/etc/fstab
:10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
-
Create the shared directories. These may be different depending on your NFS mount locations.
mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data
-
Download/install GitLab Omnibus using steps 1 and 2 from GitLab downloads. Do not complete other steps on the download page.
-
Create/edit
/etc/gitlab/gitlab.rb
and use the following configuration. Be sure to change theexternal_url
to match your eventual GitLab front-end URL. Depending your the NFS configuration, you may need to change some GitLab data locations. See NFS documentation for/etc/gitlab/gitlab.rb
configuration values for various scenarios. The example below assumes you've added NFS mounts in the default data locations. Additionally the UID and GIDs given are just examples and you should configure with your preferred values.external_url 'https://gitlab.example.com' # Prevent GitLab from starting if NFS data mounts are not available high_availability['mountpoint'] = '/var/opt/gitlab/git-data' # Disable components that will not be on the GitLab application server roles ['application_role'] # PostgreSQL connection details gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server gitlab_rails['db_password'] = 'DB password' # Redis connection details gitlab_rails['redis_port'] = '6379' gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server gitlab_rails['redis_password'] = 'Redis Password' # Ensure UIDs and GIDs match between servers for permissions via NFS user['uid'] = 9000 user['gid'] = 9000 web_server['uid'] = 9001 web_server['gid'] = 9001 registry['uid'] = 9002 registry['gid'] = 9002
Note: To maintain uniformity of links across HA clusters, the
external_url
on the first application server as well as the additional application servers should point to the external url that users will use to access GitLab. In a typical HA setup, this will be the url of the load balancer which will route traffic to all GitLab application servers in the HA cluster.Note: When you specify
https
in theexternal_url
, as in the example above, GitLab assumes you have SSL certificates in/etc/gitlab/ssl/
. If certificates are not present, Nginx will fail to start. See Nginx documentation for more information.
First GitLab application server
As a final step, run the setup rake task only on the first GitLab application server. Do not run this on additional application servers.
- Initialize the database by running
sudo gitlab-rake gitlab:setup
. - Run
sudo gitlab-ctl reconfigure
to compile the configuration.
WARNING: Only run this setup task on NEW GitLab instances because it will wipe any existing data.
Extra configuration for additional GitLab application servers
Additional GitLab servers (servers configured after the first GitLab server) need some extra configuration.
-
Configure shared secrets. These values can be obtained from the primary GitLab server in
/etc/gitlab/gitlab-secrets.json
. Add these to/etc/gitlab/gitlab.rb
prior to running the firstreconfigure
.gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860' gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa' gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d' gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964'
-
Run
touch /etc/gitlab/skip-auto-migrations
to prevent database migrations from running on upgrade. Only the primary GitLab application server should handle migrations. -
Optional Configure host keys. Copy all contents(primary and public keys) inside
/etc/ssh/
on the primary application server to/etc/ssh
on all secondary servers. This prevents false man-in-the-middle-attack alerts when accessing servers in your High Availability cluster behind a load balancer. -
Run
sudo gitlab-ctl reconfigure
to compile the configuration.
For a more detailed walkthrough, see the GitLab documentation
Load Balancers
In an active/active GitLab configuration, you will need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or the exact configuration is beyond the scope of GitLab documentation. We hope that if you're managing HA systems like GitLab you have a load balancer of choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation will outline what ports and protocols you need to use with GitLab.
Basic ports
LB Port | Backend Port | Protocol |
---|---|---|
80 | 80 | HTTP [^1] |
443 | 443 | TCP or HTTPS [^1] [^2] |
22 | 22 | TCP |
GitLab Pages Ports
If you're using GitLab Pages with custom domain support you will need some
additional port configurations.
GitLab Pages requires a separate virtual IP address. Configure DNS to point the
pages_external_url
from /etc/gitlab/gitlab.rb
at the new virtual IP address. See the
GitLab Pages documentation for more information.
LB Port | Backend Port | Protocol |
---|---|---|
80 | Varies [^3] | HTTP |
443 | Varies [^3] | TCP [^4] |
For a more detailed walkthrough, see the GitLab documentation
Next
You can close this issue now