GitLab support will be removed with Artemis 8.0.0.
Please use LocalVC and Jenkins instead for new installations with Jenkins as the CI system.
For existing Jenkins and GitLab Setups, you can migrate to LocalVC with this not merged Pull Request.
This section describes how to set up a programming exercise environment
based on Jenkins and GitLab. Optional commands are in curly brackets {}.
The following assumes that all instances run on separate servers. If you
have one single server, or your own NGINX instance, just skip all NGINX
related steps and use the configurations provided under Separate NGINX
Configurations
If you want to setup everything on your local development computer,
ignore all NGINX related steps.Just make sure that you use
unique port mappings for your Docker containers (e.g.8081for
GitLab,8082for Jenkins,8080for Artemis)
In order to use Artemis with Jenkins as Continuous Integration
Server and Gitlab as Version Control Server, you have to configure
the file application-prod.yml (Production Server) or
application-artemis.yml (Local Development) accordingly. Please note
that all values in <..> have to be configured properly. These values
will be explained below in the corresponding sections. If you want to set up a local environment, copy the values
below into your application-artemis.yml or application-local.yml file (the latter is recommended), and follow
the Gitlab Server Quickstart guide.
artemis:course-archives-path:./exports/coursesrepo-clone-path:./reposrepo-download-clone-path:./repos-downloadbcrypt-salt-rounds:11# The number of salt rounds for the bcrypt password hashing. Lower numbers make it faster but more unsecure and vice versa.# Please use the bcrypt benchmark tool to determine the best number of rounds for your system. https://github.com/ls1intum/bcrypt-Benchmarkuser-management:use-external:falseinternal-admin:username:artemis_adminpassword:artemis_adminaccept-terms:falselogin:account-name:TUMversion-control:url:http://localhost:8081user:rootpassword:artemis_admin# created in Gitlab Server Quickstart step 2token:artemis-gitlab-token# generated in Gitlab Server Quickstart steps 4 and 5continuous-integration:user:artemis_adminpassword:artemis_adminurl:http://localhost:8082vcs-credentials:artemis_gitlab_admin_credentialsartemis-authentication-token-key:artemis_notification_plugin_tokenartemis-authentication-token-value:artemis_adminbuild-timeout:30git:name:Artemisemail:artemis@xcit.tum.dejenkins:internal-urls:ci-url:http://jenkins:8080vcs-url:http://gitlab:80use-crumb:falseserver:port:8080url:http://172.17.0.1:8080# `http://host.docker.internal:8080` for Windows
In addition, you have to start Artemis with the profiles gitlab and
jenkins so that the correct adapters will be used, e.g.:
For a local setup on Windows you can use http://host.docker.internal appended
by the chosen ports as the version-control and continuous-integration url.
Make sure to change the server.url value in application-dev.yml
or application-prod.yml accordingly. This value will be used for the
communication hooks from GitLab to Artemis and from Jenkins to Artemis.
In case you use a different port than 80 (http) or 443 (https) for the
communication, you have to append it to the server.url value,
e.g. 127.0.0.1:8080.
When you start Artemis for the first time, it will automatically create
an admin user.
Note: Sometimes Artemis does not generate the admin user which may lead to a startup
error. You will have to create the user manually in the MySQL database and in GitLab. Make sure
both are set up correctly and follow these steps:
Use the tool mentioned above to generate a password hash.
Connect to the database via a client like MySQL Workbench
and execute the following query to create the user. Replace artemis_admin and HASHED_PASSWORD with your
chosen username and password:
4. Create a user in Gitlab (http://your-gitlab-domain/admin/users/new) and make sure that the username and
email are the same as the user from the database:
5. Edit the new admin user (http://your-gitlab-domain/admin/users/artemis_admin/edit) to set the password to the
same value as in the database:
The following steps describes how to set up the GitLab server in a semi-automated way.
This is ideal as a quickstart for developers. For a more detailed setup, see
Manual Gitlab Server Setup.
In a production setup, you have to at least change the root password (by either specifying it in step 1 or extracting
the random password in step 2) and generate random access tokens (instead of the pre-defined values).
Set the variable GENERATE_ACCESS_TOKENS to true in the gitlab-local-setup.sh script and use the generated
tokens instead of the predefined ones.
Start the GitLab container defined in docker/gitlab-jenkins-mysql.yml by running
If you want to generate a random password for the root user, remove the part before dockercompose from
the command. GitLab passwords must not contain commonly used combinations of words and letters.
The file uses the GITLAB_OMNIBUS_CONFIG environment variable to configure the Gitlab instance after the container
is started.
It disables prometheus monitoring, sets the ssh port to 2222, and adjusts the monitoring endpoint whitelist
by default.
Wait a couple of minutes since GitLab can take some time to set up. Open the instance in your browser
(usually http://localhost:8081).
You can then login using the username root and your password (which defaults to artemis_admin,
if you used the command from above).
If you did not specify the password, you can get the initial one using:
Insert the GitLab root user password in the file application-local.yml (in src/main/resources) and insert
the GitLab admin account.
If you copied the template from above and used the default password, this is already done for you.
You can also manually create in by navigating to http://localhost:8081/-/profile/personal_access_tokens?name=Artemis+Admin+token&scopes=api,read_api,read_user,read_repository,write_repository,sudo and
generate a token with all scopes.
Copy this token into the ADMIN_PERSONAL_ACCESS_TOKEN field in the
docker/gitlab/gitlab-local-setup.sh file.
If you used the command to generate the token, you don’t have to change the gitlab-local-setup.sh file.
Adjust the GitLab setup by running, this will configure GitLab’s network setting to allow local requests:
This script can also generate random access tokens, which should be used in a production setup. Change the
variable $GENERATE_ACCESS_TOKENS to true to generate the random tokens and insert them into the Artemis
configuration file.
GitLab provides no possibility to set a users password via API without forcing the user to change it afterwards
(see Issue 19141).
Therefore, you may want to patch the official gitlab docker image.
Thus, you can use the following Dockerfile:
FROMgitlab/gitlab-ce:latestRUNsed-i'/^.*user_params\[:password_expires_at\] = Time.current if admin_making_changes_for_another_user.*$/s/^/#/'/opt/gitlab/embedded/service/gitlab-rails/lib/api/users.rb
This Dockerfile disables the mechanism that sets the password to expired state after changed via API.
If you want to use this custom image, you have to build the image and replace all occurrences of
gitlab/gitlab-ce:latest in the following instructions by your chosen image name.
Pull the latest GitLab Docker image (only if you don’t use your custom gitlab image)
Run the image (and change the values for hostname and ports). Add
-p2222:22 if cloning/pushing via ssh should be possible. As
GitLab runs in a docker container and the default port for SSH (22)
is typically used by the host running Docker, we change the port
GitLab uses for SSH to 2222. This can be adjusted if needed.
Make sure to remove the comments from the command before running it.
dockerrun-itd--namegitlab\--hostnameyour.gitlab.domain.com\ # Specify the hostname--restartalways\-m3000m\ # Optional argument to limit the memory usage of Gitlab-p8081:80-p443:443\ # Alternative 1: If you are NOT running your own NGINX instance-p<someportofyourchoosing>:80\ # Alternative 2: If you ARE running your own NGINX instance-p2222:22\ # Remove this if cloning via SSH should not be supported-vgitlab_data:/var/opt/gitlab\-vgitlab_logs:/var/log/gitlab\-vgitlab_config:/etc/gitlab\gitlab/gitlab-ce:latest
Wait a couple of minutes until the container is deployed and GitLab
is set up, then open the instance in you browser.
You can get the initial password for the root user using
dockerexecgitlabcat/etc/gitlab/initial_root_password.
We recommend to rename the root admin user to artemis. To rename
the user, click on the image on the top right and select Settings.
Now select Account on the left and change the username. Use the
same password in the Artemis configuration file
application-artemis.yml
letsencrypt['enable']=true# GitLab 10.5 and 10.6 require this optionexternal_url"https://your.gitlab.domain.com"# Must use https protocolletsencrypt['contact_emails']=['gitlab@your.gitlab.domain.com']# Optionalnginx['redirect_http_to_https']=truenginx['redirect_http_to_https_port']=80
Reconfigure GitLab to generate the certificate.
# Save your changes and finally run
gitlab-ctlreconfigure
If this command fails, try using
gitlab-ctlrenew-le-certs
Login to GitLab using the Artemis admin account and go to the profile
settings (upper right corner → Preferences)
(Optional, only necessary for local setup) Allow outbound requests to local network
There is a known limitation for the local setup: webhook URLs for the
communication between GitLab and Artemis and between GitLab and Jenkins
cannot include local IP addresses. This option can be deactivated in
GitLab on <https://gitlab-url>/admin/application_settings/network →
Outbound requests. Another possible solution is to register a local URL,
e.g. using ngrok, to be available over a domain
the Internet.
Adjust the monitoring-endpoint whitelist. Run the following command
This will disable the firewall for all IP addresses. If you only want to
allow the server that runs Artemis to query the information, replace
0.0.0.0/0 with ARTEMIS.SERVER.IP.ADDRESS/32
If you use SSH and use a different port than 2222, you have to
adjust the port above.
Disable prometheus.
As we encountered issues with the Prometheus log files not being deleted and therefore filling up the disk space,
we decided to disable Prometheus within GitLab.
If you also want to disable prometheus, edit the configuration again using
Artemis can clone/push the repositories during setup and for the online code editor using SSH.
If the SSH key is not present, the username + token will be used as fallback (and all git operations will use
HTTP(S) instead of SSH).
You first have to create a SSH key (locally), e.g. using ssh-keygen (more information on how to create a SSH
key can be found e.g. at ssh.com or
at gitlab.com).
The list of supported ciphers can be found at Apache Mina.
It is recommended to use a password to secure the private key, but it is not mandatory.
Please note that the private key file must be named ìd_rsa, id_dsa, id_ecdsa or id_ed25519,
depending on the ciphers used.
You now have to extract the public key and add it to GitLab.
Open the public key file (usually called id_rsa.pub (when using RSA)) and copy it’s content (you can also
use catid_rsa.pub to show the public key).
Navigate to GITLAB-URL/-/profile/keys and add the SSH key by pasting the content of the public key.
<ssh-key-path> is the path to the folder containing the id_rsa file (but without the filename). It will
be used in the configuration of Artemis to specify where Artemis should look for the key and store
the known_hosts file.
<ssh-private-key-password> is the password used to secure the private key. It is also needed for the
configuration of Artemis, but can be omitted if no password was set (e.g. for development environments).
Note that upgrading to a major version may require following an upgrade path. You can view supported paths
here.
Start a GitLab container just as described in Start-Gitlab and wait for a couple of minutes. GitLab
should configure itself automatically. If there are no issues, you can
delete the old container using dockerrmgitlab_old and the olf
image (see dockerimages) using dockerrmi<old-image-id>.
You can also remove all old images using dockerimageprune-a
The following steps describe how to deploy a pre-configured version of the Jenkins server.
This is ideal as a quickstart for developers. For a more detailed setup, see
Manual Jenkins Server Setup.
In a production setup, you have to at least change the user credentials (in the file jenkins-casc-config-gitlab.yml) and
generate random access tokens.
1. Create a new access token in GitLab named Jenkins and give it api and read_repository rights. You can
do either do it manually or using the following command:
Jenkins is then reachable under http://localhost:8082/ and you can login using the credentials specified
in jenkins-casc-config-gitlab.yml (defaults to artemis_admin as both username and password).
The application-local.yml must be adapted with the values configured in jenkins-casc-config-gitlab.yml:
Open the src/main/resources/config/application-jenkins.yml and change the following:
Again, if you are using a development setup, the template in the beginning of this page already contains the
correct values.
Run the following command to get the latest jenkins LTS docker image.
dockerpulljenkins/jenkins:lts
Create a custom docker image
In order to install and use Maven with Java in the Jenkins container,
you have to first install maven, then download Java and finally
configure Maven to use Java instead of the default version.
You also need to install Swift and SwiftLint if you want to be able to
create Swift programming exercises.
To perform all these steps automatically, you can prepare a Docker
image:
Create a Dockerfile with the content found here <docker/jenkins/Dockerfile>.
Copy it in a file named Dockerfile, e.g. in
the folder /opt/jenkins/ using vimDockerfile.
Now run the command dockerbuild--no-cache-tjenkins-artemis.
This might take a while because Docker will download Java, but this
is only required once.
If you run your own NGINX or if you install Jenkins on a local development computer, then skip the next steps (4-7)
Create a file increasing the maximum file size for the nginx proxy.
The nginx-proxy uses a default file limit that is too small for the
plugin that will be uploaded later. Skip this step if you have your
own NGINX instance.
The NGINX default timeout is pretty low. For plagiarism check and unlocking student repos for the exam a higher
timeout is advisable. Therefore we write our own nginx.conf and load it in the container.
Run the NGINX proxy docker container, this will automatically setup
all reverse proxies and force https on all connections. (This image
would also setup proxies for all other running containers that have
the VIRTUAL_HOST and VIRTUAL_PORT environment variables). Skip this
step if you have your own NGINX instance.
The nginx proxy needs another docker-container to generate
letsencrypt certificates. Run the following command to start it (make
sure to change the email-address). Skip this step if you have your
own NGINX instance.
Run Jenkins by executing the following command (change the hostname
and choose which port alternative you need)
dockerrun-itd--namejenkins\--restartalways\-vjenkins_data:/var/jenkins_home\-v/var/run/docker.sock:/var/run/docker.sock\-v/usr/bin/docker:/usr/bin/docker:ro\-eVIRTUAL_HOST=your.jenkins.domain-eVIRTUAL_PORT=8080\ # Alternative 1: If you are NOT using a separate NGINX instance-eLETSENCRYPT_HOST=your.jenkins.domain\ # Only needed if Alternative 1 is used-p8082:8080\ # Alternative 2: If you ARE using a separate NGINX instance OR you ARE installing Jenkins on a local development computer-uroot\jenkins/jenkins:lts
If you still need the old setup with Python & Maven installed locally, use jenkins-artemis instead of
jenkins/jenkins:lts.
Also note that you can omit the -uroot, -v/var/run/docker.sock:/var/run/docker.sock and
-v/usr/bin/docker:/usr/bin/docker:ro parameters, if you do not want to run Docker builds on the Jenkins controller
(but e.g. use remote agents).
Open Jenkins in your browser (e.g. localhost:8082) and setup the
admin user account (install all suggested plugins). You can get the
initial admin password using the following command.
# Jenkins highlights the password in the logs, you can't miss it
dockerlogs-fjenkins
oralternatively
dockerexecjenkinscat/var/jenkins_home/secrets/initialAdminPassword
Set the chosen credentials in the Artemis configuration
application-artemis.yml
In a local setup, you have to disable CSRF otherwise some API endpoints will return HTTP Status 403 Forbidden.
This is done be executing the following command:
dockercompose-fdocker/<Jenkinssetuptobelaunched>.ymlexec-Tjenkinsddof=/var/jenkins_home/init.groovy<docker/jenkins/jenkins-disable-csrf.groovy
The last step is to disable the use-crumb option in application-local.yml:
Note: The custom Jenkins Dockerfile takes advantage of the
Plugin Installation Manager Tool for Jenkins
to automatically install the plugins listed below. If you used the Dockerfile, you can skip these steps and
Server Notification Plugin.
The list of plugins is maintained in docker/jenkins/plugins.yml.
You will need to install the following plugins (apart from the
recommended ones that got installed during the setup process):
Timestamper for adding the
time to every line of the build output (Timestamper might already be installed)
Pipeline for defining the
build description using declarative files (Pipeline might already be installed)
Note: This is a suite of plugins that will install multiple plugins
Pipeline Maven to use maven within the pipelines. If you want to
use Docker for your build agents you may also need to install
Docker Pipeline .
Matrix Authorization Strategy Plugin for configuring permissions
for users on a project and build plan level (Matrix Authorization Strategy might already be installed).
The plugins above (and the pipeline-setup associated with it) got introduced in Artemis 4.7.3.
If you are using exercises that were created before 4.7.3, you also have to install these plugins:
Please note that this setup is deprecated and will be removed in the future.
Please migrate to the new pipeline-setup if possible.
Multiple SCMs for combining the
exercise test and assignment repositories in one build
Post Build Task for preparing build
results to be exported to Artemis
Xvfb for exercises based on GUI
libraries, for which tests have to have some virtual display
Choose “Download now and install after restart” and checking the
“Restart Jenkins when installation is complete and no jobs are running” box
Artemis needs to receive a notification after every build, which
contains the test results and additional commit information. For that
purpose, we developed a Jenkins plugin, that can aggregate and POST
JUnit formatted results to any URL.
You can download the current release of the plugin
here
(Download the .hpi file). Go to the Jenkins plugin page (Manage
Jenkins → System Configuration → Plugins) and install the downloaded file under the
Advanced settings tab under Deploy Plugin
Create a new access token in GitLab named Jenkins and give it
api rights and read_repository rights. For detailed
instructions on how to create such a token follow Gitlab Access
Token.
Copy the generated token and create new Jenkins credentials:
Kind: GitLab API token
Scope: Global
API token: your.copied.token
Leave the ID field blank
The description is up to you
Go to the Jenkins settings Manage Jenkins → System. There
you will find the GitLab settings. Fill in the URL of your GitLab
instance and select the just created API token in the credentials
dropdown. After you click on “Test Connection”, everything should
work fine. If you have problems finding the right URL for your local docker setup,
you can try http://host.docker.internal:8081 for Windows or http://docker.for.mac.host.internal:8081 for Mac
if GitLab is reachable over port 8081.
Copy the generated ID (e.g. ea0e3c08-4110-4g2f-9c83-fb2cdf6345fa)
of the new credentials and put it into the Artemis configuration file
application-artemis.yml
In order to upgrade Jenkins to a newer version, you need to rebuild the Docker image targeting the new version.
The stable LTS versions can be viewed through the changelog
and the corresponding Docker image can be found on
dockerhub.
Open the Jenkins Dockerfile and replace the value of FROM with jenkins/jenkins:lts.
After running the command dockerpulljenkins/jenkins:lts, this will use the latest LTS version
in the following steps.
You can also use a specific LTS version.
For example, if you want to upgrade Jenkins to version 2.289.2, you will need to use the
jenkins/jenkins:2.289.2-lts image.
If you’re using dockercompose, you can simply use the following command and skip the next steps.
You can remove the backup container if it’s no longer needed:
dockerrmjenkins_old
You should also update the Jenkins plugins regularly due to security
reasons. You can update them directly in the Web User Interface in the
Plugin Manager.
An alternative way of adding a build agent that will use docker (similar to the remote agents below) but running
locally, can be done using the jenkins/ssh-agent docker image docker image.
You might want to run the builds on additional Jenkins agents, especially if a large amount of students should use
the system at the same time.
Jenkins supports remote build agents: The actual compilation of the students submissions happens on these
other machines but the whole process is transparent to Artemis.
This guide explains setting up a remote agent on an Ubuntu virtual machine that supports docker builds.
Add a new user to the remote machine that Jenkins will use: sudoadduser--disabled-password--gecos""jenkins
Add the jenkins user to the docker group (This allows the jenkins user to interact with docker):
sudousermod-a-Gdockerjenkins
Generate a new SSH key locally (e.g. using ssh-keygen) and add the public key to the .ssh/authorized_keys
file of the jenkins user on the agent VM.
Validate that you can connect to the build agent machine using SSH and the generated private key and validate that
you can use docker (docker ps should not show an error)
Log in with your normal account on the build agent machine and install Java: sudoaptinstalldefault-jre
Add a new secret in Jenkins, enter private key you just generated and add the passphrase, if set:
Add a new node (select a name and select Permanent Agent):
Set the number of executors so that it matches your machine’s specs: This is the number of concurrent builds
this agent can handle. It is recommended to match the number of cores of the machine,
but you might want to adjust this later if needed.
Set the remote root directory to /home/jenkins/remote_agent.
Set the usage to Only build jobs with label expressions matching this node.
This ensures that only docker-jobs will be built on this agent, and not other jobs.
Add a label docker to the agent.
Set the launch method to Launch via SSH and add the host of the machine.
Select the credentials you just created and select Manually trusted key Verification Strategy
as Host key verification Strategy.
Save it.
Wait for some moments while jenkins installs it’s remote agent on the agent’s machine.
You can track the progress using the Log page when selecting the agent. System information should also be available.
Change the settings of the master node to be used only for specific jobs.
This ensures that the docker tasks are not executed on the master agent but on the remote agent.
Artemis supports user management in Jenkins as of version 4.11.0. Creating an account in Artemis will also create an
account on Jenkins using the same password. This enables users to login and access Jenkins. Updating and/or deleting
users from Artemis will also lead to updating and/or deleting from Jenkins.
Unfortunately, Jenkins does not provide a Rest API for user management which present the following caveats:
The username of a user is treated as a unique identifier in Jenkins.
It’s not possible to update an existing user with a single request.
We update by deleting the user from Jenkins and recreating it with the updated data.
In Jenkins, users are created in an on-demand basis.
For example, when a build is performed, its change log is computed and as a result commits from users
who Jenkins has never seen may be discovered and created.
Since Jenkins users may be re-created automatically, issues may occur such as 1) creating a user, deleting it,
and then re-creating it and 2) changing the username of the user and reverting back to the previous one.
Updating a user will re-create it in Jenkins and therefore remove any additionally saved Jenkins-specific
user data such as API access tokens.
Artemis takes advantage of the Project-based Matrix Authorization Strategy plugin to support build plan
access control in Jenkins.
This enables specific Artemis users to access build plans and execute actions such as triggering a build.
This section explains the changes required in Jenkins in order to set up build plan access control:
Navigate to Manage Jenkins → Plugins → Installed plugins and make sure that you have the
Matrix Authorization Strategy plugin installed
Navigate to Manage Jenkins → Security and navigate to the “Authorization” section
Select the “Project-based Matrix Authorization Strategy” option
In the table make sure that the “Read” permission under the “Overall” section is assigned to
the “Authenticated Users” user group.
In the table make sure that all “Administer” permission is assigned to all administrators.
You are finished. If you want to fine-tune permissions assigned to teaching assistants and/or instructors,
you can change them within the JenkinsJobPermission.java file.