This section describes how to set up a programming exercise environment
based on Jenkins and LocalVC, which is integrated in Artemis. Optional commands are in curly brackets {}.
The following assumes that all instances run on separate servers. If you
have one single server, or your own NGINX instance, just skip all NGINX
related steps and use the configurations provided under Separate NGINX
Configurations
If you want to setup everything on your local development computer,
ignore all NGINX related steps.Just make sure that you use
unique port mappings for your Docker containers (e.g.8081for
Jenkins,8080for Artemis)
In order to use Artemis with Jenkins as Continuous Integration
Server and LocalVC as integrated Version Control Server, you have to configure
the file application-prod.yml (Production Server) or
application-artemis.yml (Local Development) accordingly. Please note
that all values in <..> have to be configured properly. These values
will be explained below in the corresponding sections. If you want to set up a local environment, copy the values
below into your application-artemis.yml or application-local.yml file (the latter is recommended).
artemis:course-archives-path:./exports/coursesrepo-clone-path:./reposrepo-download-clone-path:./repos-downloadbcrypt-salt-rounds:11# The number of salt rounds for the bcrypt password hashing. Lower numbers make it faster but more unsecure and vice versa.# Please use the bcrypt benchmark tool to determine the best number of rounds for your system. https://github.com/ls1intum/bcrypt-Benchmarkuser-management:use-external:falseinternal-admin:username:artemis_adminpassword:artemis_adminaccept-terms:falselogin:account-name:TUMversion-control:url:http://localhost:8080user:rootpassword:dummy# has to be set, but does not matter for LocalVCbuild-agent-git-username:jenkinsbuild-agent-git-password:artemis_admin# choose some strong password and username (gives read access to all repositories)continuous-integration:user:artemis_adminpassword:artemis_adminurl:http://localhost:8082vcs-credentials:artemis_localvc_credentialsartemis-authentication-token-key:artemis_notification_plugin_tokenartemis-authentication-token-value:artemis_adminbuild-timeout:30git:name:Artemisemail:artemis@xcit.tum.dejenkins:# only required if Artemis and Jenkins cannot communicate on their public URLs# e.g., Jenkins is only available in a local container networkinternal-urls:ci-url:http://jenkins:8080vcs-url:http://172.17.0.1:8080# `http://host.docker.internal:8080` for Windowsuse-crumb:falseserver:port:8080url:http://172.17.0.1:8080# `http://host.docker.internal:8080` for Windows
In addition, you have to start Artemis with the profiles localvc and
jenkins so that the correct adapters will be used, e.g.:
For a local setup on Windows you can use http://host.docker.internal appended
by the chosen ports as the continuous-integration url and the internal vcs url.
Make sure to change the server.url and artemis.version-control.url value in application-dev.yml
or application-prod.yml accordingly. The server.url value will be used for the
communication hooks from Jenkins to Artemis.
In case you use a different port than 80 (http) or 443 (https) for the
communication, you have to append it to the both urls value,
e.g. 127.0.0.1:8080.
When you start Artemis for the first time, it will automatically create
an admin user.
Note: Sometimes Artemis does not generate the admin user which may lead to a startup
error. You will have to create the user manually in the MySQL database. Make sure
both are set up correctly and follow these steps:
Use the tool mentioned above to generate a password hash.
Connect to the database via a client like MySQL Workbench
and execute the following query to create the user. Replace artemis_admin and HASHED_PASSWORD with your
chosen username and password:
The following steps describe how to deploy a pre-configured version of the Jenkins server.
This is ideal as a quickstart for developers. For a more detailed setup, see
Manual Jenkins Server Setup.
In a production setup, you have to at least change the user credentials (in the file jenkins-casc-config-localvc.yml).
You can now first build and deploy Jenkins, then you can also start the other services which weren’t started yet:
Jenkins is then reachable under http://localhost:8082/ and you can login using the credentials specified
in jenkins-casc-config-localvc.yml (defaults to artemis_admin as both username and password).
The application-local.yml must be adapted with the values configured in jenkins-casc-config-localvc.yml:
artemis:user-management:use-external:falseinternal-admin:username:artemis_adminpassword:artemis_adminversion-control:url:http://localhost:8080user:rootpassword:dummy# have to be set, but does not matter for LocalVCbuild-agent-git-username:jenkinsbuild-agent-git-password:artemis_admin# choose some strong password and username (gives read access to all repositories)continuous-integration:user:artemis_adminpassword:artemis_adminurl:http://localhost:8082vcs-credentials:artemis_localvc_credentialsartemis-authentication-token-key:artemis_notification_plugin_tokenartemis-authentication-token-value:artemis_admin
Open the src/main/resources/config/application-jenkins.yml and change the following:
Again, if you are using a development setup, the template in the beginning of this page already contains the
correct values.
jenkins:internal-urls:ci-url:http://jenkins:8080vcs-url:http://172.17.0.1:8080# `http://host.docker.internal:8080` for Windows
You’re done. You can now run Artemis with the LocalVC/Jenkins environment.
If you run your own NGINX or if you install Jenkins on a local development computer, you can skip this section.
Create a file increasing the maximum file size for the Nginx proxy.
The nginx-proxy uses a default file limit that is too small for the
plugin that will be uploaded later. Skip this step if you have your
own NGINX instance.
The NGINX default timeout is pretty low. For plagiarism check and unlocking student repos for the exam a higher
timeout is advisable. Therefore we write our own nginx.conf and load it in the container.
Run the NGINX proxy docker container, this will automatically setup
all reverse proxies and force https on all connections. (This image
would also setup proxies for all other running containers that have
the VIRTUAL_HOST and VIRTUAL_PORT environment variables). Skip this
step if you have your own NGINX instance.
The nginx proxy needs another docker-container to generate
letsencrypt certificates. Run the following command to start it (make
sure to change the email-address). Skip this step if you have your
own NGINX instance.
Run Jenkins by executing the following command (change the hostname
and choose which port alternative you need)
dockerrun-itd--namejenkins\--restartalways\-vjenkins_data:/var/jenkins_home\-v/var/run/docker.sock:/var/run/docker.sock\-v/usr/bin/docker:/usr/bin/docker:ro\-eVIRTUAL_HOST=your.jenkins.domain-eVIRTUAL_PORT=8080\ # Alternative 1: If you are NOT using a separate NGINX instance-eLETSENCRYPT_HOST=your.jenkins.domain\ # Only needed if Alternative 1 is used-p8082:8080\ # Alternative 2: If you ARE using a separate NGINX instance OR you ARE installing Jenkins on a local development computer-uroot\jenkins/jenkins:lts
Note that you can omit the -uroot, -v/var/run/docker.sock:/var/run/docker.sock and
-v/usr/bin/docker:/usr/bin/docker:ro parameters, if you do not want to run Docker builds on the Jenkins controller
(but e.g. use remote agents).
Open Jenkins in your browser (e.g. localhost:8082) and setup the
admin user account (install all suggested plugins). You can get the
initial admin password using the following command.
# Jenkins highlights the password in the logs, you can't miss it
dockerlogs-fjenkins
oralternatively
dockerexecjenkinscat/var/jenkins_home/secrets/initialAdminPassword
Set the chosen credentials in the Artemis configuration
Note: The custom Jenkins Dockerfile takes advantage of the
Plugin Installation Manager Tool for Jenkins
to automatically install the plugins listed below. If you used the Dockerfile, you can skip these steps and
Server Notification Plugin.
The list of plugins is maintained in docker/jenkins/plugins.yml.
You will need to install the following plugins (apart from the
recommended ones that got installed during the setup process):
Timestamper for adding the
time to every line of the build output (Timestamper might already be installed)
Pipeline for defining the
build description using declarative files (Pipeline might already be installed)
Note: This is a suite of plugins that will install multiple plugins
Pipeline Maven to use maven within the pipelines. If you want to
use Docker for your build agents you may also need to install
Docker Pipeline .
Matrix Authorization Strategy Plugin for configuring permissions
for users on a project and build plan level (Matrix Authorization Strategy might already be installed).
The plugins above (and the pipeline-setup associated with it) got introduced in Artemis 4.7.3.
If you are using exercises that were created before 4.7.3, you also have to install these plugins:
Please note that this setup is deprecated and will be removed in the future.
Please migrate to the new pipeline-setup if possible.
Multiple SCMs for combining the
exercise test and assignment repositories in one build
Post Build Task for preparing build
results to be exported to Artemis
Xvfb for exercises based on GUI
libraries, for which tests have to have some virtual display
Choose “Download now and install after restart” and checking the
“Restart Jenkins when installation is complete and no jobs are running” box
Artemis needs to receive a notification after every build, which
contains the test results and additional commit information. For that
purpose, we developed a Jenkins plugin, that can aggregate and POST
JUnit formatted results to any URL.
You can download the current release of the plugin
here
(Download the .hpi file). Go to the Jenkins plugin page (Manage
Jenkins → System Configuration → Plugins) and install the downloaded file under the
Advanced settings tab under Deploy Plugin
Copy the generated ID (e.g. ea0e3c08-4110-4g2f-9c83-fb2cdf6345fa)
of the new credentials and put it into the Artemis configuration file
application-artemis.yml
In order to upgrade Jenkins to a newer version, you need to rebuild the Docker image targeting the new version.
The stable LTS versions can be viewed through the changelog
and the corresponding Docker image can be found on
dockerhub.
Open the Jenkins Dockerfile and replace the value of FROM with jenkins/jenkins:lts.
After running the command dockerpulljenkins/jenkins:lts, this will use the latest LTS version
in the following steps.
You can also use a specific LTS version.
For example, if you want to upgrade Jenkins to version 2.289.2, you will need to use the
jenkins/jenkins:2.289.2-lts image.
If you’re using dockercompose, you can simply use the following command and skip the next steps.
You can remove the backup container if it’s no longer needed:
dockerrmjenkins_old
You should also update the Jenkins plugins regularly due to security
reasons. You can update them directly in the Web User Interface in the
Plugin Manager.
An alternative way of adding a build agent that will use docker (similar to the remote agents below) but running
locally, can be done using the jenkins/ssh-agent docker image docker image.
You might want to run the builds on additional Jenkins agents, especially if a large amount of students should use
the system at the same time.
Jenkins supports remote build agents: The actual compilation of the students submissions happens on these
other machines but the whole process is transparent to Artemis.
This guide explains setting up a remote agent on an Ubuntu virtual machine that supports docker builds.
Add a new user to the remote machine that Jenkins will use: sudoadduser--disabled-password--gecos""jenkins
Add the jenkins user to the docker group (This allows the jenkins user to interact with docker):
sudousermod-a-Gdockerjenkins
Generate a new SSH key locally (e.g. using ssh-keygen) and add the public key to the .ssh/authorized_keys
file of the jenkins user on the agent VM.
Validate that you can connect to the build agent machine using SSH and the generated private key and validate that
you can use docker (docker ps should not show an error)
Log in with your normal account on the build agent machine and install Java: sudoaptinstalldefault-jre
Add a new secret in Jenkins, enter private key you just generated and add the passphrase, if set:
Add a new node (select a name and select Permanent Agent):
Set the number of executors so that it matches your machine’s specs: This is the number of concurrent builds
this agent can handle. It is recommended to match the number of cores of the machine,
but you might want to adjust this later if needed.
Set the remote root directory to /home/jenkins/remote_agent.
Set the usage to Only build jobs with label expressions matching this node.
This ensures that only docker-jobs will be built on this agent, and not other jobs.
Add a label docker to the agent.
Set the launch method to Launch via SSH and add the host of the machine.
Select the credentials you just created and select Manually trusted key Verification Strategy
as Host key verification Strategy.
Save it.
Wait for some moments while jenkins installs it’s remote agent on the agent’s machine.
You can track the progress using the Log page when selecting the agent. System information should also be available.
Change the settings of the master node to be used only for specific jobs.
This ensures that the docker tasks are not executed on the master agent but on the remote agent.
Artemis supports user management in Jenkins as of version 4.11.0. Creating an account in Artemis will also create an
account on Jenkins using the same password. This enables users to login and access Jenkins. Updating and/or deleting
users from Artemis will also lead to updating and/or deleting from Jenkins.
Unfortunately, Jenkins does not provide a Rest API for user management which present the following caveats:
The username of a user is treated as a unique identifier in Jenkins.
It’s not possible to update an existing user with a single request.
We update by deleting the user from Jenkins and recreating it with the updated data.
In Jenkins, users are created in an on-demand basis.
For example, when a build is performed, its change log is computed and as a result commits from users
who Jenkins has never seen may be discovered and created.
Since Jenkins users may be re-created automatically, issues may occur such as 1) creating a user, deleting it,
and then re-creating it and 2) changing the username of the user and reverting back to the previous one.
Updating a user will re-create it in Jenkins and therefore remove any additionally saved Jenkins-specific
user data such as API access tokens.
Artemis takes advantage of the Project-based Matrix Authorization Strategy plugin to support build plan
access control in Jenkins.
This enables specific Artemis users to access build plans and execute actions such as triggering a build.
This section explains the changes required in Jenkins in order to set up build plan access control:
Navigate to Manage Jenkins → Plugins → Installed plugins and make sure that you have the
Matrix Authorization Strategy plugin installed
Navigate to Manage Jenkins → Security and navigate to the “Authorization” section
Select the “Project-based Matrix Authorization Strategy” option
In the table make sure that the “Read” permission under the “Overall” section is assigned to
the “Authenticated Users” user group.
In the table make sure that all “Administer” permission is assigned to all administrators.
You are finished. If you want to fine-tune permissions assigned to teaching assistants and/or instructors,
you can change them within the JenkinsJobPermission.java file.