Date: Wed, 11 Sep 2024 11:34:35 +0200 From: Hans Verkuil Subject: Using media-ci: best practices (v3) To: linuxtv-ci@linuxtv.org, media-committers@linuxtv.org Hi all, This is version 3 of this document. gitlab is quite stable now. There have been a few changes that are incorporated into this document, mainly relating to regression testing. I strongly suggest that maintainers start using gitlab so we are all using the same build environment. In my opinion it is now in good shape. =============================================================================== Step one is to create an account for the GitLab instance: https://gitlab.freedesktop.org/freedesktop/freedesktop/-/wikis/home The next step is to get a fork of the media_staging tree so you can start developing. For that email Ricardo or Hidenori and they will arrange for this. Once a fork has been made for you you can start working. I created two remotes for that: git remote add gitlab-media git@gitlab.freedesktop.org:linux-media/media-staging.git git remote add gitlab-hverkuil git@gitlab.freedesktop.org:linux-media/users/hverkuil.git git remote update gitlab-media git remote update gitlab-hverkuil git checkout -b gitlab-media gitlab-media/master git checkout -b gitlab-hverkuil gitlab-hverkuil/master (feel free to pick other names, this is just what I use at the moment). Your tree should be visible here: https://gitlab.freedesktop.org/linux-media/users One very confusing issue when you look at that gitlab page is that the sidebar may not be visible if your browser window is too narrow. It will appear if you make it wider or if you hover at the left side of your browser window. The first thing you need to do as a maintainer is to go to the sidebar and select Settings, then CI/CD, then expand Variables. Now add a new variable FULL_CI with value 1. This enables additional tests. You can pin commonly used sidebar items so they are easier to reach. I did that for the Pipelines and CI/CD settings. Note: it is useful to read the README.md of media-ci here: https://gitlab.freedesktop.org/linux-media/media-ci It has a link to two browser extensions (one for Chrome, one for Firefox) that make it easier to read the log files. Highly recommended to install the extension! =============================================================================== Step two is to actually use the build system for testing patches. You do that by creating a branch, add the patches you want to test, and once you have pushed the branch the build will start. You can either wait for the email with the build results to arrive, or select Build/Pipelines in the sidebar to watch the progress. A note on the email: as I understand it, by default you will get an email notification if the build failed, or a previously failed build is now fixed (that can be the case if there was a network error failure for one of the jobs, see below). You do not get an email by default if the build was successful. I recommend that you enable that in your settings: on the https://gitlab.freedesktop.org/linux-media/users/ page (replace with your user ID) click on the 'bell' icon at the top, select 'Custom' (you probably have to scroll down to the last item on the list), then on the "Custom notification events" dialog that appears, check the three 'Pipeline' notifications. Since it can take quite a bit of time to finish the build, I find it useful to just push the branch and then do other things waiting for an email notification to arrive. The 'report' job at the end of the pipeline has a good overview: select it, click the 'Browse' button, then report.htm. Click the link, then you see the report. I think there is still room for improvement here with respect to the formatting, but it does a decent job. Note that you can cancel jobs in the pipeline: e.g. if you are just fixing a 'doc' issue, then you can cancel all build jobs and only leave the doc job to verify that your changes fixed the documentation problem. Something to remember is that the CI system looks at the patches between your master branch and the branch you are testing. So if your master branch is out of date compared to the upstream media_stage tree, then too many patches are checked by e.g. checkpatch. So you want to keep your master branch up to date to avoid that. I made a small gitlabupdate.sh script for that (where 'staging' is my checked out master branch of the media_stage repo): git co staging git pull git co gitlab-hverkuil git reset --hard staging git push -f gitlab-hverkuil HEAD:master I run it whenever the media_stage tree was updated. And my branch that I want to test is always rebased on top of my gitlab-hverkuil branch. Note that no build is started when you update your gitlab master branch. If for some reason you want to do that, then you need to add a RUN_CI_DEFAULT_BRANCH variable with value 1. I can't think of any reason why you would want to do that, though. =============================================================================== By default regression tests are run if: 1) Changes were made to one or more of the drivers in drivers/media/test-drivers. 2) Changes were made to the core kernel APIs for V4L2, MC, DVB or CEC. You also manually disable or enable them by setting the TEST_MEDIA_TARGET variable. The value is either 'bypass' to disable the regression tests, or the argument to the contrib/test/test-media script that is part of the v4l-utils repo. For most people it can be set to 'mc': this runs all tests, except for the cec specific tests. If you want those, then set it to 'all'. If you only changed one of the test drivers (e.g. vim2m), then set this to 'vim2m' and only the regression tests for that driver will be run. Since these build servers do not have kvm enabled, it takes quite a long time to run a standard 'mc' test: about 45 minutes. If you need to do this a lot, then it might be better to run this locally (instructions are in the media-ci README.md, but note that I have not tested that). With kvm support enabled the mc test takes less than 10 minutes. By default the regression tests use the latest v4l-utils master branch. You can select a custom v4l-utils repo/branch by setting the V4L_UTILS_REPO variable to the repo@branch. Again, I have added this, but with a _ prefix when I don't need it. Example of the value of this variable: git://linuxtv.org/hverkuil/v4l-utils.git@branch It is also possible to set CI variables when pushing the git branch: git push -o ci.variable=TEST_MEDIA_TARGET=mc \ -o ci.variable=V4L_UTILS_REPO=git://linuxtv.org/hverkuil/v4l-utils.git@branch Very useful. If you made changes that would make a standard test-media regression script to fail, then you need to provide your own v4l-utils git repo with the fixes added so the test-media script passes again. When posting your patch series you must also post a patch series with the v4l-utils changes so others can run the test. This also ensures that you are forced to keep the v4l-utils code in sync with your kernel changes. =============================================================================== Hopefully this is a useful guide for people to start with. If you encounter problems, please make a new issue for media-ci. Also, if the virtme regression tests are unstable (i.e. sometimes they produce a fail, but they work on a retry), then please let me know. It really should be stable. Regards, Hans