Chris McCall

DevOps & Software Development

GitOps: Getting Started with Helm

Chris McCall

• 8 min read
Share
GitOps: Getting Started with Helm

When I first started working in Kubernetes I tried to avoid Helm charts for my installs. I wanted to make sure I was learning how all the parts went together, so I wrote out all my resources in static yaml. That worked out well while studying for the CKA, but now it's time for GitOps at home. For me, that means moving my static yaml definitions to Helm charts.

I’m going to convert Jellyfin to start. Sure, there are plenty online to choose from, but then I wouldn’t learn anything.

I'm working inside of a repo that I intend to describe all of my applications in, so my folder structure looks like this:

/
├── applications/
├── environments/
│   └── prod
├── .gitignore
└── .gitlab-ci.yml

$ cd applications
$ helm create jellyfin

I'm not going to use everything that Helm generates, so after a second my folder structure looks like this:

/
├── applications/
│   └── jellyfin
|       └── templates
|           ├── deployment.yaml
|           ├── NOTES.txt
|           ├── _helpers.tpl
|           ├── hpa.yaml
|           ├── serviceaccount.yaml
|           ├── service.yaml
|           ├── httproute.yaml
|           └── ingress.yaml
|       ├── values.yaml
│       └── Chart.yaml
├── environments/
│   └── prod
|       └── jellyfin-values.yaml
├── .gitignore
└── .gitlab-ci.yml

Mainly, I removed the sub-charts directory and created an empty file, jellyfin-values.yaml to hold my overrides later.

Chart definition

Chart.yaml is the information about our chart, so set a description and set appVersion to the version of the application the chart will deploy. In this case it is “12.0-rc5”.

# Chart.yaml
apiVersion: v2
name: jellyfin
description: Jellyfin Server

type: application
version: 0.1.0
appVersion: "12.0-rc5"

Now let’s take a look at our current deployment and pick out what specifics, if any, are unique to this Jellyfin install.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: jellyfin-server
  name: jellyfin
  namespace: jellyfin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jellyfin-server
  template:
    metadata:
      labels:
        app: jellyfin-server
    spec:
      securityContext:
        privileged: true
      nodeSelector:
        hardwareAcceleration: qsv
      hostNetwork: true
      containers:
      - image: docker.io/jellyfin/jellyfin:12.0-rc5
        name: jellyfin
        volumeMounts:
        - name: jellyfin-config
          mountPath: /config
        - name: jellyfin-cache
          mountPath: /cache
        - name: jellyfin-media
          mountPath: /media
        - name: gpu
          mountPath: /dev/dri
        ports:
        - containerPort: 8096
          protocol: TCP
      volumes:
      - name: jellyfin-config
        hostPath:
          path: /var/lib/jellyfin/config
          type: DirectoryOrCreate
      - name: jellyfin-cache
        hostPath:
          path: /var/lib/jellyfin/cache
          type: DirectoryOrCreate
      - name: jellyfin-media
        nfs:
          server: <nas ip>
          path: <nfs path>
      - name: gpu
        hostPath:
          path: /dev/dri
          type: Directory

I'll focus on these:

  • hostNetwork needs to be set to true, and UDP port 1900 needs to be open for LAN discovery
  • Http port 8096 is the default and there isn’t a command line option to override it.
  • Needs at least 3 volumeMounts
  • securityContext and volumeMounts need certain settings if you want to use hardware transcoding
  • I want this to run on a specific node that has certain capabilities(Intel QuickSync)
  • Privileged: true. I had used this setting to get hardware acceleration working, but this should be done in a better way.

Values

With these details in mind, let's go to our chart values file and make some changes. I'll start with the easiest ones.

  • I know what the default image should be for the application
  • Jellyfin has a default http port of 8096
  • There are some volumes that have to be defined for the application to function

Image repo and port, done.

# values.yaml

image:
  repository: docker.io/jellyfin/jellyfin
  pullPolicy: IfNotPresent
  tag: ""
<...>
service:
  type: ClusterIP
  port: 8096

Next, go to the top of the file and add a jellyfin section. Define the volumes and hardware acceleration values.

# values.yaml

# Default values for jellyfin.
jellyfin:
  configVolume:
    # Not setting a configVolume will use emptyDir
    path: ""
  cacheVolume:
    # Not setting a cacheVolume will use emptyDir
    path: ""
  mediaVolume:
    # Not setting a mediaVolume will use emptyDir
    nfsServer: ""
    nfsPath: ""
  hardwareAcceleration:
    enabled: false
    # Currently only supporting the intel driver
    driver: "intel"
    # Find the group id with: getent group render
    renderGroupId: 109

Template>volumes

With these values in place I want to start implementing them in the deployment so that I can start testing the output. In templates/deployment.yaml update volumeMounts and volumes.

# templates/deployment.yaml

# This section...

		  {{- with .Values.volumeMounts }}
          volumeMounts:
            {{- toYaml . | nindent 12 }}
          {{- end }}
      {{- with .Values.volumes }}
      volumes:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      
# ...becomes this:

          volumeMounts:
            - name: jellyfin-config
              mountPath: /config
            - name: jellyfin-cache
              mountPath: /cache
            - name: jellyfin-media
              mountPath: /media
      volumes:
        - name: jellyfin-config
          {{- if .Values.jellyfin.configVolume.path }}
          hostPath:
            path: {{ .Values.jellyfin.configVolume.path }}
            type: DirectoryOrCreate
          {{- else }}
          emptyDir: {}
          {{- end }}
        - name: jellyfin-cache
          {{- if .Values.jellyfin.configVolume.path }}
          hostPath:
            path: {{ .Values.jellyfin.cacheVolume.path }}
            type: DirectoryOrCreate
          {{- else }}
          emptyDir: {}
          {{- end }}
        - name: jellyfin-media
          {{- if .Values.jellyfin.mediaVolume.nfsServer }}
          nfs:
            server: {{ .Values.jellyfin.mediaVolume.nfsServer }}
            path: {{ .Values.jellyfin.mediaVolume.nfsPath }}
          {{- else }}
          emptyDir: {}
          {{- end }}

Obviously it would be nice to support other volume types and all that, but I'll save that for another revision. Speaking of which, let’s see how close we are getting. To do this use the helm template command.

$ helm template ./jellyfin

# Source: jellyfin/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: release-name-jellyfin
  <...>
spec:
  replicas: 1
  <...>
  template:
    <...>
    spec:
      serviceAccountName: release-name-jellyfin
      containers:
        - name: jellyfin
          image: "docker.io/jellyfin/jellyfin:12.0-rc5"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8096
              protocol: TCP
          <...>
          volumeMounts:
            - name: jellyfin-config
              mountPath: /config
            - name: jellyfin-cache
              mountPath: /cache
            - name: jellyfin-media
              mountPath: /media
      volumes:
        - name: jellyfin-config
          emptyDir: {}
        - name: jellyfin-cache
          emptyDir: {}
        - name: jellyfin-media
          emptyDir: {}

The template rendered and our defaults are working, now I want to test it with some override values. Edit the empty jellyfin-values.yaml file.

# environments/prod/jellyfin-values.yaml

jellyfin:
  configVolume:
    path: "/config/path"
  cacheVolume:
    # Not setting a cacheVolume will use emptyDir
    path: "/cache/path"
  mediaVolume:
    # Not setting a mediaVolume will use emptyDir
    nfsServer: "192.168.1.2"
    nfsPath: "/nfs/path"

Run the template command again, but passing in the override values with the -f option this time.

$ helm template ./jellyfin -f ../environments/prod/jellyfin-values.yaml

# Source: jellyfin/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  <...>
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: jellyfin
      app.kubernetes.io/instance: release-name
  template:
    <...>
    spec:
      serviceAccountName: release-name-jellyfin
      containers:
        - name: jellyfin
          image: "docker.io/jellyfin/jellyfin:12.0-rc5"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8096
              protocol: TCP
          <...>
          volumeMounts:
            - name: jellyfin-config
              mountPath: /config
            - name: jellyfin-cache
              mountPath: /cache
            - name: jellyfin-media
              mountPath: /media
      volumes:
        - name: jellyfin-config
          hostPath:
            path: /config/path
            type: DirectoryOrCreate
        - name: jellyfin-cache
          hostPath:
            path: /cache/path
            type: DirectoryOrCreate
        - name: jellyfin-media
          nfs:
            server: 192.168.1.2
            path: /nfs/path

Now for the hardware acceleration logic.

Template>hardware acceleration

To use hardware acceleration we need to set the groupId in the pod security context, add a new resource for the intel driver, and use the driver in the container by specifying a resource limit.

This gets a little messy to view here, but the gist of these sections is that I wanted to allow securityContext to be set alongside the renderGroupId without one overwriting the other. Same deal with resources.limits.

# templates/deployment.yaml

# Output the renderGroupId, but join it with any other group ids
# that were defined in podSecurityContext

# spec.template.spec
      {{- if or .Values.podSecurityContext  .Values.jellyfin.hardwareAcceleration.enabled }}
        {{- $securityContext := deepCopy .Values.podSecurityContext | default (dict) }}
        {{- /* Combine contextGroups if defined in both podSecurityContext and hardwareAcceleration */ -}}
        {{- if .Values.jellyfin.hardwareAcceleration.enabled }}
          {{- $podSecurityContextGroups := default (list) $securityContext.supplementalGroups }}
          {{- $_ := set $securityContext "supplementalGroups" (concat $podSecurityContextGroups (list .Values.jellyfin.hardwareAcceleration.renderGroupId)) }}
        {{- end }}
      securityContext:
        {{- toYaml $securityContext | nindent 8 }}
      {{- end }}

# Configure limits in resources to use the intel driver
# Combine limits with any limits defined in .Values.resources

# spec.template.spec.containers[0].resources
          {{- if or .Values.resources .Values.jellyfin.hardwareAcceleration.enabled }}
            {{- $resourceDef := deepCopy .Values.resources | default (dict) }}
            {{- /* Combine limits if defined in resources and hardwareAcceleration is enabled */ -}}
            {{- if .Values.jellyfin.hardwareAcceleration.enabled }}
              {{- $resourceLimits := default (dict) $resourceDef.limits }}
              {{- if eq .Values.jellyfin.hardwareAcceleration.driver "intel" }}
                {{- $_ := set $resourceLimits "gpu.intel.com/i915" 1 }}
                {{- $_ := set $resourceDef "limits" $resourceLimits }}
              {{- end }}
            {{- end }}
          resources:
            {{- toYaml $resourceDef | nindent 12 }}
          {{- end }}

Intel GPU driver

The Intel GPU driver is what allows the removal of the privileged security context. Intel's installation instructions have this running on every node, but my machines are very old and slow so I want to cut out anything I don't need.

Create a new file, intelgpudriver.yaml that defines the intel driver. I created the driver file from editing the dry-run output of intel’s alternative installation method. I only want this to deploy on the same node as Jellyfin, so I converted the daemonset to a deployment.

$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin?ref=v0.36.0' \
--dry-run=client -o yaml > intelgpudriver.yaml

# templates/intelgpudriver.yaml

{{- if and .Values.jellyfin.hardwareAcceleration.enabled (eq .Values.jellyfin.hardwareAcceleration.driver "intel") }}
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: intel-gpu-plugin
  name: intel-gpu-plugin
spec:
  selector:
    matchLabels:
      app: intel-gpu-plugin
  template:
    metadata:
      labels:
        app: intel-gpu-plugin
    spec:
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        image: intel/intel-gpu-plugin:0.36.0
        imagePullPolicy: IfNotPresent
        name: intel-gpu-plugin
        resources:
          limits:
            cpu: 100m
            memory: 90Mi
          requests:
            cpu: 40m
            memory: 45Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          seLinuxOptions:
            type: container_device_plugin_t
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - mountPath: /dev/dri
          name: devfs
          readOnly: true
        - mountPath: /sys/class/drm
          name: sysfsdrm
          readOnly: true
        - mountPath: /var/lib/kubelet/device-plugins
          name: kubeletsockets
        - mountPath: /var/run/cdi
          name: cdipath
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      volumes:
      - hostPath:
          path: /dev/dri
        name: devfs
      - hostPath:
          path: /sys/class/drm
        name: sysfsdrm
      - hostPath:
          path: /var/lib/kubelet/device-plugins
        name: kubeletsockets
      - hostPath:
          path: /var/run/cdi
          type: DirectoryOrCreate
        name: cdipath
{{- end }}

Client Discovery

Client discovery allows Jellyfin clients to find your server with UDP broadcast. As far as I can tell this only works with host networking. I have an idea for how I might want to solve this, but haven't worked on it yet. If you have other ideas, lemme know.

# values.yaml

# Default values for jellyfin.
jellyfin:
  configVolume:
    # Not setting a configVolume will use emptyDir
    path: ""
  cacheVolume:
    # Not setting a cacheVolume will use emptyDir
    path: ""
  mediaVolume:
    # Not setting a mediaVolume will use emptyDir
    nfsServer: ""
    nfsPath: ""
  hardwareAcceleration:
    enabled: false
    # Find the groupId with: getent group render
    renderGroupId: 109
  # Client discovery allows LAN clients to discover the server automatically.
  # enabling this features will set hostNetwork to true
  localClientDiscovery:
    enabled: false

# templates/deployment.yaml

<...>
      {{- if .Values.jellyfin.localClientDiscovery.enabled }}
      hostNetwork: true
      {{- end }}

Run the template command again to test the output.

Great, now the chart can match the original deployment. This feels like a good base to continue developing my chart as time goes on. I'm done with the template rendering, but still want to do a little more to wrap up. I think it would be nice to warn against a few gotchas when someone creates a release. What if they(me) forget to set a config path, which will default to emptyDir, and spend a bunch of time setting up the server? If I change the port, I might want to be reminded that for this application you still have to edit the port value inside of the web UI. We can do these things inside of NOTES.txt.

Notes

Notes defines messages that can be presented after an install. They are processed through the template engine, so we can use the syntax we've been using. Helm already created a default set of messages that print out how to access the application. I'm going to leave that in there and just put my new stuff above it.

These messages go into the top of NOTES.txt:

{{- if ne (int .Values.service.port) 8096 }}
- Default port changed:
  To change the port that Jellyfin listens, you must also change the port value from inside the web admin interface.

{{- end }}
{{- if not .Values.jellyfin.configVolume.path }}
- jellyfin.configVolume.path not set, will default to emptyDir

{{- end }}
{{- if not .Values.jellyfin.cacheVolume.path }}
- jellyfin.cacheVolume.path not set, will default to emptyDir

{{- end }}
{{- if not .Values.jellyfin.mediaVolume.nfsPath }}
- jellyfin.mediaVolume.nfsPath not set, will default to emptyDir

{{- end }}

The notes don’t output with the template command, so to test we have to use install with dry-run:

$ helm install jellyfin ./jellyfin \
-f ../environments/prod/jellyfin-values.yaml --dry-run

Install!

Now prove your work. Install with helm:

$ helm install jellyfin ./jellyfin \
-f ../environments/prod/jellyfin-values.yaml