diff --git a/jellyfin-kubernetes-sop.md b/jellyfin-kubernetes-sop.md new file mode 100644 index 0000000..75105a2 --- /dev/null +++ b/jellyfin-kubernetes-sop.md @@ -0,0 +1,603 @@ +# Jellyfin on Kubernetes --- Deployment & Operations SOP + +> **Scope:** Deploy Jellyfin with Helm on Kubernetes, pin it to the +> dedicated storage node, persist config/cache on Longhorn, expose it +> through NodePort/Nginx Proxy Manager, and upload media through a +> multi-hop SSH path. + +## 1. Final Architecture + +``` text +User / Browser + | + | HTTPS: media.jutsugen.top + v +Nginx Proxy Manager + | + | HTTP -> Kubernetes NodeIP:31096 + v +NodePort Service + | + v +Jellyfin Pod :8096 + | + +-- /config -> Longhorn PVC (5Gi) + +-- /cache -> Longhorn PVC (10Gi) + +-- /media -> storage-node hostPath (read-only) + +-- /downloads -> storage-node hostPath +``` + +Dedicated node: + +``` text +kurasuta-storage-vm-exp +Internal IP: 10.20.30.32 +Label: kubernetes.io/hostname=kurasuta-storage-vm-exp +Taint: storage=true:NoSchedule +``` + +## 2. Prerequisites + +- Working Kubernetes cluster and Helm. +- Longhorn StorageClass available (`longhorn`). +- Storage node labelled and tainted as shown above. +- Nginx Proxy Manager can reach a Kubernetes node IP. +- DNS record for `media.jutsugen.top` points to the NPM entry point. +- A `jellyfin` Linux user exists on the storage node with UID/GID + `1001:1001`. + +## 3. Storage Node Directories + +On `kurasuta-storage-vm-exp`, use: + +``` text +/home/jellyfin/jellyfin/ +├── config/ +├── downloads/ +└── media/ + ├── movies/ + └── series/ +``` + +Verify identity and directory ownership: + +``` bash +id jellyfin +ls -ld /home/jellyfin/jellyfin /home/jellyfin/jellyfin/* +``` + +Expected Jellyfin identity: + +``` text +uid=1001(jellyfin) gid=1001(jellyfin) +``` + +Host media/download directories should be accessible to that user. + +## 4. Storage Design + +Use the following split: + + ----------------------------------------------------------------------- + Container path Backend Purpose + ----------------------- ----------------------- ----------------------- + `/config` Longhorn PVC, 5Gi Jellyfin DB, users, + settings, metadata + state + + `/cache` Longhorn PVC, 10Gi Cache/transcoding + working data + + `/media` hostPath Movies and series + + `/downloads` hostPath Downloader staging area + ----------------------------------------------------------------------- + +Do **not** define duplicate `config` or `media` volumes. The chart's +built-in persistence configuration and custom volumes must not produce +two volumes with the same name or two mounts on the same path. + +## 5. Pod Security Context + +The official Jellyfin image is used: + +``` text +docker.io/jellyfin/jellyfin:latest +``` + +Run Jellyfin as the storage user's UID/GID: + +``` yaml +podSecurityContext: + runAsUser: 1001 + runAsGroup: 1001 + fsGroup: 1001 +``` + +`fsGroup: 1001` is important for writable Longhorn PVCs. Without it, +Jellyfin may fail with an error similar to: + +``` text +System.UnauthorizedAccessException: Access to the path '/config/log' is denied. +Permission denied +``` + +## 6. Scheduling on the Storage Node + +Pin Jellyfin to the storage node: + +``` yaml +nodeSelector: + kubernetes.io/hostname: kurasuta-storage-vm-exp +``` + +The node has this taint: + +``` text +storage=true:NoSchedule +``` + +Therefore the pod requires a matching toleration: + +``` yaml +tolerations: + - key: storage + operator: Equal + value: "true" + effect: NoSchedule +``` + +Without this toleration, the node selector can match but the pod will +remain unschedulable. + +## 7. Media and Download hostPaths + +The final hostPath mapping is: + +``` yaml +volumes: + - name: media + hostPath: + path: /home/jellyfin/jellyfin/media + type: Directory + - name: downloads + hostPath: + path: /home/jellyfin/jellyfin/downloads +``` + +Container mounts: + +``` yaml +volumeMounts: + - name: media + mountPath: /media + readOnly: true + - name: downloads + mountPath: /downloads +``` + +Keeping `/media` read-only protects source media from accidental +modification/deletion by Jellyfin. + +## 8. Network Exposure + +Kubernetes Ingress is not required because Nginx Proxy Manager is the +external reverse proxy. + +Expose Jellyfin as NodePort: + +``` text +Service type: NodePort +Service port: 8096 +NodePort: 31096 +``` + +Expected service: + +``` text +jellyfin NodePort 8096:31096/TCP +``` + +Traffic path: + +``` text +NPM -> http://10.20.30.32:31096 -> Service -> Jellyfin Pod:8096 +``` + +Do not use HTTPS directly against port `31096`; this backend is HTTP. +TLS is terminated by Nginx Proxy Manager. + +## 9. Helm Validation Before Deployment + +From the chart directory, validate values/chart syntax: + +``` bash +helm lint -f override_values.yaml . +``` + +Expected: + +``` text +1 chart(s) linted, 0 chart(s) failed +``` + +Render the chart before installation and inspect it: + +``` bash +helm template jellyfin -f override_values.yaml . > jellyfin.yaml +``` + +Verify the rendered Deployment contains: + +``` text +image: docker.io/jellyfin/jellyfin:latest +runAsUser: 1001 +runAsGroup: 1001 +fsGroup: 1001 +/config -> jellyfin-config PVC +/cache -> jellyfin-cache PVC +/media -> /home/jellyfin/jellyfin/media +/downloads -> /home/jellyfin/jellyfin/downloads +nodeSelector -> kurasuta-storage-vm-exp +toleration -> storage=true:NoSchedule +``` + +Also verify there are no duplicate `config`/`media` volume names or +mount paths. + +## 10. Deploy Jellyfin + +Create the namespace: + +``` bash +kubectl create namespace jellyfin +``` + +Install/upgrade the release: + +``` bash +helm upgrade --install jellyfin \ + --namespace jellyfin \ + -f override_values.yaml \ + . +``` + +Helm release status must include the namespace: + +``` bash +helm status jellyfin --namespace jellyfin +``` + +A Helm status of `deployed` only confirms resource submission; always +verify the pod separately. + +## 11. Runtime Verification + +Check the pod: + +``` bash +kubectl get pods -n jellyfin -o wide +``` + +Expected: + +``` text +READY STATUS RESTARTS +1/1 Running 0 +``` + +Confirm it is scheduled on: + +``` text +kurasuta-storage-vm-exp +``` + +Check PVCs: + +``` bash +kubectl get pvc -n jellyfin +``` + +Both config and cache claims should be `Bound`. + +If the pod crashes: + +``` bash +kubectl describe pod -n jellyfin +kubectl logs -n jellyfin +``` + +The permission issue encountered during this deployment was fixed by +ensuring the Longhorn volume had appropriate group access through the +pod security context (`fsGroup`). + +## 12. Service and NodePort Verification + +Check Service: + +``` bash +kubectl get svc -n jellyfin +``` + +Check EndpointSlice: + +``` bash +kubectl get endpointslices.discovery.k8s.io -n jellyfin +``` + +The Jellyfin endpoint should point to the pod on port `8096`. + +Test ClusterIP: + +``` bash +curl -v --max-time 5 http://:8096/ +``` + +Test NodePort: + +``` bash +curl -v --max-time 5 http://10.20.30.32:31096/ +``` + +A successful Jellyfin root request can return: + +``` text +HTTP/1.1 302 Found +Server: Kestrel +Location: web/ +``` + +NodePort was also verified through the other cluster node IPs, +confirming kube-proxy forwarding was functional. + +## 13. kube-proxy Notes + +The cluster's kube-proxy reported: + +``` text +Using iptables proxy +Using iptables Proxier +``` + +The ConfigMap had `mode: ""`, but runtime logs confirmed iptables mode. + +Historical kube-proxy logs also contained RBAC `forbidden` errors for +watching Services, EndpointSlices, Nodes, and ServiceCIDRs. Because +current ClusterIP and NodePort forwarding worked, these errors were not +the Jellyfin blocker, but they should be audited separately as cluster +maintenance. + +## 14. Nginx Proxy Manager + +Create a Proxy Host with: + +``` text +Domain: media.jutsugen.top +Scheme: http +Forward IP: 10.20.30.32 +Forward Port: 31096 +WebSockets: enabled +``` + +First verify HTTP proxying. After the dashboard works through the +domain, configure the SSL certificate in NPM and force HTTPS as +appropriate. + +Final path: + +``` text +https://media.jutsugen.top + | + v +Nginx Proxy Manager (TLS termination) + | + | HTTP + v +10.20.30.32:31096 + | + v +Jellyfin +``` + +## 15. Jellyfin Library Layout + +Recommended host layout: + +``` text +/home/jellyfin/jellyfin/media/ +├── movies/ +│ ├── Interstellar (2014)/ +│ │ └── Interstellar (2014).mkv +│ └── Inception (2010)/ +│ └── Inception (2010).mkv +└── series/ + └── Breaking Bad/ + └── Season 01/ + ├── Breaking Bad S01E01.mkv + └── Breaking Bad S01E02.mkv +``` + +Inside Jellyfin, use container paths---not host paths: + +``` text +Movies library -> /media/movies +Shows library -> /media/series +``` + +Do not configure `/home/jellyfin/...` in the Jellyfin UI because that +path exists on the Kubernetes node, not inside the container. + +## 16. Manual Media Upload Through Multi-Hop SSH + +Network path: + +``` text +Windows PC + | + v +Jump Server: 36.255.71.187:22001 + | + v +Master: 103.209.42.105:22002 + | + v +Storage: 10.20.30.32:22 +``` + +The first jump requires password + MFA. The master is reachable only +through the first jump and uses a private key already present on that +jump server. + +A standard multi-hop `ProxyJump` from Windows failed because the Windows +SSH client tried to authenticate to the second hop itself. The chosen +pattern was to let Jump-1 execute the second SSH hop and use its +existing key. + +Example Windows SSH config: + +``` sshconfig +Host jump1 + HostName 36.255.71.187 + User mosiur + Port 22001 + +Host storage-node + HostName 10.20.30.32 + User root + Port 22 + ProxyCommand ssh jump1 "ssh -p 22002 -W %h:%p mosiur@103.209.42.105" +``` + +Test: + +``` powershell +ssh storage-node +``` + +Once authentication is working, copy a movie: + +``` powershell +scp "C:\Movies\movie.mkv" storage-node:/home/jellyfin/jellyfin/media/movies/ +``` + +Or a directory: + +``` powershell +scp -r "C:\Movies\Interstellar (2014)" storage-node:/home/jellyfin/jellyfin/media/movies/ +``` + +### SSH troubleshooting notes + +`Permission denied (publickey)` on the second hop means the network +connection reached the server but authentication failed. + +`Host key verification failed` means server identity verification +failed, not MFA/password validation. Verify host-key fingerprints before +accepting changed/new host keys. + +For storage root login, `PermitRootLogin yes` alone does not guarantee +password authentication. Effective SSH configuration, PAM, password +state, `Match` blocks, and `PasswordAuthentication` can still affect +login. + +## 17. Adding Media to Jellyfin + +After the file exists on the storage node under: + +``` text +/home/jellyfin/jellyfin/media/movies/ +``` + +open Jellyfin: + +``` text +Dashboard -> Libraries -> Add Media Library +``` + +For movies: + +``` text +Content type: Movies +Display name: Movies +Folder: /media/movies +``` + +For series: + +``` text +Content type: Shows +Display name: Series +Folder: /media/series +``` + +Save and run a library scan if required. Test one media file end-to-end +before bulk uploading the library. + +## 18. Optional Automated Download Architecture + +For an automated workflow, the planned stack can be: + +``` text +Prowlarr -> Radarr/Sonarr -> qBittorrent -> /downloads + | + v + organized import + | + v + /media/movies or /media/series + | + v + Jellyfin +``` + +Deploy and validate one component at a time, starting with the download +client. Only use sources/content you are authorized to download. + +## 19. Quick Troubleshooting Matrix + + ----------------------------------------------------------------------- + Symptom First check + ----------------------------------- ----------------------------------- + Pod `Pending` Node selector, taint/toleration, + PVC binding + + `/config/log` permission denied `fsGroup`, PVC permissions + + Pod `CrashLoopBackOff` `kubectl logs` + + `kubectl describe pod` + + Service has no backend EndpointSlice / selector + + ClusterIP works, NodePort fails kube-proxy/NodePort path + + HTTPS to `:31096` gives Use HTTP; TLS terminates at NPM + wrong-version error + + `Permission denied (publickey)` Authentication/key for that hop + over SSH + + `Host key verification failed` `known_hosts`/host fingerprint + + Jellyfin cannot see uploaded movie File location, permissions, + `/media/...` library path + ----------------------------------------------------------------------- + +## 20. Final State Checklist + +``` text +[✓] Dedicated storage node selected +[✓] storage=true:NoSchedule tolerated +[✓] Jellyfin runs as UID/GID 1001 +[✓] Config persisted on Longhorn +[✓] Cache persisted on Longhorn +[✓] Media mounted from storage node +[✓] Downloads directory mounted +[✓] Jellyfin pod 1/1 Running +[✓] ClusterIP functional +[✓] NodePort 31096 functional +[✓] Nginx Proxy Manager routing functional +[✓] Jellyfin dashboard reachable +[✓] Multi-hop media transfer path established +[→] Create Movies/Series libraries and add media +[ ] Optional automated downloader stack +```