fix(playout): improve the way hashlib is called in libretime_playout/player (#3135)

### Description

Improves the way hashlib is called in libretime_playout/player so that
is isn't broken on systems with python < 3.9

The way it is currently called in
site-packages/libretime_playout/player/file.py, in the section where
scheduled files are copied to the cache dir for playout, specifies the
usedforsecurity=False flag as follows:
`hasher = hashlib.md5(usedforsecurity=False)`

hashlib.md5 did not support this flag until Python 3.9. Attempting to
specify the flag directly as an argument to hashlib.md5(), in an older
python environment (such as that in Ubuntu Focal 20.04), is unsafe, and
can cause hashlib.md5() to balk/throw an exception, which results in
file copy operations failing. This then precipitates into playout
problems, as scheduled media is missing from the cache folder.


This PR instead calls using hashlib.new(), and passes the argument to
that, as follows:
`hasher = hashlib.new('md5', usedforsecurity=False)`

This method of calling with the flag argument is safer, because the
constructor will take it or leave it gracefully, regardless of whether
the system has a version of hashlib that supports the `usedforsecurity`
flag. AFAICT, it improves (fixes) function on older systems without
negatively impacting others.

### Testing Notes

**What I did:**
Before applying this patch, we were experiencing occasional but fairly
regular periods of silence when the system was supposed to be playing a
song or track. This behavior was consistent with errors such as the
following being present in the playout log:
```
2025-01-15 14:01:28,331 | INFO     | libretime_playout.player.file:copy_file:47 - copying file 19834 to cache /var/lib/libretime/playout/scheduler/19834.mp3
2025-01-15 14:01:28,466 | ERROR    | libretime_playout.player.file:copy_file:77 - could not copy file 19834 to /var/lib/libretime/playout/scheduler/19834.mp3: 'usedforsecurity' is an invalid keyword argument for openssl_md5()
Traceback (most recent call last):
  File "/opt/libretime/lib/python3.8/site-packages/libretime_playout/player/file.py", line 70, in copy_file
    file_event.filesize = self.report_file_size_and_md5_to_api(
  File "/opt/libretime/lib/python3.8/site-packages/libretime_playout/player/file.py", line 89, in report_file_size_and_md5_to_api
    hasher = hashlib.md5(usedforsecurity=False)
TypeError: 'usedforsecurity' is an invalid keyword argument for openssl_md5()
```
_For more information about the characterization and results of this
problem, see issue #3134_

**Testing on running systems**
After the patch was applied, these errors were no longer seen. I first
tested this on a dev server, and then on a live server, with
approximately 100 distinct tracks of playout occurring over about 24
hours. There were no file copy failures, and no related playout
problems, which was a major and immediate improvement.

**Testing installer, fresh installs**
***Ubuntu 20.04***
I deployed a patch to the installer and installed it on a blank system
running Ubuntu 20.04 and tested it to make sure the fix was applied and
worked.

***Debian 11***
I deployed patch to the installer and installed it on a blank system
running Debian Bullseye (which runs python = 3.9) , and tested it there
to make sure it did not break anything or introduce a regression.


### **Links**

Closes: #3134

.chglog/CHANGELOG.md.tpl

@@ -1,22 +0,0 @@
-{{ range .Versions }}<a name="{{ .Tag.Name }}"></a>
-
-## {{ if .Tag.Previous }}[{{ .Tag.Name }}]({{ $.Info.RepositoryURL }}/compare/{{ .Tag.Previous.Name }}...{{ .Tag.Name }}){{ else }}{{ .Tag.Name }}{{ end }} ({{ datetime "2006-01-02" .Tag.Date }})
-
-- [Release note](https://libretime.org/docs/releases/{{ .Tag.Name }}/)
-
-{{ range .CommitGroups -}}
-### {{ .Title }}
-
-{{ range reverse .Commits -}}
-- {{ if .Scope }}**{{ .Scope }}:** {{ end }}{{ .Subject }}
-{{ end }}
-{{ end -}}
-
-{{- if .RevertCommits -}}
-### Reverts
-
-{{ range .RevertCommits -}}
-- {{ .Revert.Header }}
-{{ end }}
-{{ end -}}
-{{ end -}}

.chglog/config.yml

@@ -1,25 +0,0 @@
-style: github
-template: CHANGELOG.md.tpl
-info:
-  title: CHANGELOG
-  repository_url: https://github.com/libretime/libretime
-options:
-  commits:
-    filters:
-      Type: [feat, fix, docs, test, ci]
-    sort_by: Date
-  commit_groups:
-    title_maps:
-      feat: Features
-      fix: Bug Fixes
-      docs: Documentation
-      test: Tests
-      ci: CI
-    sort_by: Custom
-    title_order: [feat, fix, docs, test, ci]
-  header:
-    pattern: "^(\\w*)(?:\\(([\\w\\$\\.\\-\\*\\s]*)\\))?\\:\\s(.*)$"
-    pattern_maps:
-      - Type
-      - Scope
-      - Subject

.codespellignore

@@ -4,6 +4,8 @@ HDA
 ro
 
 # Names
+conet
+falso
 flor
 
 # TODO: See https://github.com/savonet/liquidsoap/issues/1654

.env.dev

@@ -1,3 +1,3 @@
 LIBRETIME_VERSION=main
-LIBRETIME_CONFIG_FILEPATH=./docker/config.dev.yml
+LIBRETIME_CONFIG_FILEPATH=./dev/config.yml
 NGINX_CONFIG_FILEPATH=./docker/nginx.conf

.github/ISSUE_TEMPLATE/config.yml

@@ -5,6 +5,6 @@ contact_links:
     url: https://discourse.libretime.org/
     about: Please find existing questions and discussions here.
 
-  - name: LibreTime Chat
+  - name: LibreTime Chat (#libretime:matrix.org)
-    url: https://chat.libretime.org/
+    url: https://matrix.to/#/#libretime:matrix.org
     about: Discuss with the LibreTime community.

.github/release-please-config.json

@@ -0,0 +1,20 @@
+{
+  "$schema": "https://raw.githubusercontent.com/googleapis/release-please/main/schemas/config.json",
+  "bootstrap-sha": "26737abad231d96fc198fbf12c043f2d867be79c",
+  "include-component-in-tag": false,
+  "include-v-in-tag": false,
+  "packages": {
+    ".": {
+      "release-type": "simple",
+      "package-name": "libretime",
+      "extra-files": [
+        "analyzer/setup.py",
+        "api/setup.py",
+        "api-client/setup.py",
+        "playout/setup.py",
+        "shared/setup.py",
+        "worker/setup.py"
+      ]
+    }
+  }
+}

.github/release-please-manifest.json

@@ -0,0 +1 @@
+{".":"4.2.0"}

.github/renovate.json

@@ -6,8 +6,11 @@
     "commitMessageAction": "lock file maintenance",
     "commitMessageExtra": "({{packageFile}})",
     "branchTopic": "lock-file-maintenance-{{packageFile}}",
-    "schedule": ["after 4am and before 5am on monday"]
+    "schedule": ["after 4am and before 5am on monday"],
+    "automerge": true,
+    "automergeType": "branch"
   },
+  "baseBranches": ["main"],
   "labels": ["dependencies"],
   "packageRules": [
     {
@@ -24,17 +27,9 @@
       "rangeStrategy": "widen"
     },
     {
-      "matchPaths": ["website/**"],
+      "matchManagers": ["github-actions", "pre-commit"],
-      "addLabels": ["javascript"]
+      "automerge": true,
-    },
+      "automergeType": "branch"
-    {
-      "matchUpdateTypes": ["patch"],
-      "matchPaths": [
-        ".github/workflows/*",
-        ".pre-commit-config.yaml",
-        "website/**"
-      ],
-      "automerge": true
     }
   ]
 }

.github/stale.yml

@@ -1,57 +0,0 @@
-# Configuration for probot-stale - https://github.com/probot/stale
-
-# Number of days of inactivity before an Issue or Pull Request becomes stale (5 months)
-daysUntilStale: 150
-
-# Number of days of inactivity before an Issue or Pull Request with the stale label is closed. (1 month)
-# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale.
-daysUntilClose: 30
-
-# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable
-exemptLabels:
-  - "status: pinned"
-  - "status: maybe later"
-  - "security"
-
-# Set to true to ignore issues in a project (defaults to false)
-exemptProjects: false
-
-# Set to true to ignore issues in a milestone (defaults to false)
-exemptMilestones: true
-
-# Set to true to ignore issues with an assignee (defaults to false)
-exemptAssignees: true
-
-# Label to use when marking as stale
-staleLabel: "status: stalled"
-
-# Comment to post when marking as stale. Set to `false` to disable
-markComment: >
-  This issue has been automatically marked as stale because it has not had
-  activity in the last 5 months. It will be closed if no activity occurs in
-  the next month.
-
-  Please chat to us on [discourse](https://discourse.libretime.org/) or
-  ask for help on our [chat](https://chat.libretime.org/) if you have any
-  questions or need further support with getting this issue resolved.
-
-  You may also label an issue as *pinned* if you would like to make sure
-  that it does not get closed by this bot.
-
-# Comment to post when removing the stale label.
-# unmarkComment: >
-#   Your comment here.
-
-# Comment to post when closing a stale Issue or Pull Request.
-closeComment: >
-  This issue has been automatically closed after is was marked as stale and
-  did not receive any further inputs.
-
-  Feel free to let us know on [discourse](https://discourse.libretime.org/) or
-  ask for help on our [chat](https://chat.libretime.org/) if you feel this
-  issue should not have been closed.
-
-  Thank you for your contributions.
-
-# Limit the number of actions per hour, from 1-30. Default is 30
-limitPerRun: 30

.github/workflows/_python.yml

@@ -25,11 +25,11 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: "3.x"
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: ~/.cache/pip
           key: ${{ runner.os }}-pip-${{ inputs.context }}-${{ hashFiles(format('{0}/{1}', inputs.context, '**/setup.py')) }}
@@ -51,10 +51,8 @@ jobs:
       fail-fast: false
       matrix:
         release:
-          - buster
-          - bullseye
-          - bionic
           - focal
+          - bullseye
           - jammy
 
     container:
@@ -65,9 +63,9 @@ jobs:
         shell: bash
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: ~/.cache/pip
           key: ${{ matrix.release }}-pip-${{ inputs.context }}-${{ hashFiles(format('{0}/{1}', inputs.context, '**/setup.py')) }}
@@ -75,11 +73,11 @@ jobs:
             ${{ matrix.release }}-pip-${{ inputs.context }}
 
       - name: Test
-        run: make test
+        run: make test-coverage
         working-directory: ${{ inputs.context }}
 
       - name: Report coverage
-        uses: codecov/codecov-action@v3
+        uses: codecov/codecov-action@v5
         with:
           files: ${{ inputs.context }}/coverage.xml
           flags: ${{ inputs.context }}

.github/workflows/actions/build-container/action.yml

@@ -1,31 +0,0 @@
-name: Build container
-description: Build and push a container
-
-inputs:
-  target:
-    required: true
-
-runs:
-  using: composite
-  steps:
-    - uses: docker/metadata-action@v4
-      id: meta
-      with:
-        images: ${{ env.REGISTRY }}/${{ env.NAMESPACE }}/${{ inputs.target }}
-        tags: |
-          type=ref,event=branch
-          type=semver,pattern={{version}}
-          type=semver,pattern={{major}}.{{minor}}
-
-    - uses: docker/build-push-action@v3
-      with:
-        context: .
-        pull: true
-        push: ${{ github.event_name == 'push' }}
-        build-args: |
-          LIBRETIME_VERSION=${{ env.LIBRETIME_VERSION }}
-        target: ${{ inputs.target }}
-        tags: ${{ steps.meta.outputs.tags }}
-        labels: ${{ steps.meta.outputs.labels }}
-        cache-from: type=gha,scope=${{ inputs.target }}
-        cache-to: type=gha,scope=${{ inputs.target }},mode=max

.github/workflows/analyzer.yml

@@ -1,21 +1,27 @@
 name: Analyzer
 
 on:
+  workflow_dispatch:
   push:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/analyzer.yml
       - analyzer/**
       - shared/**
+      - tools/python*
 
   pull_request:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/analyzer.yml
       - analyzer/**
       - shared/**
+      - tools/python*
+
+  schedule:
+    - cron: 0 1 * * 1
 
 jobs:
   python:

.github/workflows/api-client.yml

@@ -1,21 +1,27 @@
 name: API Client
 
 on:
+  workflow_dispatch:
   push:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/api-client.yml
       - api-client/**
       - shared/**
+      - tools/python*
 
   pull_request:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/api-client.yml
       - api-client/**
       - shared/**
+      - tools/python*
+
+  schedule:
+    - cron: 0 1 * * 1
 
 jobs:
   python:

.github/workflows/api-schema.yml

@@ -1,6 +1,7 @@
 name: API schema
 
 on:
+  workflow_dispatch:
   push:
     branches: [main]
     paths:
@@ -20,15 +21,15 @@ jobs:
   check:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
 
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: "3.x"
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: ~/.cache/pip
           key: ${{ runner.os }}-pip-api-${{ hashFiles('api/**/setup.py') }}
@@ -41,7 +42,7 @@ jobs:
 
       - name: Get pull request commit range
         if: github.event_name == 'pull_request'
-        run: echo "COMMIT_RANGE=origin/${{ github.base_ref }}..${{ github.sha }}" >> $GITHUB_ENV
+        run: echo "COMMIT_RANGE=${{ github.sha }}~1...${{ github.sha }}" >> $GITHUB_ENV
 
       - name: Get push commit range
         if: github.event_name == 'push'
@@ -53,11 +54,11 @@ jobs:
               git checkout $commit
 
               make --quiet schema
+              git diff -- schema.yml
               git add schema.yml
               git diff-index --quiet HEAD -- || {
                 echo "ERROR: Schema is outdated for commit $commit"
                 git show --quiet
-                git diff -- schema.yml
                 exit 1
               }
           done
@@ -70,11 +71,11 @@ jobs:
 
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
 
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           repository: libretime/client
           path: client

.github/workflows/api.yml

@@ -1,21 +1,27 @@
 name: API
 
 on:
+  workflow_dispatch:
   push:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/api.yml
       - api/**
       - shared/**
+      - tools/python*
 
   pull_request:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/api.yml
       - api/**
       - shared/**
+      - tools/python*
+
+  schedule:
+    - cron: 0 1 * * 1
 
 jobs:
   python:
@@ -30,10 +36,8 @@ jobs:
       fail-fast: false
       matrix:
         release:
-          - buster
-          - bullseye
-          - bionic
           - focal
+          - bullseye
           - jammy
 
     services:
@@ -58,9 +62,9 @@ jobs:
       LIBRETIME_DATABASE_HOST: postgres
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: ~/.cache/pip
           key: ${{ matrix.release }}-pip-api-${{ hashFiles('api/**/setup.py') }}
@@ -68,11 +72,11 @@ jobs:
             ${{ matrix.release }}-pip-api
 
       - name: Test
-        run: make test
+        run: make test-coverage
         working-directory: api
 
       - name: Report coverage
-        uses: codecov/codecov-action@v3
+        uses: codecov/codecov-action@v5
         with:
           files: api/coverage.xml
           flags: api

.github/workflows/backport.yml

@@ -0,0 +1,34 @@
+name: Backport
+
+on:
+  pull_request_target:
+    types:
+      - closed
+      - labeled
+
+jobs:
+  backport:
+    runs-on: ubuntu-latest
+
+    # Only react to merged PRs for security reasons.
+    # See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target.
+    if: >
+      github.event.pull_request.merged
+      && (
+        github.event.action == 'closed'
+        || (
+          github.event.action == 'labeled'
+          && contains(github.event.label.name, 'backport')
+        )
+      )
+    steps:
+      - uses: jooola/backport@main
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          title_template: <%= title %> (<%= base %>)
+          body_template: |
+            Backport <%= mergeCommitSha %> from #<%= number %>.
+
+            BEGIN_COMMIT_OVERRIDE
+            <%= title %>
+            END_COMMIT_OVERRIDE

.github/workflows/command.yml

@@ -1,21 +0,0 @@
-name: Command
-
-on:
-  issue_comment:
-    types: [created]
-
-jobs:
-  dispatch:
-    name: Dispatch
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Dispatch website preview
-        uses: peter-evans/slash-command-dispatch@v3
-        with:
-          token: ${{ secrets.COMMAND_DISPATCH_TOKEN }}
-          issue-type: pull-request
-          dispatch-type: workflow
-          commands: website-preview
-          static-args: |
-            pull-request-number=${{ github.event.issue.number }}

.github/workflows/container.yml

@@ -3,86 +3,101 @@ name: Container
 on:
   push:
     tags: ["[0-9]+.[0-9]+.[0-9]+*"]
-    branches: [main]
+    branches: [main, stable-*]
   pull_request:
-    branches: [main]
+    branches: [main, stable-*]
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
 
 jobs:
-  build:
+  meta:
     runs-on: ubuntu-latest
 
-    env:
+    strategy:
-      REGISTRY: ghcr.io
+      matrix:
-      NAMESPACE: ${{ github.repository_owner }}
+        target: [analyzer, api, legacy, playout, worker]
 
     if: ${{ github.repository_owner == 'libretime' }}
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
-      - uses: docker/setup-buildx-action@v2
+      - name: Update Docker Hub description
-
+        if: github.event_name == 'push'
-      - uses: docker/login-action@v2
+        uses: peter-evans/dockerhub-description@v4
         with:
-          registry: ${{ env.REGISTRY }}
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+          repository: libretime/libretime-${{ matrix.target }}
+          readme-filepath: ./README.md
+
+      - uses: docker/metadata-action@v5
+        id: meta
+        with:
+          bake-target: ${{ matrix.target }}
+          images: |
+            ghcr.io/libretime/libretime-${{ matrix.target }}
+            docker.io/libretime/libretime-${{ matrix.target }}
+          tags: |
+            type=ref,event=branch
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+
+      - name: Upload metadata bake file
+        uses: actions/upload-artifact@v4
+        with:
+          name: meta-${{ matrix.target }}
+          path: ${{ steps.meta.outputs.bake-file }}
+
+  build:
+    needs: [meta]
+    runs-on: ubuntu-latest
+
+    if: ${{ github.repository_owner == 'libretime' }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: docker/setup-buildx-action@v3
+
+      - name: Login ghcr.io
+        if: github.event_name == 'push'
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
+      - name: Login docker.io
+        if: github.event_name == 'push'
+        uses: docker/login-action@v3
+        with:
+          registry: docker.io
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Download all metadata bake files
+        uses: actions/download-artifact@v4
+        with:
+          pattern: meta-*
+
       - name: Guess LIBRETIME_VERSION
         run: |
           make VERSION
           echo "LIBRETIME_VERSION=$(cat VERSION | tr -d [:blank:])" >> $GITHUB_ENV
 
-      - name: Build python-builder
+      - name: Build
-        uses: docker/build-push-action@v3
+        uses: docker/bake-action@v5
         with:
-          context: .
           pull: true
-          target: python-builder
+          push: ${{ github.event_name == 'push' }}
-          cache-from: type=gha,scope=python-builder
+          files: |
-          cache-to: type=gha,scope=python-builder,mode=max
+            docker-bake.json
-
+            meta-analyzer/docker-metadata-action-bake.json
-      - name: Build python-base
+            meta-api/docker-metadata-action-bake.json
-        uses: docker/build-push-action@v3
+            meta-legacy/docker-metadata-action-bake.json
-        with:
+            meta-playout/docker-metadata-action-bake.json
-          context: .
+            meta-worker/docker-metadata-action-bake.json
-          pull: true
+          set: |
-          target: python-base
+            *.cache-from=type=gha,scope=container
-          cache-from: type=gha,scope=python-base
+            *.cache-to=type=gha,scope=container,mode=max
-          cache-to: type=gha,scope=python-base,mode=max
+            *.args.LIBRETIME_VERSION=${{ env.LIBRETIME_VERSION }}
-
-      - name: Build python-base-ffmpeg
-        uses: docker/build-push-action@v3
-        with:
-          context: .
-          pull: true
-          target: python-base-ffmpeg
-          cache-from: type=gha,scope=python-base-ffmpeg
-          cache-to: type=gha,scope=python-base-ffmpeg,mode=max
-
-      - name: Build analyzer
-        uses: ./.github/workflows/actions/build-container
-        with:
-          target: libretime-analyzer
-
-      - name: Build api
-        uses: ./.github/workflows/actions/build-container
-        with:
-          target: libretime-api
-
-      - name: Build playout
-        uses: ./.github/workflows/actions/build-container
-        with:
-          target: libretime-playout
-
-      - name: Build worker
-        uses: ./.github/workflows/actions/build-container
-        with:
-          target: libretime-worker
-
-      - name: Build legacy
-        uses: ./.github/workflows/actions/build-container
-        with:
-          target: libretime-legacy

.github/workflows/dev-tools.yml

@@ -15,16 +15,12 @@ jobs:
     strategy:
       matrix:
         include:
-          - distribution: ubuntu
-            release: bionic
           - distribution: ubuntu
             release: focal
-          - distribution: ubuntu
-            release: jammy
-          - distribution: debian
-            release: buster
           - distribution: debian
             release: bullseye
+          - distribution: ubuntu
+            release: jammy
           - distribution: debian
             release: bookworm
 
@@ -34,10 +30,10 @@ jobs:
 
     steps:
       - name: Checkout
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Login to the Container registry
-        uses: docker/login-action@v2
+        uses: docker/login-action@v3
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
@@ -59,7 +55,7 @@ jobs:
           COPY packages.list packages.list
           EOF
 
-          [[ "${{ matrix.release }}" =~ "bionic|focal" ]] && \
+          [[ "${{ matrix.release }}" == "focal" ]] && \
           cat <<EOF >> Dockerfile
           RUN DEBIAN_FRONTEND=noninteractive apt-get --quiet update && \
               DEBIAN_FRONTEND=noninteractive apt-get --quiet install -y software-properties-common && \
@@ -82,7 +78,7 @@ jobs:
           EOF
 
       - name: Build and push
-        uses: docker/build-push-action@v3
+        uses: docker/build-push-action@v6
         with:
           context: .
           push: ${{ github.repository_owner == 'libretime' }}

.github/workflows/docs.yml

@@ -2,20 +2,21 @@ name: Docs
 
 on:
   push:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/vale/**
       - .github/workflows/docs.yml
       - docs/**
-      - website/**
 
   pull_request:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/vale/**
       - .github/workflows/docs.yml
       - docs/**
-      - website/**
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
 
 jobs:
   lint:
@@ -23,9 +24,9 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: |
             /usr/local/bin/vale*
@@ -42,6 +43,7 @@ jobs:
             errata-ai/vale \
             vale_{version}_Linux_64-bit.tar.gz --extract vale \
             /usr/local/bin/vale \
+            --version v2.21.3 \
             --version-file '{destination}.version'
 
       - name: Add annotations matchers
@@ -51,5 +53,26 @@ jobs:
       - name: Run Vale
         run: |
           vale sync
-          vale --output line docs website/src/pages || true
+          vale --output line docs || true
           vale --output line --minAlertLevel=error docs/releases
+
+  sync:
+    name: Sync
+    if: >
+      github.repository_owner == 'libretime' &&
+      github.event_name == 'push'
+
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - uses: actions/checkout@v4
+        with:
+          repository: libretime/website
+          path: website
+          ssh-key: "${{ secrets.WEBSITE_DEPLOY_KEY }}"
+
+      - name: Sync docs changes
+        run: tools/ci-sync-docs.sh ${{ github.event.before }}..${{ github.sha }}

.github/workflows/housekeeping.yml

@@ -10,6 +10,9 @@ on:
         required: true
         default: "5"
 
+permissions:
+  issues: write
+
 jobs:
   find_closed_references:
     if: github.repository_owner == 'libretime'
@@ -17,9 +20,9 @@ jobs:
 
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
-      - uses: actions/setup-node@v3
+      - uses: actions/setup-node@v4
         with:
           node-version: "16"
 
@@ -27,7 +30,7 @@ jobs:
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
           issueLimit: ${{ github.event.inputs.issueLimit || '5' }}
-          ignore: .git,/docs/releases/*,/website/versioned*,CHANGELOG.md
+          ignore: .git,/docs/releases/*,CHANGELOG.md
 
   find_broken_links:
     if: github.repository_owner == 'libretime'
@@ -35,9 +38,9 @@ jobs:
 
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: .lycheecache
           key: housekeeping-find-broken-links-${{ github.sha }}
@@ -45,23 +48,59 @@ jobs:
 
       - name: Check Links
         id: lychee
-        uses: lycheeverse/lychee-action@v1.5.1
+        uses: lycheeverse/lychee-action@v2.1.0
         with:
           args: >-
             '**/*.md'
-            --exclude-path 'website/versioned_docs'
             --require-https
             --exclude-all-private
-            --exclude-mail
             --exclude 'example\.(com|org)'
             --exclude '\$server_name\$request_uri'
             --exclude '%7Bvars.version%7D'
             --exclude 'https://dir.xiph.org/cgi-bin/yp-cgi'
             --exclude 'https://radio.indymedia.org/cgi-bin/yp-cgi'
             --exclude 'https://www.ascap.com'
+            --exclude 'https://www.youtube-nocookie.com'
             --exclude 'github\.com/libretime/libretime/(issues|pulls)'
+            --exclude 'https://packages.ubuntu.com/bionic/php7.2'
+            --exclude 'https://packages.ubuntu.com/bionic/python3'
             --cache
             --max-cache-age 2d
           fail: true
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+  find_stale_issues:
+    if: github.repository_owner == 'libretime'
+    name: Find stale issues
+
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/stale@v9
+        with:
+          stale-issue-message: >
+            This issue has been automatically marked as stale because it has not had
+            activity in the last 5 months. It will be closed if no activity occurs in
+            the next month.
+
+            Please chat to us on the [forum](https://discourse.libretime.org/) or
+            ask for help on [#libretime:matrix.org](https://matrix.to/#/#libretime:matrix.org)
+            if you have any questions or need further support with getting this issue resolved.
+
+            You may also label an issue as *pinned* if you would like to make sure
+            that it does not get closed by this bot.
+          close-issue-message: >
+            This issue has been automatically closed after is was marked as stale and
+            did not receive any further inputs.
+
+            Feel free to let us know on the [forum](https://discourse.libretime.org/) or
+            ask for help on [#libretime:matrix.org](https://matrix.to/#/#libretime:matrix.org)
+            if you feel this issue should not have been closed.
+
+            Thank you for your contributions.
+          days-before-issue-stale: 150
+          days-before-issue-close: 30
+          stale-issue-label: "status: stalled"
+          exempt-issue-labels: "status: pinned,status: maybe later,security,is: feature-request"
+          exempt-issue-assignees: true
+          exempt-issue-milestones: true

.github/workflows/legacy.yml

@@ -1,20 +1,24 @@
 name: Legacy
 
 on:
+  workflow_dispatch:
   push:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/legacy.yml
       - api/**
       - legacy/**
 
   pull_request:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/legacy.yml
       - api/**
       - legacy/**
 
+  schedule:
+    - cron: 0 1 * * 1
+
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
@@ -26,12 +30,10 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          - php-version: "7.2" # Bionic
+          - php-version: "7.4" # Focal, Bullseye
-          - php-version: "7.3" # Buster
-          - php-version: "7.4" # Bullseye, Focal
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: shivammathur/setup-php@v2
         with:
           php-version: ${{ matrix.php-version }}
@@ -46,14 +48,12 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          - php-version: "7.2" # Bionic
+          - php-version: "7.4" # Focal, Bullseye
-          - php-version: "7.3" # Buster
-          - php-version: "7.4" # Bullseye, Focal
 
     env:
       ENVIRONMENT: testing
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Setup PostgreSQL
         run: |
@@ -62,6 +62,7 @@ jobs:
           sudo -u postgres psql -c 'CREATE DATABASE libretime;'
           sudo -u postgres psql -c "CREATE USER libretime WITH PASSWORD 'libretime';"
           sudo -u postgres psql -c 'GRANT CONNECT ON DATABASE libretime TO libretime;'
+          sudo -u postgres psql -c 'ALTER DATABASE libretime OWNER TO libretime;'
           sudo -u postgres psql -c 'ALTER USER libretime CREATEDB;'
 
       - name: Setup PHP
@@ -72,9 +73,9 @@ jobs:
       - name: Get Composer Cache Directory
         id: composer-cache
         run: |
-          echo "::set-output name=dir::$(composer config cache-files-dir)"
+          echo "dir=$(composer config cache-files-dir)" >> $GITHUB_OUTPUT
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: ${{ steps.composer-cache.outputs.dir }}
           key: ${{ runner.os }}-composer-${{ hashFiles('**/composer.lock') }}
@@ -84,3 +85,36 @@ jobs:
       - name: Run tests
         run: make test
         working-directory: legacy
+
+  locale:
+    runs-on: ubuntu-latest
+
+    if: >
+      github.repository_owner == 'libretime' && (
+        github.event_name == 'schedule' ||
+        github.event_name == 'workflow_dispatch'
+      )
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          token: ${{ secrets.LIBRETIME_BOT_TOKEN }}
+
+      - name: Install dependencies
+        run: |
+          DEBIAN_FRONTEND=noninteractive sudo apt-get update
+          DEBIAN_FRONTEND=noninteractive sudo apt-get install -y gettext
+
+      - name: Update locales
+        run: |
+          git config --global user.name "libretime-bot"
+          git config --global user.email "libretime-bot@users.noreply.github.com"
+
+          git pull
+
+          make -C legacy/locale update
+
+          git add legacy/locale
+          git diff-index --quiet HEAD -- legacy/locale || {
+            git commit --message "chore(legacy): update locales"
+            git push
+          }

.github/workflows/playout.yml

@@ -1,23 +1,29 @@
 name: Playout
 
 on:
+  workflow_dispatch:
   push:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/playout.yml
       - playout/**
       - api-client/**
       - shared/**
+      - tools/python*
 
   pull_request:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/playout.yml
       - playout/**
       - api-client/**
       - shared/**
+      - tools/python*
+
+  schedule:
+    - cron: 0 1 * * 1
 
 jobs:
   python:

.github/workflows/pr.yml

@@ -12,7 +12,7 @@ jobs:
     name: Validate PR title
     runs-on: ubuntu-latest
     steps:
-      - uses: amannn/action-semantic-pull-request@v4.6.0
+      - uses: amannn/action-semantic-pull-request@v5.5.3
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:
@@ -27,6 +27,8 @@ jobs:
             ui
             worker
             deps
+            main
+            stable
           subjectPattern: ^(?![A-Z]).+$
           subjectPatternError: |
             The subject "{subject}" found in the pull request title "{title}"

.github/workflows/project.yml

@@ -1,11 +1,12 @@
 name: Project
 
 on:
+  workflow_dispatch:
   push:
-    branches: [main]
+    branches: [main, stable-*]
   pull_request:
     types: [opened, reopened, synchronize, edited]
-    branches: [main]
+    branches: [main, stable-*]
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -14,67 +15,26 @@ jobs:
   pre-commit:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v4
+
+      - uses: actions/setup-python@v5
         with:
           python-version: "3.x"
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: ~/.cache/pip
           key: ${{ runner.os }}-project-pre-commit-pip-${{ hashFiles('.pre-commit-config.yaml') }}
           restore-keys: |
             ${{ runner.os }}-project-pre-commit-pip
 
-      - uses: pre-commit/action@v3.0.0
+      - uses: pre-commit/action@v3.0.1
-
-  check-shell:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v4
-        with:
-          python-version: "3.x"
-
-      - uses: actions/cache@v3
-        with:
-          path: ~/.cache/pip
-          key: ${{ runner.os }}-project-check-shell-pip
-          restore-keys: |
-            ${{ runner.os }}-project-check-shell-pip
-
-      - uses: actions/cache@v3
-        with:
-          path: |
-            /usr/local/bin/shellcheck*
-            /usr/local/bin/shfmt*
-          key: ${{ runner.os }}-project-check-shell-tools
-          restore-keys: |
-            ${{ runner.os }}-project-check-shell-tools
-
-      - run: |
-          python -m venv venv && source venv/bin/activate
-          pip install gh-release-install
-
-          sudo venv/bin/gh-release-install \
-            koalaman/shellcheck \
-            shellcheck-{tag}.linux.x86_64.tar.xz --extract shellcheck-{tag}/shellcheck \
-            /usr/local/bin/shellcheck \
-            --version-file '{destination}.version'
-
-          sudo venv/bin/gh-release-install \
-            mvdan/sh \
-            shfmt_{tag}_linux_amd64 \
-            /usr/local/bin/shfmt \
-            --version-file '{destination}.version'
-
-      - run: SEVERITY=warning make shell-check
 
   test-tools:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: "3.x"
       - run: make all

.github/workflows/release-please.yml

@@ -0,0 +1,21 @@
+name: Release-Please
+
+on:
+  push:
+    branches:
+      - main
+      - stable
+
+jobs:
+  release-please:
+    # Do not run on forks.
+    if: github.repository == 'libretime/libretime'
+
+    runs-on: ubuntu-latest
+    steps:
+      - uses: google-github-actions/release-please-action@v4
+        with:
+          token: ${{ secrets.LIBRETIME_BOT_TOKEN }}
+          config-file: .github/release-please-config.json
+          manifest-file: .github/release-please-manifest.json
+          target-branch: ${{ github.ref_name }}

.github/workflows/release.yml

@@ -9,7 +9,7 @@ jobs:
   release:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: shivammathur/setup-php@v2
         with:
           php-version: 7.4
@@ -22,10 +22,9 @@ jobs:
       - name: Build tarball
         run: make tarball
 
-      - name: Create Release
+      - name: Upload tarball
-        uses: softprops/action-gh-release@v1
+        uses: softprops/action-gh-release@v2
         with:
-          body_path: docs/releases/${{ github.ref_name }}.md
+          files: |
-          draft: true
+            libretime-*.tar.gz
-          prerelease: true
+            sha256sums.txt
-          files: libretime-*.tar.gz

.github/workflows/shared.yml

@@ -1,19 +1,25 @@
 name: Shared
 
 on:
+  workflow_dispatch:
   push:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/shared.yml
       - shared/**
+      - tools/python*
 
   pull_request:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/shared.yml
       - shared/**
+      - tools/python*
+
+  schedule:
+    - cron: 0 1 * * 1
 
 jobs:
   python:

.github/workflows/website-preview-command.yml

@@ -1,118 +0,0 @@
-name: Website Preview
-
-on:
-  workflow_dispatch:
-    inputs:
-      pull-request-number:
-        description: "Pull request number to preview"
-        required: true
-        type: string
-
-  pull_request_target:
-    types: [closed]
-    branches: [main]
-    paths:
-      - website/**
-      - docs/**
-
-env:
-  # PREVIEW_DEPLOY_KEY is present in the secrets
-  PREVIEW_EXTERNAL_REPOSITORY: libretime/libretime.github.io
-  PREVIEW_EXTERNAL_REPOSITORY_BRANCH: gh-pages
-  PREVIEW_URL: https://libretime.github.io
-  PREVIEW_BASE_URL: /
-
-jobs:
-  deploy:
-    if: github.event_name == 'workflow_dispatch'
-    name: Deploy
-    runs-on: ubuntu-latest
-
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Checkout pull request
-        run: hub pr checkout ${{ github.event.inputs.pull-request-number }}
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-
-      - name: Setup Node
-        uses: actions/setup-node@v3
-        with:
-          node-version: "16"
-          cache: yarn
-          cache-dependency-path: website/yarn.lock
-
-      - name: Install
-        working-directory: website
-        run: yarn install --frozen-lockfile
-
-      - name: Build
-        working-directory: website
-        run: yarn build
-        env:
-          URL: ${{ env.PREVIEW_URL }}
-          BASE_URL: ${{ env.PREVIEW_BASE_URL }}pr-${{ github.event.inputs.pull-request-number }}/
-
-      - name: Deploy
-        uses: peaceiris/actions-gh-pages@v3
-        with:
-          external_repository: ${{ env.PREVIEW_EXTERNAL_REPOSITORY }}
-          deploy_key: ${{ secrets.PREVIEW_DEPLOY_KEY }}
-          publish_dir: website/build
-          destination_dir: pr-${{ github.event.inputs.pull-request-number }}
-          full_commit_message: "deploy pr-${{ github.event.inputs.pull-request-number }}"
-          keep_files: true
-
-      - name: Find deployment comment
-        uses: peter-evans/find-comment@v2
-        id: find-comment
-        with:
-          issue-number: ${{ github.event.inputs.pull-request-number }}
-          comment-author: github-actions[bot]
-          body-includes: Website preview deployment
-
-      - name: Notify deployment succeeded
-        if: ${{ success() }}
-        uses: peter-evans/create-or-update-comment@v2
-        with:
-          issue-number: ${{ github.event.inputs.pull-request-number }}
-          comment-id: ${{ steps.find-comment.outputs.comment-id }}
-          edit-mode: replace
-          body: |
-            **:rocket: Website preview deployment succeeded!**
-
-            Website preview: ${{ env.PREVIEW_URL }}${{ env.PREVIEW_BASE_URL }}pr-${{ github.event.inputs.pull-request-number }}/
-            New docs preview: ${{ env.PREVIEW_URL }}${{ env.PREVIEW_BASE_URL }}pr-${{ github.event.inputs.pull-request-number }}/docs/next/
-            Workflow: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
-
-      - name: Notify deployment failed
-        if: ${{ failure() }}
-        uses: peter-evans/create-or-update-comment@v2
-        with:
-          issue-number: ${{ github.event.inputs.pull-request-number }}
-          comment-id: ${{ steps.find-comment.outputs.comment-id }}
-          edit-mode: replace
-          body: |
-            **:boom: Website preview deployment failed!**
-
-            Workflow: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
-
-  clean:
-    if: github.event_name == 'pull_request_target' && github.event.action == 'closed'
-    name: Clean
-    runs-on: ubuntu-latest
-
-    steps:
-      - uses: actions/checkout@v3
-        with:
-          repository: ${{ env.PREVIEW_EXTERNAL_REPOSITORY }}
-          ref: ${{ env.PREVIEW_EXTERNAL_REPOSITORY_BRANCH }}
-          ssh-key: ${{ secrets.PREVIEW_DEPLOY_KEY }}
-
-      - name: Remove files
-        run: rm -fR pr-${{ github.event.pull_request.number }}
-
-      - uses: endbug/add-and-commit@v9
-        with:
-          message: "clean pr-${{ github.event.pull_request.number }}"

.github/workflows/website.yml

@@ -1,53 +0,0 @@
-name: Website
-
-on:
-  push:
-    branches: [main]
-    paths:
-      - .github/workflows/website.yml
-      - docs/**
-      - website/**
-
-  pull_request:
-    branches: [main]
-    paths:
-      - .github/workflows/website.yml
-      - docs/**
-      - website/**
-
-jobs:
-  deploy:
-    name: Deploy
-    runs-on: ubuntu-latest
-
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Setup Node
-        uses: actions/setup-node@v3
-        with:
-          node-version: "16"
-          cache: yarn
-          cache-dependency-path: ./website/yarn.lock
-
-      - uses: actions/cache@v3
-        with:
-          path: website/.docusaurus
-          key: docusaurus-main-${{ github.sha }}
-          restore-keys: |
-            docusaurus-main-
-
-      - name: Install
-        working-directory: website
-        run: yarn install --frozen-lockfile
-
-      - name: Build
-        working-directory: website
-        run: yarn build
-
-      - name: Deploy
-        if: github.ref == 'refs/heads/main'
-        uses: peaceiris/actions-gh-pages@v3
-        with:
-          github_token: ${{ secrets.GITHUB_TOKEN }}
-          publish_dir: website/build

.github/workflows/worker.yml

@@ -1,19 +1,25 @@
 name: Worker
 
 on:
+  workflow_dispatch:
   push:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/worker.yml
       - worker/**
+      - tools/python*
 
   pull_request:
-    branches: [main]
+    branches: [main, stable-*]
     paths:
       - .github/workflows/_python.yml
       - .github/workflows/worker.yml
       - worker/**
+      - tools/python*
+
+  schedule:
+    - cron: 0 1 * * 1
 
 jobs:
   python:

.gitignore

@@ -8,6 +8,13 @@
 *~
 VERSION
 
+/dev/certs/*
+/dev/playout/*
+
+/website/
+
+!.gitkeep
+
 ## Github Python .gitignore
 ## See https://github.com/github/gitignore/blob/master/Python.gitignore
 ################################################################################

.pre-commit-config.yaml

@@ -3,7 +3,7 @@
 # See https://pre-commit.com/hooks.html for more hooks
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.3.0
+    rev: v5.0.0
     hooks:
       - id: check-added-large-files
       - id: check-case-conflict
@@ -26,48 +26,63 @@ repos:
         exclude: \.ambr$
 
       - id: name-tests-test
-        # TODO: Remove once the django api uses pytest
+        exclude: ^api
-        exclude: ^(api.*)$
 
   - repo: https://github.com/pre-commit/mirrors-prettier
-    rev: v2.7.1
+    rev: v3.1.0
     hooks:
       - id: prettier
         files: \.(md|mdx|yml|yaml|js|jsx|ts|tsx|json|css)$
-        exclude: ^legacy/public(?!/js/airtime)
+        exclude: ^(legacy/public(?!/js/airtime)|CHANGELOG.md$|.github/release-please-manifest.json)
 
   - repo: https://github.com/asottile/pyupgrade
-    rev: v2.38.2
+    rev: v3.19.0
     hooks:
       - id: pyupgrade
-        args: [--py3-plus, --py36-plus]
+        args: [--py38-plus]
 
-  - repo: https://github.com/psf/black
+  - repo: https://github.com/adamchainz/django-upgrade
-    rev: 22.8.0
+    rev: 1.22.2
+    hooks:
+      - id: django-upgrade
+        args: [--target-version, "4.2"]
+
+  - repo: https://github.com/pycqa/isort
+    rev: 5.13.2
+    hooks:
+      - id: isort
+        args: [--resolve-all-configs]
+
+  - repo: https://github.com/psf/black-pre-commit-mirror
+    rev: 24.10.0
     hooks:
       - id: black
 
-  # - repo: https://github.com/pycqa/isort
-  #   rev: 5.9.3
-  #   hooks:
-  #     - id: isort
-  #       args: ["--profile", "black", "--filter-files"]
-
   - repo: https://github.com/codespell-project/codespell
-    rev: v2.2.1
+    rev: v2.3.0
     hooks:
       - id: codespell
-        args:
+        args: [--ignore-words=.codespellignore]
-          - --ignore-words=.codespellignore
-          - --builtin=clear,rare,informal
         exclude: (^api/schema.yml|^legacy.*|yarn\.lock)$
 
   - repo: local
     hooks:
+      - id: shfmt
+        name: shfmt
+        language: docker_image
+        entry: mvdan/shfmt -i 2 -ci -sr -kp -w
+        types: [shell]
+
+      - id: shellcheck
+        name: shellcheck
+        language: docker_image
+        entry: koalaman/shellcheck --color=always --severity=warning
+        types: [shell]
+
       - id: requirements.txt
         name: requirements.txt
         description: Generate requirements.txt
-        entry: tools/extract_requirements.py dev
+        entry: tools/extract_requirements.py dev sentry
         pass_filenames: false
         language: script
         files: setup.py$
@@ -80,6 +95,14 @@ repos:
         language: script
         files: ^installer/config.yml$
 
+      - id: legacy-migrations-version
+        name: legacy-migrations-version
+        description: Ensure valid schema version for migrations
+        entry: tools/legacy-migrations-version.sh
+        pass_filenames: false
+        language: script
+        files: ^api/libretime_api/legacy/migrations
+
       - id: legacy-assets-checksum-update
         name: legacy-assets-checksum-update
         description: Update legacy assets checksum
@@ -87,3 +110,11 @@ repos:
         pass_filenames: false
         language: script
         files: ^legacy
+
+      - id: api-schema-update
+        name: api-schema-update
+        description: Ensure API schema is up to date
+        entry: make -C api schema
+        pass_filenames: false
+        language: system
+        files: ^api

.vale.ini

@@ -2,17 +2,16 @@ StylesPath = .github/vale/styles
 MinAlertLevel = warning
 
 Packages = \
-    https://github.com/errata-ai/Google/releases/latest/download/Google.zip, \
     https://github.com/errata-ai/Microsoft/releases/latest/download/Microsoft.zip
 
 Vocab = Docs
 
 [*.md]
-BasedOnStyles = Vale, Google, Microsoft, LibreTime
+BasedOnStyles = Vale, Microsoft, LibreTime
 
 # Exclude emoji shortcodes `:tada:`
 BlockIgnores = (:[a-z-_]+:)
 
-Google.Units = False
 Microsoft.GeneralURL = False
+Microsoft.RangeFormat = False
 Vale.Spelling = False

CHANGELOG.md

@@ -1,4 +1,437 @@
-<a name="3.0.0-beta.1"></a>
+# Changelog
+
+## [4.2.0](https://github.com/libretime/libretime/compare/4.1.0...4.2.0) (2024-06-22)
+
+
+### Features
+
+* **legacy:** add current date macro to string block criteria ([#3013](https://github.com/libretime/libretime/issues/3013)) ([451652b](https://github.com/libretime/libretime/commit/451652bc4002b142ab9cf33ae517451c4966134f))
+* **legacy:** add filename block criteria ([#3015](https://github.com/libretime/libretime/issues/3015)) ([4642b6c](https://github.com/libretime/libretime/commit/4642b6c08ef813ab5dc7354f73141239f5c145e0))
+
+
+### Bug Fixes
+
+* pin pip version to &lt;24.1 to allow installing pytz (celery) ([#3043](https://github.com/libretime/libretime/issues/3043)) ([646bc81](https://github.com/libretime/libretime/commit/646bc817246a1e3e0d8107c2b69d726681c643b6))
+* playlist allocates inaccurate time to smartblocks ([#3026](https://github.com/libretime/libretime/issues/3026)) ([2b43e51](https://github.com/libretime/libretime/commit/2b43e51ed140bf307e491f0fcb7b84f95709d604))
+
+
+### Performance Improvements
+
+* optimize the api image health check ([#3038](https://github.com/libretime/libretime/issues/3038)) ([d99d6e1](https://github.com/libretime/libretime/commit/d99d6e1a68f20b3f4255296cd22ac80a90adc020))
+* optimize the rabbitmq health check ([#3037](https://github.com/libretime/libretime/issues/3037)) ([9684214](https://github.com/libretime/libretime/commit/96842144257855df86085b052ed8ff87562bc049))
+
+## [4.1.0](https://github.com/libretime/libretime/compare/4.0.0...4.1.0) (2024-05-05)
+
+
+### Features
+
+* **api:** implement file deletion ([#2960](https://github.com/libretime/libretime/issues/2960)) ([9757b1b](https://github.com/libretime/libretime/commit/9757b1b78c98a33f233163c77eb1b2ad6e0f0efe))
+* build schedule events exclusively in playout ([#2946](https://github.com/libretime/libretime/issues/2946)) ([40b4fc7](https://github.com/libretime/libretime/commit/40b4fc7f66004ee3bcb61c9961ec2c48bbcbc6cb))
+* **legacy:** add aac/opus support to dashboard player ([#2881](https://github.com/libretime/libretime/issues/2881)) ([95283ef](https://github.com/libretime/libretime/commit/95283efc1f9a63376a99184ef69b699beba45802))
+* **legacy:** disable public radio page and redirect to login ([#2903](https://github.com/libretime/libretime/issues/2903)) ([170d095](https://github.com/libretime/libretime/commit/170d09545e4fcfeeb95f9fc5c355329764501854))
+* **legacy:** trim overbooked shows after autoloading a playlist ([#2897](https://github.com/libretime/libretime/issues/2897)) ([a95ce3d](https://github.com/libretime/libretime/commit/a95ce3d2296bb864b379dcce14090bd821c1dfc9))
+* **legacy:** visual cue point editor ([#2947](https://github.com/libretime/libretime/issues/2947)) ([da02e74](https://github.com/libretime/libretime/commit/da02e74f2115cb76a6435fab5ab2667a8c622b98))
+* start celery worker programmatically ([#2988](https://github.com/libretime/libretime/issues/2988)) ([9c548b3](https://github.com/libretime/libretime/commit/9c548b365ec114c6789d2a69e66cc721da6ae100))
+
+
+### Bug Fixes
+
+* **analyzer:** backslash non utf-8 data when probing replaygain ([#2931](https://github.com/libretime/libretime/issues/2931)) ([29f73e0](https://github.com/libretime/libretime/commit/29f73e0dcb1fd668a79a2ffedc33e16172277376)), closes [#2910](https://github.com/libretime/libretime/issues/2910)
+* apply replay gain preferences on scheduled files ([#2945](https://github.com/libretime/libretime/issues/2945)) ([35d0dec](https://github.com/libretime/libretime/commit/35d0dec4a887cdaea2d73dc9bee60eb6624a2aca))
+* **deps:** update dependency friendsofphp/php-cs-fixer to &lt;3.49.1 ([#2899](https://github.com/libretime/libretime/issues/2899)) ([3e05748](https://github.com/libretime/libretime/commit/3e05748d2d1180b8dad55b6f997e6aa7117735f1))
+* **deps:** update dependency friendsofphp/php-cs-fixer to &lt;3.51.1 ([#2963](https://github.com/libretime/libretime/issues/2963)) ([22c303c](https://github.com/libretime/libretime/commit/22c303cfffdc777177bd74273e2c24da58cf1682))
+* **deps:** update dependency friendsofphp/php-cs-fixer to &lt;3.53.1 ([#2972](https://github.com/libretime/libretime/issues/2972)) ([9192aaa](https://github.com/libretime/libretime/commit/9192aaa2bb2dada470e03537493160d9b14a42f4))
+* **deps:** update dependency gunicorn to v22 (security) ([#2993](https://github.com/libretime/libretime/issues/2993)) ([a2cf769](https://github.com/libretime/libretime/commit/a2cf7697a97bbc4faf89fd7bc9ba9ecc235bf873))
+* incorrect docker compose version ([#2975](https://github.com/libretime/libretime/issues/2975)) ([634e6e2](https://github.com/libretime/libretime/commit/634e6e236d908994d586c946bbe28bcba8a357fa))
+* **installer:** setup the worker entrypoint ([#2996](https://github.com/libretime/libretime/issues/2996)) ([71b20ae](https://github.com/libretime/libretime/commit/71b20ae3c974680d814062c5a0bfa51a105dde61))
+* **legacy:** allow deleting file with api token ([#2995](https://github.com/libretime/libretime/issues/2995)) ([86da46e](https://github.com/libretime/libretime/commit/86da46ee3a54676298e30301846be890d1ea93ae))
+* **legacy:** allow updating track types code ([#2955](https://github.com/libretime/libretime/issues/2955)) ([270aa08](https://github.com/libretime/libretime/commit/270aa08ae6c7207de1cc3ea552dabeb018bcfe0d))
+* **legacy:** avoid crash when lot of streams in configuration ([#2915](https://github.com/libretime/libretime/issues/2915)) ([12dd477](https://github.com/libretime/libretime/commit/12dd47731290bf539be7a2a81571f8ada223e9c4))
+* **legacy:** ensure validation is performed on the track type form ([#2985](https://github.com/libretime/libretime/issues/2985)) ([5ad69bf](https://github.com/libretime/libretime/commit/5ad69bf0b76ff2e5065551b6a7d154cb26834605))
+* **legacy:** fix hidden fields in edit file form ([#2932](https://github.com/libretime/libretime/issues/2932)) ([f4b260f](https://github.com/libretime/libretime/commit/f4b260fdf70c0dd1830166d3856239dae5366599))
+* **legacy:** replay_gain_modifier should be a system preference ([#2943](https://github.com/libretime/libretime/issues/2943)) ([37d1a76](https://github.com/libretime/libretime/commit/37d1a7685e37e45734553a0eb4a4da793ca858cb))
+* remove obsolete docker compose version ([#2982](https://github.com/libretime/libretime/issues/2982)) ([fb0584b](https://github.com/libretime/libretime/commit/fb0584b021fd1c966181c7ab3989938cdfe4e642))
+* trigger legacy tasks manager every 5m ([#2987](https://github.com/libretime/libretime/issues/2987)) ([7040d0e](https://github.com/libretime/libretime/commit/7040d0e4bd92911a9072226f49ad59ce575d6ed9))
+* **worker:** ensure celery beat is started ([#3007](https://github.com/libretime/libretime/issues/3007)) ([bfde17e](https://github.com/libretime/libretime/commit/bfde17edf7fcc2bfd55263756e6ec3e455f11740))
+
+## [4.0.0](https://github.com/libretime/libretime/compare/3.2.0...4.0.0) (2024-01-07)
+
+
+### ⚠ BREAKING CHANGES
+
+* The media file serving is now handled by Nginx instead of the API service. The `storage.path` field is now used in the Nginx configuration, so make sure to update the Nginx configuration file if you change it.
+* **installer:** The default listen port for the installer is now `8080`. We recommend that you put a reverse proxy in front of LibreTime.
+* **installer:** The `--update-nginx` flag was removed from the installer. The nginx configuration deployed by the installer will now always be overwritten. Make sure to move your customizations to a reverse proxy configuration.
+* The default system output (`stream.outputs.system[].kind`) changed from `alsa` to `pulseaudio`. Make sure to update your configuration file if you rely on the default system output.
+* The `general.secret_key` configuration field is now required. Make sure to update your configuration file and add a secret key.
+
+### Features
+
+* default system output is now `pulseaudio` ([#2842](https://github.com/libretime/libretime/issues/2842)) ([083ee3f](https://github.com/libretime/libretime/commit/083ee3f1dd74441e288b4d63178ae9cea12ba286)), closes [#2542](https://github.com/libretime/libretime/issues/2542)
+* disable uvicorn worker lifespan ([#2845](https://github.com/libretime/libretime/issues/2845)) ([8743c84](https://github.com/libretime/libretime/commit/8743c84d0f007a5430e9059c197a261e613cc642))
+* **installer:** add the `--storage-path` flag ([#2865](https://github.com/libretime/libretime/issues/2865)) ([5b23852](https://github.com/libretime/libretime/commit/5b23852f8d144f0c7cdeb62831f7b1a27872b40e))
+* **installer:** change default listen port to 8080 ([#2852](https://github.com/libretime/libretime/issues/2852)) ([f72b7f9](https://github.com/libretime/libretime/commit/f72b7f9c9727800a9d77d64c540c12f272bb0ae3))
+* **installer:** remove the `--update-nginx` flag ([#2851](https://github.com/libretime/libretime/issues/2851)) ([35d7eac](https://github.com/libretime/libretime/commit/35d7eace13c2b9667fdb41fec0788118e0c5e63f))
+* **playout:** configure device for alsa and pulseaudio system outputs ([#2654](https://github.com/libretime/libretime/issues/2654)) ([06af18b](https://github.com/libretime/libretime/commit/06af18b84e7dfaad95e3b55dda22ec1ddad27050))
+* rewrite cloud-init config ([#2853](https://github.com/libretime/libretime/issues/2853)) ([8406d52](https://github.com/libretime/libretime/commit/8406d520d7a7bea4060be8a00e360bcf413cb2d5))
+* run python in optimized mode ([#2874](https://github.com/libretime/libretime/issues/2874)) ([3f7fc99](https://github.com/libretime/libretime/commit/3f7fc99b6b343fbc8df319d8130ba8247aea96d8))
+* the `general.secret_key` configuration field is now required ([#2841](https://github.com/libretime/libretime/issues/2841)) ([0d2d1a2](https://github.com/libretime/libretime/commit/0d2d1a26731a2b41ce5e574ed6de9950eaae4153)), closes [#2426](https://github.com/libretime/libretime/issues/2426)
+* use nginx to serve media files ([#2860](https://github.com/libretime/libretime/issues/2860)) ([4603c17](https://github.com/libretime/libretime/commit/4603c1759f29b8a1adb3e83d610ca00e778d76bd))
+
+
+### Bug Fixes
+
+* add parent function name in setValue exception ([#2777](https://github.com/libretime/libretime/issues/2777)) ([c764a5a](https://github.com/libretime/libretime/commit/c764a5a648ac6cf6c1f63cd9be6de9fe07c40988))
+* **api:** ensure non ascii paths are handled by X-Accel-Redirect ([#2861](https://github.com/libretime/libretime/issues/2861)) ([0ce63f3](https://github.com/libretime/libretime/commit/0ce63f3bf0448580170024cdde96ee351ee5c358))
+* **api:** enum schema description ([#2803](https://github.com/libretime/libretime/issues/2803)) ([976b70e](https://github.com/libretime/libretime/commit/976b70ed32a0e774cc0b72b8332372be32799ed1))
+* **api:** let nginx handle the media file content type ([#2862](https://github.com/libretime/libretime/issues/2862)) ([72268ad](https://github.com/libretime/libretime/commit/72268ad9bb1a96b24efda7143b9371d6fd98ca03))
+* **api:** move gunicorn worker config to python file ([#2854](https://github.com/libretime/libretime/issues/2854)) ([43221d9](https://github.com/libretime/libretime/commit/43221d9d7f34ba98a14db9906e350cb494a86b25))
+* **api:** paths with question marks chars are handled by X-Accel-Redirect ([#2875](https://github.com/libretime/libretime/issues/2875)) ([b2c1ceb](https://github.com/libretime/libretime/commit/b2c1ceb89fafc76f18ec650d19ec0ff03e4a20b0))
+* **deps:** update dependency friendsofphp/php-cs-fixer to &lt;3.42.1 (main) ([#2765](https://github.com/libretime/libretime/issues/2765)) ([8ae4dce](https://github.com/libretime/libretime/commit/8ae4dce9e7c013c1f66f1b4d5da4a8c91d3419b7))
+* **deps:** update dependency friendsofphp/php-cs-fixer to &lt;3.43.2 (main) ([#2848](https://github.com/libretime/libretime/issues/2848)) ([62e5f4d](https://github.com/libretime/libretime/commit/62e5f4dfbb76ab1919c4905570cc34274c685cef))
+* **deps:** update dependency friendsofphp/php-cs-fixer to &lt;3.45.1 (main) ([#2855](https://github.com/libretime/libretime/issues/2855)) ([6f84328](https://github.com/libretime/libretime/commit/6f8432838058be6ef1cfa7858f17b8272929896e))
+* **deps:** update dependency friendsofphp/php-cs-fixer to &lt;3.46.1 (main) ([#2868](https://github.com/libretime/libretime/issues/2868)) ([4827dbc](https://github.com/libretime/libretime/commit/4827dbce711262e90238bb3b6c0a35b1ce3d6877))
+* **legacy:** allow uploading opus files ([#2804](https://github.com/libretime/libretime/issues/2804)) ([f252a16](https://github.com/libretime/libretime/commit/f252a16637e113ceb1dd340fb7aad31af9c23ff0))
+* **legacy:** declare previously undeclared variable ([#2793](https://github.com/libretime/libretime/issues/2793)) ([e2cfbf4](https://github.com/libretime/libretime/commit/e2cfbf4c038f28874a206df5805f04f69a40647b))
+* **legacy:** ensure last played criteria works with never played files ([#2840](https://github.com/libretime/libretime/issues/2840)) ([24ee383](https://github.com/libretime/libretime/commit/24ee3830c23f7147f82febe3d3c6743d5ae8d4e6))
+* **playout:** increase file download chunk size to 8192 bytes ([#2863](https://github.com/libretime/libretime/issues/2863)) ([7ed1be1](https://github.com/libretime/libretime/commit/7ed1be1816abef20b9ae59a8c66a9e48a34f37c5))
+* **playout:** remove empty file when the download request failed ([#2864](https://github.com/libretime/libretime/issues/2864)) ([2facbfa](https://github.com/libretime/libretime/commit/2facbfaff23d4df0e7531b82f04f932bb2c4c9a4))
+* **worker:** unbound variable when episode url returns HTTP 404 ([#2844](https://github.com/libretime/libretime/issues/2844)) ([3f39689](https://github.com/libretime/libretime/commit/3f396895e588e62183e01d17927d9bdbea512ee0))
+
+## [3.2.0](https://github.com/libretime/libretime/compare/3.1.0...3.2.0) (2023-10-16)
+
+- [Release note](https://libretime.org/docs/releases/3.2.0/)
+
+### Features
+
+- **legacy:** move session store to database ([#2523](https://github.com/libretime/libretime/issues/2523))
+- **api:** add email configuration
+- add mobile devices stream config field ([#2744](https://github.com/libretime/libretime/issues/2744))
+
+### Bug Fixes
+
+- **playout:** liquidsoap aac output syntax errors
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.20.1 (stable) ([#2602](https://github.com/libretime/libretime/issues/2602))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.21.2 ([#2612](https://github.com/libretime/libretime/issues/2612))
+- libretime process leaks and lsof high cpu usage ([#2615](https://github.com/libretime/libretime/issues/2615))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.22.1 ([#2633](https://github.com/libretime/libretime/issues/2633))
+- libretime process leaks and lsof high cpu usage ([#2615](https://github.com/libretime/libretime/issues/2615))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.23.1 (stable) ([#2656](https://github.com/libretime/libretime/issues/2656))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.26.1 (stable) ([#2678](https://github.com/libretime/libretime/issues/2678))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.26.1 (main) ([#2677](https://github.com/libretime/libretime/issues/2677))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.26.2 ([#2686](https://github.com/libretime/libretime/issues/2686))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.26.2 ([#2687](https://github.com/libretime/libretime/issues/2687))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.27.1 (main) ([#2714](https://github.com/libretime/libretime/issues/2714))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.27.1 (stable) ([#2715](https://github.com/libretime/libretime/issues/2715))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.34.1 ([#2723](https://github.com/libretime/libretime/issues/2723))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.35.2 ([#2738](https://github.com/libretime/libretime/issues/2738))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.35.2 ([#2722](https://github.com/libretime/libretime/issues/2722))
+
+### Documentation
+
+- update chat links to point to matrix ([#2571](https://github.com/libretime/libretime/issues/2571))
+- fix broken link ([#2616](https://github.com/libretime/libretime/issues/2616))
+
+### Tests
+
+- **playout:** check unsupported liquidsoap aac output
+
+## [3.1.0](https://github.com/libretime/libretime/compare/3.0.2...3.1.0) (2023-05-26)
+
+- [Release note](https://libretime.org/docs/releases/3.1.0/)
+
+### Features
+
+- drop Ubuntu Bionic support
+- drop Python 3.6 support
+- drop Debian Buster support
+- drop Liquidsoap 1.1 support
+- drop Liquidsoap 1.3 support
+- drop Python 3.7 support
+- drop cc_stream_setting table
+- delete cc_pref stream preferences rows
+- **legacy:** remove db allowed_cors_origins preference ([#2095](https://github.com/libretime/libretime/issues/2095))
+- configure cue points analysis per track type
+- **playout:** use jinja2 env for template loading
+- **playout:** add jinja2 quote filter for liquidsoap
+- **playout:** use liquidsoap interactive variables
+- **playout:** remove unused liquidsoap outputs connection status
+- **playout:** remove unused liquidsoap restart function
+- **playout:** remove unused liquidsoap output namespace
+- replace loguru with logging
+- **playout:** use jinja to configure liquidsoap outputs
+- **playout:** enable vorbis metadata per icecast output
+- **playout:** use shared app for cli commands
+- **installer:** configure timezone using timedatectl ([#2418](https://github.com/libretime/libretime/issues/2418))
+- **playout:** don't serialize message twice
+- add python packages version
+- add sentry sdk
+- use secret_key config field instead of api_key ([#2444](https://github.com/libretime/libretime/issues/2444))
+- **api-client:** remove unused api v1 calls
+- **api-client:** rewrite api-client v1 using abstract client
+- **playout:** move liquidsoap auth to notify cli
+- **playout:** replace schedule event dicts with objects
+- **api:** add cors headers middleware ([#2479](https://github.com/libretime/libretime/issues/2479))
+- **playout:** replace thread timeout with socket timeout
+- remove dev files from tarball
+- include tarball checksums in releases
+- set icecast mount default charset to UTF-8
+- **playout:** allow harbor ssl configuration
+- **api:** install gunicorn/uvicorn from pip
+- install inside a python3 venv
+
+### Bug Fixes
+
+- **deps:** update dependency adbario/php-dot-notation to v3 ([#2226](https://github.com/libretime/libretime/issues/2226))
+- **deps:** update dependency league/uri to v6.7.2
+- **legacy:** set platform requirements to php ^7.4
+- **playout:** remove outdated liquidsoap code
+- **playout:** add types
+- **api:** allow single digit version for legacy schema
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.12.1
+- remove systemd ProtectHome feature ([#2243](https://github.com/libretime/libretime/issues/2243))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.13.1 ([#2249](https://github.com/libretime/libretime/issues/2249))
+- **worker:** replace deprecated cgi.parse_header
+- **installer:** install missing sudo
+- **installer:** set home and login when running as postgres
+- **legacy:** add log entry on task run ([#2316](https://github.com/libretime/libretime/issues/2316))
+- **legacy:** log errors on connect check failure ([#2317](https://github.com/libretime/libretime/issues/2317))
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.13.2
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.13.3
+- **legacy:** advanced search by track type id
+- **legacy:** move forked deps to the libretime namespace
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.14.4
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.14.5
+- **legacy:** ensure options is a dict during json encoding
+- **legacy:** don't use dict assignment on object ([#2384](https://github.com/libretime/libretime/issues/2384))
+- **playout:** quote escape strings in liquidsoap entrypoint
+- **legacy:** do not delete audio file when removing artwork ([#2395](https://github.com/libretime/libretime/issues/2395))
+- **playout:** use explicit ids for liquidsoap components
+- **playout:** skip the identified queue instead of the current
+- **playout:** use the same number of schedule queues
+- **legacy:** on air light fails when no shows are scheduled
+- **playout:** flush liquidsoap response before sending new
+- **playout:** use package loader for liquidsoap templates
+- **playout:** %else is not defined
+- **playout:** when shows ends, next shows starts without fade-in/fade-out ([#2412](https://github.com/libretime/libretime/issues/2412))
+- **playout:** legacy pushes non validated data
+- **playout:** explicit ogg vorbis icecast encoder
+- **playout:** prevent unbound variables
+- **playout:** use int for liquidsoap queues map
+- **shared:** return type confusion
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.15.2
+- **api:** explicit FileImportStatusEnum in schema
+- pin postgresql version in docker-compose
+- pin rabbitmq version in docker-compose
+- allow overriding docker-compose predefined environment
+- move docker specific setup to dockerfile
+- **api:** cast string value to int enum ([#2461](https://github.com/libretime/libretime/issues/2461))
+- **playout:** quote incompatible <py3.9 type hints
+- **installer:** bump setuptools to ~=67.3 ([#2387](https://github.com/libretime/libretime/issues/2387))
+- **playout:** use new api-client v1
+- **playout:** catch oserror in liquidsoap client
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.16.1 (main) ([#2490](https://github.com/libretime/libretime/issues/2490))
+- **api:** require django >=4.2.0,<4.3
+- **api:** upgrade psycopg to v3.1
+- **playout:** remove unused ecasound package ([#2496](https://github.com/libretime/libretime/issues/2496))
+- **installer:** ignore whitespace during diff
+- **legacy:** don't print track_type id in show builder table ([#2510](https://github.com/libretime/libretime/issues/2510))
+- **legacy:** remove composer superuser warning ([#2515](https://github.com/libretime/libretime/issues/2515))
+- **legacy:** keep datatable settings between views ([#2519](https://github.com/libretime/libretime/issues/2519))
+- **api:** upgrade django code (pre-commit)
+- **analyzer:** remove unused python3 package
+- **deps:** update dependency friendsofphp/php-cs-fixer to <3.17.1 (main) ([#2556](https://github.com/libretime/libretime/issues/2556))
+
+### Documentation
+
+- **playout:** add simple inputs pipeline schema ([#2240](https://github.com/libretime/libretime/issues/2240))
+- add DOCKER_BUILDKIT env variable for docker-compose v1 ([#2270](https://github.com/libretime/libretime/issues/2270))
+- no need to update release note path
+- adapt c4 to our workflows
+- stop providing maintenance releases for old distributions
+- add pulseaudio output in containers tutorial ([#2166](https://github.com/libretime/libretime/issues/2166))
+- remove warning about docker install ([#2411](https://github.com/libretime/libretime/issues/2411))
+- docker-compose env variables setup
+- add instructions for the sentry setup ([#2441](https://github.com/libretime/libretime/issues/2441))
+- upgrade by migrating to a new server
+- fix database backup and restore commands
+- move contributing to docs/contribute
+- split developer and contributor manual
+- extract dev workflows from contributing docs
+- add some history notes
+- move release docs in the release section
+- fix broken links
+- ignore range format during docs linting
+- only use microsoft styling guide
+- move configuration documentation
+- rename setup to install
+- split install guide per install method
+- docker config template install with envsubst ([#2517](https://github.com/libretime/libretime/issues/2517))
+- improve reverse proxy docs
+- improve install guides
+- add certbot setup guide
+- ensure example values are replaced
+- fix broken link ([#2532](https://github.com/libretime/libretime/issues/2532))
+- add note about unused packages
+- improve airtime migration guide ([#2564](https://github.com/libretime/libretime/issues/2564))
+- split airtime migration into more steps ([#2565](https://github.com/libretime/libretime/issues/2565))
+- remove setup without reverse proxy
+- fix icecast certificates bundle command
+- install using a reverse proxy by default
+- be consistent with example domain ([#2568](https://github.com/libretime/libretime/issues/2568))
+- add 3.1.x distribution releases support
+
+### Tests
+
+- liquidsoap package from ppa is version 1.4.2 ([#2223](https://github.com/libretime/libretime/issues/2223))
+- **playout:** refresh snapshots after major upgrade
+- re-enable pylint logging-fstring-interpolation
+- **playout:** more entrypoint config test cases
+- **playout:** generated liquidsoap script syntax
+- **playout:** silence existing broad-exception-caught errors
+- **playout:** allow pylint failure
+- **playout:** check untyped defs with mypy
+- **api:** fix linting errors
+- **shared:** fix linting errors
+- **api:** fix linting errors
+- **shared:** fix linting errors
+- **playout:** class creation
+- **api-client:** allow linters failure
+- **playout:** move liq_conn fixture to conftest
+- **playout:** liquidsoap wait for version
+- **api:** add django-upgrade pre-commit hook
+
+### CI
+
+- test project weekly
+- enable renovate for 3.0.x ([#2277](https://github.com/libretime/libretime/issues/2277))
+- sync docs with libretime/website repository
+- pin vale version to v2.21.3
+- don't squash commits during docs sync
+- always print diff when schema changes
+- check if locale are up to date
+- update locales weekly, not for every commit ([#2403](https://github.com/libretime/libretime/issues/2403))
+- use bake file for container build
+- allow manual ci trigger
+- use bot to update locales
+- replace deprecated set-output ([#2408](https://github.com/libretime/libretime/issues/2408))
+- update docker hub containers description
+- replace stale bot with stale action ([#2421](https://github.com/libretime/libretime/issues/2421))
+- allow Falso as a word in codespell
+- allow Falso as a word in codespell
+- run all tests on python tools changes
+- don't run stale bot on feature requests ([#2527](https://github.com/libretime/libretime/issues/2527))
+
+### Reverts
+
+- chore(api): install django-rest-framework from git ([#2518](https://github.com/libretime/libretime/issues/2518))
+
+## [3.0.2](https://github.com/libretime/libretime/compare/3.0.1...3.0.2) (2023-02-21)
+
+- [Release note](https://libretime.org/docs/releases/3.0.2/)
+
+### Bug Fixes
+
+- **legacy:** advanced search by track type id
+- **legacy:** refresh lock files
+- **legacy:** move forked deps to the libretime namespace
+- **legacy:** improve error messages and logs
+- **installer:** allow different actions on template_file
+- **installer:** print diff on file deployment
+- **installer:** only setup nginx on first install
+- **installer:** print unsupported distribution error ([#2368](https://github.com/libretime/libretime/issues/2368))
+- **installer:** create systemd dirs if missing ([#2379](https://github.com/libretime/libretime/issues/2379))
+
+### Documentation
+
+- add DOCKER_BUILDKIT env variable for docker-compose v1 ([#2270](https://github.com/libretime/libretime/issues/2270))
+- check logs before checking services status
+- add small faq for troubleshooting
+
+### Tests
+
+- **playout:** refresh snapshots after major upgrade ([#2381](https://github.com/libretime/libretime/issues/2381))
+
+### CI
+
+- don't squash commits during docs sync
+- test project weekly
+
+## [3.0.1](https://github.com/libretime/libretime/compare/3.0.0...3.0.1) (2022-12-20)
+
+- [Release note](https://libretime.org/docs/releases/3.0.1/)
+
+### Bug Fixes
+
+- remove systemd ProtectHome feature ([#2244](https://github.com/libretime/libretime/issues/2244))
+- **installer:** install missing sudo
+- **installer:** set home and login when running as postgres
+- **legacy:** add log entry on task run ([#2316](https://github.com/libretime/libretime/issues/2316))
+- **legacy:** log errors on connect check failure ([#2317](https://github.com/libretime/libretime/issues/2317))
+- **worker:** replace deprecated cgi.parse_header
+
+### Documentation
+
+- no need to update release note path
+
+### Tests
+
+- liquidsoap package from ppa is version 1.4.2 ([#2233](https://github.com/libretime/libretime/issues/2233))
+
+### CI
+
+- run tests on 3.0.x
+- enable renovate bot on 3.0.x
+- sync docs with libretime/website repository
+- pin vale version to v2.21.3
+
+## [3.0.0](https://github.com/libretime/libretime/compare/3.0.0-beta.2...3.0.0) (2022-10-10)
+
+- [Release note](https://libretime.org/docs/releases/3.0.0/)
+
+### Bug Fixes
+
+- clean exit by catching keyboard interrupt ([#2206](https://github.com/libretime/libretime/issues/2206))
+- **legacy:** missing plupload uk_UA translation
+- **legacy:** jquery i18n translations for plupload
+- **legacy:** gracefully handle missing asset checksum
+- disable some systemd security features on bionic ([#2219](https://github.com/libretime/libretime/issues/2219))
+
+### Documentation
+
+- **legacy:** how to add a new language
+
+### Tests
+
+- **analyzer:** fix wrong bit_rate values
+
+## [3.0.0-beta.2](https://github.com/libretime/libretime/compare/3.0.0-beta.1...3.0.0-beta.2) (2022-10-03)
+
+- [Release note](https://libretime.org/docs/releases/3.0.0-beta.2/)
+
+### Features
+
+- systemd service hardening ([#2186](https://github.com/libretime/libretime/issues/2186))
+- extra systemd service hardening ([#2197](https://github.com/libretime/libretime/issues/2197))
+
+### Bug Fixes
+
+- start playout service after liquidsoap ([#2164](https://github.com/libretime/libretime/issues/2164))
+- include version variable inside containers
+- change version format
+- **legacy:** add play button to stream player ([#2190](https://github.com/libretime/libretime/issues/2190))
+- **legacy:** correct log levels ([#2196](https://github.com/libretime/libretime/issues/2196))
+
+### Documentation
+
+- remove breaking change warning ([#2180](https://github.com/libretime/libretime/issues/2180))
+- fix vale linting errors
+- fix vale linting error
+
+### CI
+
+- allow failure when linting /docs/releases
+- use github.ref_name to get tag
 
 ## [3.0.0-beta.1](https://github.com/libretime/libretime/compare/3.0.0-beta.0...3.0.0-beta.1) (2022-09-23)
 
@@ -32,8 +465,6 @@
 - don't check github.com/libretime/libretime/(issues|pulls) links
 - run docs workflow on vale files changes
 
-<a name="3.0.0-beta.0"></a>
-
 ## [3.0.0-beta.0](https://github.com/libretime/libretime/compare/3.0.0-alpha.13...3.0.0-beta.0) (2022-09-16)
 
 - [Release note](https://libretime.org/docs/releases/3.0.0-beta.0/)
@@ -183,8 +614,6 @@
 - improve containers build caching
 - add container tags
 
-<a name="3.0.0-alpha.13"></a>
-
 ## [3.0.0-alpha.13](https://github.com/libretime/libretime/compare/3.0.0-alpha.12...3.0.0-alpha.13) (2022-07-15)
 
 - [Release note](https://libretime.org/docs/releases/3.0.0-alpha.13/)
@@ -345,8 +774,6 @@
 - disable codecov project status check
 - disable codecov patch status check
 
-<a name="3.0.0-alpha.12"></a>
-
 ## [3.0.0-alpha.12](https://github.com/libretime/libretime/compare/3.0.0-alpha.11...3.0.0-alpha.12) (2022-03-29)
 
 - [Release note](https://libretime.org/docs/releases/3.0.0-alpha.12/)
@@ -361,8 +788,6 @@
 - add missing data to release note
 - fix and update links ([#1714](https://github.com/libretime/libretime/issues/1714))
 
-<a name="3.0.0-alpha.11"></a>
-
 ## [3.0.0-alpha.11](https://github.com/libretime/libretime/compare/3.0.0-alpha.10...3.0.0-alpha.11) (2022-03-28)
 
 - [Release note](https://libretime.org/docs/releases/3.0.0-alpha.11/)

CONTRIBUTING.md

@@ -1,129 +0,0 @@
-# Contributing to LibreTime
-
-First and foremost, thank you! We appreciate that you want to contribute to
-LibreTime, your time is valuable, and your contributions mean a lot to us.
-
-Before any contribution, read and be prepared to adhere to our
-[code of conduct](https://github.com/libretime/organization/blob/main/CODE_OF_CONDUCT.md).
-
-In addition, LibreTime follow the standardized
-[C4 development process](https://rfc.zeromq.org/spec:42/c4/), in which you can
-find explanation about most of the development workflows for LibreTime.
-
-**How to contribute**
-
-- [Reporting bugs](#reporting-bugs)
-- [Suggesting enhancements](#suggesting-enhancements)
-- [Financially](https://libretime.org/contribute#financial)
-- [Contributing to documentation](https://libretime.org/contribute#write-documentation)
-- [Contributing to code](#code)
-
-## Reporting bugs
-
-This section guides you through submitting a bug report for LibreTime.
-Following these guidelines helps maintainers and the community understand your
-report, reproduce the behavior, and find related reports.
-
-Before creating bug reports, please check the following list, to be sure that
-you need to create one:
-
-- **Check the [LibreTime forum](https://discourse.libretime.org/)** for existing
-  questions and discussion.
-- **Check that your issue does not already exist in the
-  [issue tracker](https://github.com/libretime/libretime/issues?q=is%3aissue+label%3abug)**.
-
-> **Note:** If you find a **Closed** issue that seems like it is the same thing
-> that you're experiencing, open a new issue and include a link to the original
-> issue in the body of your new one.
-
-When you are creating a bug report, please include as many details as possible.
-Fill out the [required template](https://github.com/libretime/libretime/issues/new?labels=bug&template=bug_report.md),
-the information it asks helps the maintainers resolve the issue faster.
-
-Bugs are tracked on the [official issue tracker](https://github.com/libretime/libretime/issues).
-
-## Suggesting enhancements
-
-This section guides you through submitting an enhancement suggestion for
-LibreTime, including completely new features and minor improvements to existing
-functionality. Following these guidelines helps maintainers and the community
-understand your suggestion and find related suggestions.
-
-Before creating enhancement suggestions, please check the following list, as you
-might find out that you don't need to create one:
-
-- **Check the [LibreTime forum](https://discourse.libretime.org/)** for existing
-  questions and discussion.
-- **Check that your issue does not already exist in the
-  [issue tracker](https://github.com/libretime/libretime/issues?q=is%3aissue+label%3afeature-request)**.
-
-When you are creating an enhancement suggestion, please include as many details
-as possible. Fill in [the template](https://github.com/libretime/libretime/issues/new?labels=feature-request&template=feature_request.md),
-including the steps that you imagine you would take if the feature you're
-requesting existed.
-
-## Code
-
-Are you familiar with coding in PHP or Python? Have you made projects in
-Liquidsoap and some of the other services we use? Take a look at the
-[list of bugs and feature requests](https://github.com/libretime/libretime/issues),
-and then fork our repo and have a go! Just use the **Fork** button at the top of
-our [GitHub page](https://github.com/libretime/libretime), clone the forked repo
-to your desktop, open up a favorite editor and make some changes, and then
-commit, push, and open a pull request.
-
-Knowledge on how to use [Github](https://guides.github.com/activities/hello-world/)
-and [Git](https://git-scm.com/docs/gittutorial) will suit you well, use the
-links for a quick 101.
-
-LibreTime uses the [black](https://github.com/psf/black) coding style for Python
-and you must ensure that your code follows it. If not, the CI will fail and your
-Pull Request will not be merged. Similarly, the Python import statements are
-sorted with [isort](https://github.com/pycqa/isort). There is configuration
-provided for [pre-commit](https://pre-commit.com/), which will ensure that code
-matches the expected style and conventions when you commit changes. It is set up
-by running:
-
-```bash
-sudo apt install pre-commit
-pre-commit install
-```
-
-You can also run it anytime using:
-
-```bash
-pre-commit run --all-files
-```
-
-## Testing and CI/CD
-
-Before submitting code to the project, it's a good idea to test it first. To do
-this, it's easiest to install LibreTime in a virtual machine on your local
-system or in a cloud VM. We have instructions for setting up a virtual instance
-of LibreTime with [Vagrant](/docs/vagrant) and [Multipass](/docs/multipass).
-
-If you would like to try LibreTime in a Docker image, Odclive has instructions
-[here](https://github.com/kessibi/libretime-docker) for setting up a test image
-and a more persistent install.
-
-## Modifying the Database
-
-LibreTime is designed to work with a [PostgreSQL](https://www.postgresql.org/)
-database server running locally. LibreTime uses [PropelORM](https://github.com/propelorm/Propel)
-to interact with the ZendPHP components and create the database. The version 2
-API uses Django to interact with the same database.
-
-If you are a developer seeking to add new columns to the database here are the steps.
-
-1. Modify `legacy/build/schema.xml` with any changes.
-2. Run `dev_tools/propel_generate.sh`
-3. Update the upgrade.sql under `legacy/application/controllers/upgrade_sql/VERSION` for example
-   `ALTER TABLE imported_podcast ADD COLUMN album_override boolean default 'f' NOT NULL;`
-4. Update the models under `api/libretime_api/models/` to reflect the new
-   changes.
-
-## Documentation and financial contributions
-
-More information about how to contribute documentation or financially
-through our [OpenCollective](https://opencollective.com/libretime) can be found
-on our [website](https://libretime.org/contribute).

CONTRIBUTING.md

@@ -0,0 +1 @@
+docs/contribute.md

Dockerfile

@@ -2,7 +2,7 @@ ARG LIBRETIME_VERSION
 #======================================================================================#
 # Python Builder                                                                       #
 #======================================================================================#
-FROM python:3.10-slim-bullseye as python-builder
+FROM python:3.10-slim-bullseye AS python-builder
 
 WORKDIR /build
 
@@ -18,7 +18,7 @@ RUN pip wheel --wheel-dir . --no-deps .
 #======================================================================================#
 # Python base                                                                          #
 #======================================================================================#
-FROM python:3.10-slim-bullseye as python-base
+FROM python:3.10-slim-bullseye AS python-base
 
 ENV PYTHONDONTWRITEBYTECODE=1
 ENV PYTHONUNBUFFERED=1
@@ -28,11 +28,9 @@ ARG USER=libretime
 ARG UID=1000
 ARG GID=1000
 
-RUN adduser --disabled-password --uid=$UID --gecos '' --no-create-home ${USER}
+RUN set -eux \
-
+    && adduser --disabled-password --uid=$UID --gecos '' --no-create-home ${USER} \
-RUN install --directory --owner=${USER} \
+    && install --directory --owner=${USER} /etc/libretime /srv/libretime
-    /etc/libretime \
-    /srv/libretime
 
 ENV LIBRETIME_CONFIG_FILEPATH=/etc/libretime/config.yml
 
@@ -40,8 +38,9 @@ ENV LIBRETIME_CONFIG_FILEPATH=/etc/libretime/config.yml
 COPY tools/packages.py /tmp/packages.py
 COPY shared/packages.ini /tmp/packages.ini
 
-RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
+RUN set -eux \
-    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    && DEBIAN_FRONTEND=noninteractive apt-get update \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     $(python3 /tmp/packages.py --format=line --exclude=python bullseye /tmp/packages.ini) \
     && rm -rf /var/lib/apt/lists/* \
     && rm -f /tmp/packages.py /tmp/packages.ini
@@ -49,23 +48,25 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
 #======================================================================================#
 # Python base with ffmpeg                                                              #
 #======================================================================================#
-FROM python-base as python-base-ffmpeg
+FROM python-base AS python-base-ffmpeg
 
-RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
+RUN set -eux \
-    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    && DEBIAN_FRONTEND=noninteractive apt-get update \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     ffmpeg \
     && rm -rf /var/lib/apt/lists/*
 
 #======================================================================================#
 # Analyzer                                                                             #
 #======================================================================================#
-FROM python-base-ffmpeg as libretime-analyzer
+FROM python-base-ffmpeg AS libretime-analyzer
 
 COPY tools/packages.py /tmp/packages.py
 COPY analyzer/packages.ini /tmp/packages.ini
 
-RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
+RUN set -eux \
-    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    && DEBIAN_FRONTEND=noninteractive apt-get update \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     $(python3 /tmp/packages.py --format=line --exclude=python bullseye /tmp/packages.ini) \
     && rm -rf /var/lib/apt/lists/* \
     && rm -f /tmp/packages.py /tmp/packages.ini
@@ -82,7 +83,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
 
 COPY analyzer .
 RUN --mount=type=cache,target=/root/.cache/pip \
-    pip install --editable .
+    pip install --editable .[sentry]
 
 # Run
 USER ${UID}:${GID}
@@ -96,13 +97,14 @@ ENV LIBRETIME_VERSION=$LIBRETIME_VERSION
 #======================================================================================#
 # Playout                                                                              #
 #======================================================================================#
-FROM python-base-ffmpeg as libretime-playout
+FROM python-base-ffmpeg AS libretime-playout
 
 COPY tools/packages.py /tmp/packages.py
 COPY playout/packages.ini /tmp/packages.ini
 
-RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
+RUN set -eux \
-    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    && DEBIAN_FRONTEND=noninteractive apt-get update \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     $(python3 /tmp/packages.py --format=line --exclude=python bullseye /tmp/packages.ini) \
     && rm -rf /var/lib/apt/lists/* \
     && rm -f /tmp/packages.py /tmp/packages.ini
@@ -120,7 +122,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
 
 COPY playout .
 RUN --mount=type=cache,target=/root/.cache/pip \
-    pip install --editable .
+    pip install --editable .[sentry]
 
 # Run
 USER ${UID}:${GID}
@@ -134,10 +136,12 @@ ENV LIBRETIME_VERSION=$LIBRETIME_VERSION
 #======================================================================================#
 # API                                                                                  #
 #======================================================================================#
-FROM python-base as libretime-api
+FROM python-base AS libretime-api
 
-RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
+RUN set -eux \
-    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    && DEBIAN_FRONTEND=noninteractive apt-get update \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    curl \
     gcc \
     libc6-dev \
     libpq-dev \
@@ -147,7 +151,7 @@ WORKDIR /src
 
 COPY api/requirements.txt .
 RUN --mount=type=cache,target=/root/.cache/pip \
-    pip install --no-compile gunicorn uvicorn -r requirements.txt
+    pip install --no-compile -r requirements.txt
 
 COPY --from=python-builder /build/shared/*.whl .
 RUN --mount=type=cache,target=/root/.cache/pip \
@@ -155,7 +159,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
 
 COPY api .
 RUN --mount=type=cache,target=/root/.cache/pip \
-    pip install --editable .[prod]
+    pip install --editable .[prod,sentry]
 
 # Run
 USER ${UID}:${GID}
@@ -163,7 +167,7 @@ WORKDIR /app
 
 CMD ["/usr/local/bin/gunicorn", \
     "--workers=4", \
-    "--worker-class=uvicorn.workers.UvicornWorker", \
+    "--worker-class=libretime_api.gunicorn.Worker", \
     "--log-file", "-", \
     "--bind=0.0.0.0:9001", \
     "libretime_api.asgi"]
@@ -171,10 +175,12 @@ CMD ["/usr/local/bin/gunicorn", \
 ARG LIBRETIME_VERSION
 ENV LIBRETIME_VERSION=$LIBRETIME_VERSION
 
+HEALTHCHECK CMD ["curl", "--fail", "http://localhost:9001/api/v2/version"]
+
 #======================================================================================#
 # Worker                                                                               #
 #======================================================================================#
-FROM python-base as libretime-worker
+FROM python-base AS libretime-worker
 
 WORKDIR /src
 
@@ -183,47 +189,42 @@ RUN --mount=type=cache,target=/root/.cache/pip \
     pip install --no-compile -r requirements.txt
 
 COPY --from=python-builder /build/shared/*.whl .
+COPY --from=python-builder /build/api-client/*.whl .
 RUN --mount=type=cache,target=/root/.cache/pip \
     pip install --no-compile *.whl && rm -Rf *.whl
 
 COPY worker .
 RUN --mount=type=cache,target=/root/.cache/pip \
-    pip install --editable .
+    pip install --editable .[sentry]
 
 # Run
 USER ${UID}:${GID}
 WORKDIR /app
 
-CMD ["/usr/local/bin/celery", "worker", \
+CMD ["/usr/local/bin/libretime-worker"]
-    "--app=libretime_worker.tasks:worker", \
-    "--config=libretime_worker.config", \
-    "--time-limit=1800", \
-    "--concurrency=1", \
-    "--loglevel=info"]
-
 ARG LIBRETIME_VERSION
 ENV LIBRETIME_VERSION=$LIBRETIME_VERSION
 
 #======================================================================================#
 # Legacy                                                                               #
 #======================================================================================#
-FROM php:7.4-fpm as libretime-legacy
+FROM php:7.4-fpm AS libretime-legacy
 
 ENV LIBRETIME_CONFIG_FILEPATH=/etc/libretime/config.yml
+ENV LIBRETIME_LOG_FILEPATH=php://stderr
 
 # Custom user
 ARG USER=libretime
 ARG UID=1000
 ARG GID=1000
 
-RUN adduser --disabled-password --uid=$UID --gecos '' --no-create-home ${USER}
+RUN set -eux \
+    && adduser --disabled-password --uid=$UID --gecos '' --no-create-home ${USER} \
+    && install --directory --owner=${USER} /etc/libretime /srv/libretime
 
-RUN install --directory --owner=${USER} \
+RUN set -eux \
-    /etc/libretime \
+    && DEBIAN_FRONTEND=noninteractive apt-get update \
-    /srv/libretime
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
-
-RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
-    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     gettext \
     libcurl4-openssl-dev \
     libfreetype6-dev \
@@ -268,8 +269,9 @@ COPY legacy/composer.* ./
 RUN composer --no-cache install --no-progress --no-interaction --no-dev --no-autoloader
 
 COPY legacy .
-RUN make locale-build
+RUN set -eux \
-RUN composer --no-cache dump-autoload --no-interaction --no-dev
+    && make locale-build \
+    && composer --no-cache dump-autoload --no-interaction --no-dev
 
 # Run
 USER ${UID}:${GID}

Makefile

@@ -7,36 +7,57 @@ all: setup
 setup:
 	command -v pre-commit > /dev/null && pre-commit install
 
-# https://google.github.io/styleguide/shellguide.html
+.env:
-shell-format:
+	cp .env.dev .env
-	shfmt -f . | xargs git ls-files | xargs shfmt -i 2 -ci -sr -kp -w
 
-shell-check:
+dev-certs:
-	shfmt -f . | xargs git ls-files | xargs shfmt -i 2 -ci -sr -kp -d
+	rm -f dev/certs/fake.*
-	shfmt -f . | xargs git ls-files | xargs shellcheck --color=always --severity=$${SEVERITY:-style}
+	openssl req -x509 \
+		-newkey rsa:2048 \
+		-days 365 \
+		-nodes \
+		-subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" \
+		-keyout dev/certs/fake.key \
+		-out dev/certs/fake.crt
+	cat dev/certs/fake.{key,crt} > dev/certs/fake.pem
+
+dev: .env dev-certs
+	DOCKER_BUILDKIT=1 docker compose build
+	docker compose run --rm legacy make build
+	docker compose run --rm api libretime-api migrate
+	docker compose up -d
 
 .PHONY: VERSION
 VERSION:
 	tools/version.sh
 
-changelog:
-	tools/changelog.sh
-
 .PHONY: tarball
 tarball: VERSION
 	$(MAKE) -C legacy build
 	cd .. && tar -czf libretime-$(shell cat VERSION | tr -d [:blank:]).tar.gz \
 		--owner=root --group=root \
 		--exclude-vcs \
-		--exclude .codespellignore \
+		libretime/analyzer \
-		--exclude .git* \
+		libretime/api \
-		--exclude .pre-commit-config.yaml \
+		libretime/api-client \
-		--exclude dev_tools \
+		libretime/docs \
-		--exclude jekyll.sh \
+		libretime/installer \
+		libretime/legacy \
 		--exclude legacy/vendor/phing \
 		--exclude legacy/vendor/simplepie/simplepie/tests \
-		libretime
+		libretime/playout \
+		libretime/shared \
+		libretime/tools \
+		libretime/worker \
+		libretime/CHANGELOG.md \
+		libretime/install \
+		libretime/LICENSE \
+		libretime/Makefile \
+		libretime/README.md \
+		libretime/SECURITY.md \
+		libretime/VERSION
 	mv ../libretime-*.tar.gz .
+	sha256sum libretime-*.tar.gz > sha256sums.txt
 
 # Only clean subdirs
 clean:
@@ -44,4 +65,13 @@ clean:
 
 docs-lint:
 	vale sync
-	vale docs website/src/pages
+	vale docs
+
+website:
+	git clone git@github.com:libretime/website.git
+
+website/node_modules: website
+	yarn --cwd website install
+
+docs-dev: website website/node_modules
+	DOCS_PATH="../docs" yarn --cwd website start

README.md

@@ -1,4 +1,4 @@
-![](website/static/img/logo-512px.png)
+# [![LibreTime](https://github.com/libretime/website/blob/main/static/img/logo-512px.png)](https://github.com/libretime/libretime)
 
 [![Financial Contributors on Open Collective](https://opencollective.com/libretime/all/badge.svg?label=financial+contributors)](https://opencollective.com/libretime)
 
@@ -9,33 +9,15 @@ It is managed by a friendly inclusive community of stations from around the
 globe that use, document and improve LibreTime. Join us in fixing bugs and in
 defining how we manage the codebase going forward.
 
-We are currently ramping up development on this repository.
+Check out the [documentation](https://libretime.org/docs/) for more information and
-
-Check out the [documentation](https://libretime.org) for more information and
 start broadcasting!
 
 Please note that LibreTime is released with a [Contributor Code
 of Conduct](https://github.com/libretime/organization/blob/main/CODE_OF_CONDUCT.md).
 By participating in this project you agree to abide by its terms.
 
-Please submit enhancements, bug-fixes or comments via GitHub.
+You can find details about our development process in the
-
+[contributing](./CONTRIBUTING.md) guide.
-## Development Process
-
-The LibreTime follows the standardized [Collective Code Construction
-Contract (C4)](https://rfc.zeromq.org/spec:42/C4/). Its abstract is
-provided here.
-
-> C4 provides a standard process for contributing, evaluating and
-> discussing improvements on software projects. It defines specific
-> technical requirements for projects like a style guide, unit tests,
-> git and similar platforms. It also establishes different personas
-> for projects, with clear and distinct duties. C4 specifies a process
-> for documenting and discussing issues including seeking consensus
-> and clear descriptions, use of "pull requests" and systematic reviews.
-
-The full text of the contract is licensed under the GPL and available at
-the above link courtesy of the [ZeroMQ community](https://zeromq.org/).
 
 ## Support
 
@@ -44,7 +26,8 @@ we have a forum at [discourse.libretime.org](https://discourse.libretime.org).
 We are moving towards using the forum to provide community support and reserving
 the github issue queue for confirmed bugs and well-formed feature requests.
 
-You can also contact us through our [Mattermost instance](https://chat.libretime.org)
+You can also contact us through [Matrix
+(#libretime:matrix.org)](https://matrix.to/#/#libretime:matrix.org)
 where you can talk with other users and developers.
 
 ## Contributors
@@ -69,23 +52,8 @@ Become a financial contributor and help us sustain our community on
 [Support](https://opencollective.com/libretime/contribute) this project with
 your organization. Your logo will show up here with a link to your website.
 
-<a href="https://opencollective.com/libretime/organization/0/website">
+<a href="https://opencollective.com/libretime">
-    <img src="https://opencollective.com/libretime/organization/0/avatar.svg">
+    <img src="https://opencollective.com/libretime/organizations.svg?width=890">
-</a>
-<a href="https://opencollective.com/libretime/organization/1/website">
-    <img src="https://opencollective.com/libretime/organization/1/avatar.svg">
-</a>
-<a href="https://opencollective.com/libretime/organization/2/website">
-    <img src="https://opencollective.com/libretime/organization/2/avatar.svg">
-</a>
-<a href="https://opencollective.com/libretime/organization/3/website">
-    <img src="https://opencollective.com/libretime/organization/3/avatar.svg">
-</a>
-<a href="https://opencollective.com/libretime/organization/4/website">
-    <img src="https://opencollective.com/libretime/organization/4/avatar.svg">
-</a>
-<a href="https://opencollective.com/libretime/organization/5/website">
-    <img src="https://opencollective.com/libretime/organization/5/avatar.svg">
 </a>
 
 ## License
@@ -99,6 +67,6 @@ version 3 of the License.
 
 Copyright (c) 2011-2017 Sourcefabric z.ú.
 
-Copyright (c) 2017-2022 LibreTime Community
+Copyright (c) 2017-2023 LibreTime Community
 
 Please refer to the [LEGACY](./LEGACY.md) file for more information.

SECURITY.md

@@ -0,0 +1,14 @@
+# Security Policy
+
+## Reporting a Vulnerability
+
+**Please do not use GitHub issues for security-sensitive communication.**
+
+The LibreTime maintainers ask that known and suspected vulnerabilities to be privately and responsibly disclosed by:
+
+- sending all the required detail to [security@libretime.org](security@libretime.org),
+- or by filling a [security advisory on Github](https://github.com/libretime/libretime/security/advisories/new).
+
+A LibreTime maintainer will acknowledged the report within 3 working days.
+
+We aim to provide a security patch within 30 days, after this period the report will be disclosed to the public. The security patch will be distributed for the [maintained versions of LibreTime](https://libretime.org/docs/releases/#distributions-releases-support).

Vagrantfile

@@ -6,7 +6,7 @@
 # export VAGRANT_NO_PORT_FORWARDING=true
 # export VAGRANT_CPUS=4
 # export VAGRANT_MEMORY=4096
-# vagrant up buster
+# vagrant up bullseye
 #
 
 Vagrant.configure('2') do |config|
@@ -79,7 +79,6 @@ Vagrant.configure('2') do |config|
     LIBRETIME_POSTGRESQL_PASSWORD=libretime \
     LIBRETIME_RABBITMQ_PASSWORD=libretime \
     bash install \
-      --listen-port 8080 \
       --in-place \
       http://192.168.10.100:8080
 
@@ -99,12 +98,6 @@ Vagrant.configure('2') do |config|
     setup_libretime(os, "debian.sh")
   end
 
-  config.vm.define 'bionic' do |os|
-    os.vm.box = 'bento/ubuntu-18.04'
-    setup_nfs(config)
-    setup_libretime(os, 'debian.sh')
-  end
-
   config.vm.define 'bullseye' do |os|
     os.vm.box = 'debian/bullseye64'
     config.vm.provider 'virtualbox' do |v, override|
@@ -113,19 +106,4 @@ Vagrant.configure('2') do |config|
     setup_nfs(config, 4)
     setup_libretime(os, 'debian.sh')
   end
-
-  config.vm.define 'buster' do |os|
-    os.vm.box = 'debian/buster64'
-    config.vm.provider 'virtualbox' do |v, override|
-      override.vm.box = 'bento/debian-10'
-    end
-    setup_nfs(config)
-    setup_libretime(os, 'debian.sh')
-  end
-
-  config.vm.define 'centos' do |os|
-    os.vm.box = 'centos/8'
-    setup_nfs(config)
-    setup_libretime(os, 'centos.sh', '--selinux')
-  end
 end

analyzer/Makefile

@@ -4,11 +4,10 @@ include ../tools/python.mk
 
 PIP_INSTALL := \
 	--editable ../shared \
-	--editable .[dev]
+	--editable .[dev,sentry]
 PYLINT_ARG := libretime_analyzer tests || true
 MYPY_ARG := libretime_analyzer tests || true
 BANDIT_ARG := libretime_analyzer || true
-PYTEST_ARG := --cov=libretime_analyzer tests
 
 format: .format
 lint: .format-check .pylint .mypy .bandit
@@ -17,4 +16,5 @@ fixtures:
 	bash tests/fixtures/generate.sh
 
 test: fixtures .pytest
+test-coverage: fixtures .coverage
 clean: .clean

analyzer/install/systemd/libretime-analyzer.service

@@ -10,7 +10,6 @@ PrivateTmp=true
 PrivateUsers=true
 ProtectClock=true
 ProtectControlGroups=true
-ProtectHome=true
 ProtectHostname=true
 ProtectKernelLogs=true
 ProtectKernelModules=true
@@ -18,6 +17,7 @@ ProtectKernelTunables=true
 ProtectProc=invisible
 ProtectSystem=full
 
+Environment=PYTHONOPTIMIZE=2
 Environment=LIBRETIME_CONFIG_FILEPATH=@@CONFIG_FILEPATH@@
 Environment=LIBRETIME_LOG_FILEPATH=@@LOG_DIR@@/analyzer.log
 WorkingDirectory=@@WORKING_DIR@@/analyzer

analyzer/libretime_analyzer/__init__.py

@@ -0,0 +1,4 @@
+from importlib.metadata import version as get_version
+
+PACKAGE = __name__
+VERSION = get_version(__name__)

analyzer/libretime_analyzer/main.py

@@ -1,15 +1,20 @@
+import logging
+import os
 from pathlib import Path
 from typing import Optional
 
 import click
 from libretime_shared.cli import cli_config_options, cli_logging_options
 from libretime_shared.config import DEFAULT_ENV_PREFIX
-from libretime_shared.logging import level_from_name, setup_logger
+from libretime_shared.logging import setup_logger
 
+from . import PACKAGE, VERSION
 from .config import Config
 from .message_listener import MessageListener
 from .status_reporter import StatusReporter
 
+logger = logging.getLogger(__name__)
+
 VERSION = "1.0"
 
 DEFAULT_RETRY_QUEUE_FILEPATH = Path("retry_queue")
@@ -33,9 +38,19 @@ def cli(
     """
     Run analyzer.
     """
-    setup_logger(level_from_name(log_level), log_filepath)
+    setup_logger(log_level, log_filepath)
     config = Config(config_filepath)
 
+    if "SENTRY_DSN" in os.environ:
+        logger.info("installing sentry")
+        # pylint: disable=import-outside-toplevel
+        import sentry_sdk
+
+        sentry_sdk.init(
+            traces_sample_rate=1.0,
+            release=f"{PACKAGE}@{VERSION}",
+        )
+
     # Start up the StatusReporter process
     StatusReporter.start_thread(retry_queue_filepath)
 

analyzer/libretime_analyzer/message_listener.py

@@ -1,15 +1,17 @@
 import json
+import logging
 import signal
 import time
 from queue import Queue
 
 import pika
-from loguru import logger
 
 from .config import Config
-from .pipeline import Pipeline, PipelineStatus
+from .pipeline import Pipeline, PipelineOptions, PipelineStatus
 from .status_reporter import StatusReporter
 
+logger = logging.getLogger(__name__)
+
 EXCHANGE = "airtime-uploads"
 EXCHANGE_TYPE = "topic"
 ROUTING_KEY = ""
@@ -98,7 +100,7 @@ class MessageListener:
         Here we parse the message, spin up an analyzer process, and report the
         metadata back to the Airtime web application (or report an error).
         """
-        logger.info(f" - Received '{body}' on routing_key '{method_frame.routing_key}'")
+        logger.info("Received '%s' on routing_key '%s'", body, method_frame.routing_key)
 
         audio_file_path = ""
         # final_file_path = ""
@@ -111,17 +113,19 @@ class MessageListener:
                 body = body.decode()
             except (UnicodeDecodeError, AttributeError):
                 pass
-            msg_dict = json.loads(body)
+            msg_dict: dict = json.loads(body)
 
             file_id = msg_dict["file_id"]
             audio_file_path = msg_dict["tmp_file_path"]
             original_filename = msg_dict["original_filename"]
             import_directory = msg_dict["import_directory"]
+            options = msg_dict.get("options", {})
 
             metadata = MessageListener.spawn_analyzer_process(
                 audio_file_path,
                 import_directory,
                 original_filename,
+                options,
             )
 
             callback_url = f"{self.config.general.public_url}/rest/media/{file_id}"
@@ -161,6 +165,7 @@ class MessageListener:
         audio_file_path,
         import_directory,
         original_filename,
+        options: dict,
     ):
         metadata = {}
 
@@ -171,10 +176,11 @@ class MessageListener:
                 audio_file_path,
                 import_directory,
                 original_filename,
+                PipelineOptions(**options),
             )
             metadata = queue.get()
         except Exception as exception:
-            logger.exception(f"Analyzer pipeline exception: {exception}")
+            logger.exception("Analyzer pipeline exception: %s", exception)
             metadata["import_status"] = PipelineStatus.FAILED
 
         # Ensure our queue doesn't fill up and block due to unexpected behavior. Defensive code.

analyzer/libretime_analyzer/pipeline/__init__.py

@@ -1 +1 @@
-from .pipeline import Pipeline, PipelineStatus
+from .pipeline import Pipeline, PipelineOptions, PipelineStatus

analyzer/libretime_analyzer/pipeline/_ffmpeg.py

@@ -36,7 +36,7 @@ def probe_replaygain(filepath: Path) -> Optional[float]:
     """
     Probe replaygain will probe the given audio file and return the replaygain if available.
     """
-    cmd = _ffprobe("-i", filepath)
+    cmd = _ffprobe("-i", filepath, errors="backslashreplace")
 
     track_gain_match = _PROBE_REPLAYGAIN_RE.search(cmd.stderr)
 
@@ -75,8 +75,7 @@ def compute_silences(filepath: Path) -> List[Tuple[float, float]]:
     cmd = _ffmpeg(
         *("-i", filepath),
         "-vn",
-        *("-filter", "highpass=frequency=1000"),
+        *("-filter", "highpass=frequency=80,silencedetect=noise=-60dB:duration=0.9"),
-        *("-filter", "silencedetect=noise=0.15:duration=1"),
     )
 
     starts, ends = [], []
@@ -94,9 +93,8 @@ def compute_silences(filepath: Path) -> List[Tuple[float, float]]:
             end = float(match.group(2))
             ends.append(end)
 
-    # ffmpeg v3 (bionic) does not warn about silence end when the track ends.
+    # If one end is missing, set the last silence ending to infinity, and
-    # Set the last silence ending to infinity, and clamp it to the track duration before
+    # clamp it to the track duration before using this value.
-    # using this value.
     if len(starts) - 1 == len(ends):
         ends.append(inf)
 

analyzer/libretime_analyzer/pipeline/_utils.py

@@ -1,6 +1,7 @@
-from subprocess import PIPE, CalledProcessError, CompletedProcess, run
+import logging
+from subprocess import CalledProcessError, CompletedProcess, run
 
-from loguru import logger
+logger = logging.getLogger(__name__)
 
 
 def run_(*args, **kwargs) -> CompletedProcess:
@@ -8,15 +9,14 @@ def run_(*args, **kwargs) -> CompletedProcess:
         return run(
             args,
             check=True,
-            stdout=PIPE,
+            capture_output=True,
-            stderr=PIPE,
+            text=True,
-            universal_newlines=True,
             **kwargs,
         )
 
     except OSError as exception:  # executable was not found
         cmd = args[0]
-        logger.warning(f"Failed to run: {cmd} - {exception}. Is {cmd} installed?")
+        logger.warning("Failed to run: %s - %s. Is %s installed?", cmd, exception, cmd)
         raise exception
 
     except CalledProcessError as exception:  # returned an error code

analyzer/libretime_analyzer/pipeline/analyze_cuepoint.py

@@ -1,18 +1,18 @@
+import logging
 from datetime import timedelta
 from math import isclose
 from subprocess import CalledProcessError
 from typing import Any, Dict
 
-from loguru import logger
-
 from ._ffmpeg import compute_silences, probe_duration
 
+logger = logging.getLogger(__name__)
 
-def analyze_cuepoint(filepath: str, metadata: Dict[str, Any]) -> Dict[str, Any]:
-    """
-    Extracts the cuein and cueout times along and sets the file duration using ffmpeg.
-    """
 
+def analyze_duration(filepath: str, metadata: Dict[str, Any]) -> Dict[str, Any]:
+    """
+    Extracts the file duration using ffmpeg.
+    """
     try:
         duration = probe_duration(filepath)
 
@@ -30,7 +30,23 @@ def analyze_cuepoint(filepath: str, metadata: Dict[str, Any]) -> Dict[str, Any]:
         metadata["length"] = str(timedelta(seconds=duration))
         metadata["cuein"] = 0.0
         metadata["cueout"] = duration
+    except (CalledProcessError, OSError):
+        pass
 
+    return metadata
+
+
+def analyze_cuepoint(filepath: str, metadata: Dict[str, Any]) -> Dict[str, Any]:
+    """
+    Extracts the cuein and cueout times using ffmpeg.
+
+    This step must run after the 'analyze_duration' step.
+    """
+
+    # Duration has been computed in the 'analyze_duration' step
+    duration = metadata["length_seconds"]
+
+    try:
         silences = compute_silences(filepath)
 
         if len(silences) > 2:

analyzer/libretime_analyzer/pipeline/analyze_metadata.py

@@ -1,10 +1,26 @@
+import logging
 from datetime import timedelta
 from pathlib import Path
 from typing import Any, Dict
 
 import mutagen
 from libretime_shared.files import compute_md5
-from loguru import logger
+from mutagen.easyid3 import EasyID3
+
+logger = logging.getLogger(__name__)
+
+
+def flatten(xss):
+    return [x for xs in xss for x in xs]
+
+
+def comment_get(id3, _):
+    comments = [v.text for k, v in id3.items() if "COMM" in k or "comment" in k]
+
+    return flatten(comments)
+
+
+EasyID3.RegisterKey("comment", comment_get)
 
 
 def analyze_metadata(filepath_: str, metadata: Dict[str, Any]):
@@ -24,7 +40,7 @@ def analyze_metadata(filepath_: str, metadata: Dict[str, Any]):
     # Get audio file metadata
     extracted = mutagen.File(filepath, easy=True)
     if extracted is None:
-        logger.warning(f"no metadata were extracted for {filepath}")
+        logger.warning("no metadata were extracted for %s", filepath)
         return metadata
 
     metadata["mime"] = extracted.mime[0]
@@ -69,34 +85,36 @@ def analyze_metadata(filepath_: str, metadata: Dict[str, Any]):
     except (AttributeError, KeyError, IndexError):
         pass
 
-    extracted_tags_mapping = {
+    extracted_tags_mapping = [
-        "title": "track_title",
+        ("title", "track_title"),
-        "artist": "artist_name",
+        ("artist", "artist_name"),
-        "album": "album_title",
+        ("album", "album_title"),
-        "bpm": "bpm",
+        ("bpm", "bpm"),
-        "composer": "composer",
+        ("composer", "composer"),
-        "conductor": "conductor",
+        ("conductor", "conductor"),
-        "copyright": "copyright",
+        ("copyright", "copyright"),
-        "comment": "comment",
+        ("comment", "comment"),
-        "encoded_by": "encoder",
+        ("comment", "comments"),
-        "genre": "genre",
+        ("comment", "description"),
-        "isrc": "isrc",
+        ("encoded_by", "encoder"),
-        "label": "label",
+        ("genre", "genre"),
-        "organization": "label",
+        ("isrc", "isrc"),
-        # "length": "length",
+        ("label", "label"),
-        "language": "language",
+        ("organization", "label"),
-        "last_modified": "last_modified",
+        # ("length", "length"),
-        "mood": "mood",
+        ("language", "language"),
-        "bit_rate": "bit_rate",
+        ("last_modified", "last_modified"),
-        "replay_gain": "replaygain",
+        ("mood", "mood"),
-        # "tracknumber": "track_number",
+        ("bit_rate", "bit_rate"),
-        # "track_total": "track_total",
+        ("replay_gain", "replaygain"),
-        "website": "website",
+        # ("tracknumber", "track_number"),
-        "date": "year",
+        # ("track_total", "track_total"),
-        # "mime_type": "mime",
+        ("website", "website"),
-    }
+        ("date", "year"),
+        # ("mime_type", "mime"),
+    ]
 
-    for extracted_key, metadata_key in extracted_tags_mapping.items():
+    for extracted_key, metadata_key in extracted_tags_mapping:
         try:
             metadata[metadata_key] = extracted[extracted_key]
             if isinstance(metadata[metadata_key], list):

analyzer/libretime_analyzer/pipeline/analyze_playability.py

@@ -1,10 +1,11 @@
+import logging
 from subprocess import CalledProcessError
 from typing import Any, Dict
 
-from loguru import logger
-
 from ._liquidsoap import _liquidsoap
 
+logger = logging.getLogger(__name__)
+
 
 class UnplayableFileError(Exception):
     pass
@@ -26,6 +27,6 @@ def analyze_playability(filename: str, metadata: Dict[str, Any]):
         raise UnplayableFileError() from exception
 
     except OSError as exception:  # liquidsoap was not found
-        logger.warning(f"Failed to run: {exception}. Is liquidsoap installed?")
+        logger.warning("Failed to run: %s. Is liquidsoap installed?", exception)
 
     return metadata

analyzer/libretime_analyzer/pipeline/organise_file.py

@@ -1,8 +1,9 @@
+import logging
 import shutil
 from pathlib import Path
 from uuid import uuid4
 
-from loguru import logger
+logger = logging.getLogger(__name__)
 
 MAX_DIR_LEN = 48
 MAX_FILE_LEN = 48
@@ -42,12 +43,12 @@ def organise_file(
             return metadata
 
         dest_path = dest_path.with_name(f"{dest_path.stem}_{uuid4()}{dest_path.suffix}")
-        logger.warning(f"found existing file, using new filepath {dest_path}")
+        logger.warning("found existing file, using new filepath %s", dest_path)
 
     # Import
     dest_path.parent.mkdir(parents=True, exist_ok=True)
 
-    logger.debug(f"moving {filepath} to {dest_path}")
+    logger.debug("moving %s to %s", filepath, dest_path)
     shutil.move(filepath, dest_path)
 
     metadata["full_path"] = str(dest_path)

analyzer/libretime_analyzer/pipeline/pipeline.py

@@ -1,21 +1,22 @@
+import logging
 from enum import Enum
 from queue import Queue
-from typing import Any, Dict
+from typing import Any, Dict, Protocol
 
-from loguru import logger
+from pydantic import BaseModel
-from typing_extensions import Protocol
 
-from .analyze_cuepoint import analyze_cuepoint
+from .analyze_cuepoint import analyze_cuepoint, analyze_duration
 from .analyze_metadata import analyze_metadata
 from .analyze_playability import UnplayableFileError, analyze_playability
 from .analyze_replaygain import analyze_replaygain
 from .organise_file import organise_file
 
+logger = logging.getLogger(__name__)
+
 
 class Step(Protocol):
     @staticmethod
-    def __call__(filename: str, metadata: Dict[str, Any]):
+    def __call__(filename: str, metadata: Dict[str, Any]): ...
-        ...
 
 
 class PipelineStatus(int, Enum):
@@ -24,6 +25,10 @@ class PipelineStatus(int, Enum):
     FAILED = 2
 
 
+class PipelineOptions(BaseModel):
+    analyze_cue_points: bool = False
+
+
 class Pipeline:
     """Analyzes and imports an audio file into the Airtime library.
 
@@ -34,10 +39,11 @@ class Pipeline:
 
     @staticmethod
     def run_analysis(
-        queue,
+        queue: Queue,
-        audio_file_path,
+        audio_file_path: str,
-        import_directory,
+        import_directory: str,
-        original_filename,
+        original_filename: str,
+        options: PipelineOptions,
     ):
         """Analyze and import an audio file, and put all extracted metadata into queue.
 
@@ -78,6 +84,8 @@ class Pipeline:
             # First, we extract the ID3 tags and other metadata:
             metadata = {}
             metadata = analyze_metadata(audio_file_path, metadata)
+            metadata = analyze_duration(audio_file_path, metadata)
+            if options.analyze_cue_points:
                 metadata = analyze_cuepoint(audio_file_path, metadata)
             metadata = analyze_replaygain(audio_file_path, metadata)
             metadata = analyze_playability(audio_file_path, metadata)

analyzer/libretime_analyzer/status_reporter.py

@@ -1,5 +1,6 @@
 import collections
 import json
+import logging
 import pickle
 import queue
 import threading
@@ -7,9 +8,10 @@ import time
 from urllib.parse import urlparse
 
 import requests
-from loguru import logger
 from requests.exceptions import HTTPError
 
+logger = logging.getLogger(__name__)
+
 
 class PicklableHttpRequest:
     def __init__(self, method, url, api_key, data):
@@ -55,7 +57,7 @@ def process_http_requests(ipc_queue, http_retry_queue_path):
         # If we fail to unpickle a saved queue of failed HTTP requests, then we'll just log an error
         # and continue because those HTTP requests are lost anyways. The pickled file will be
         # overwritten the next time the analyzer is shut down too.
-        logger.error(f"Failed to unpickle {http_retry_queue_path}. Continuing...")
+        logger.error("Failed to unpickle %s. Continuing...", http_retry_queue_path)
 
     while True:
         try:
@@ -86,10 +88,12 @@ def process_http_requests(ipc_queue, http_retry_queue_path):
             with open(http_retry_queue_path, "wb") as pickle_file:
                 pickle.dump(retry_queue, pickle_file)
             return
-        except Exception as exception:  # Terrible top-level exception handler to prevent the thread from dying, just in case.
+        except (
+            Exception
+        ) as exception:  # Terrible top-level exception handler to prevent the thread from dying, just in case.
             if shutdown:
                 return
-            logger.exception(f"Unhandled exception in StatusReporter {exception}")
+            logger.exception("Unhandled exception in StatusReporter %s", exception)
             logger.info("Restarting StatusReporter thread")
             time.sleep(2)  # Throttle it
 
@@ -114,7 +118,7 @@ def send_http_request(picklable_request: PicklableHttpRequest, retry_queue):
             # The request failed with an error 500 probably, so let's check if Airtime and/or
             # the web server are broken. If not, then our request was probably causing an
             # error 500 in the media API (ie. a bug), so there's no point in retrying it.
-            logger.exception(f"HTTP request failed: {exception}")
+            logger.exception("HTTP request failed: %s", exception)
             parsed_url = urlparse(exception.response.request.url)
             if is_web_server_broken(parsed_url.scheme + "://" + parsed_url.netloc):
                 # If the web server is having problems, retry the request later:
@@ -124,11 +128,12 @@ def send_http_request(picklable_request: PicklableHttpRequest, retry_queue):
                 # notified by sentry.
     except requests.exceptions.ConnectionError as exception:
         logger.exception(
-            f"HTTP request failed due to a connection error. Retrying later. {exception}"
+            "HTTP request failed due to a connection error,  retrying later: %s",
+            exception,
         )
         retry_queue.append(picklable_request)  # Retry it later
     except Exception as exception:
-        logger.exception(f"HTTP request failed with unhandled exception. {exception}")
+        logger.exception("HTTP request failed with unhandled exception. %s", exception)
         # Don't put the request into the retry queue, just give up on this one.
         # I'm doing this to protect against us getting some pathological request
         # that breaks our code. I don't want us pickling data that potentially
@@ -210,7 +215,7 @@ class StatusReporter:
         audio_metadata["import_status"] = import_status
         audio_metadata["comment"] = reason  # hack attack
         put_payload = json.dumps(audio_metadata)
-        # logger.debug("sending http put with payload: " + put_payload)
+        # logger.debug("sending http put with payload: %s", put_payload)
 
         StatusReporter._send_http_request(
             PicklableHttpRequest(

analyzer/packages.ini

@@ -1,32 +1,16 @@
 # This file contains a list of package dependencies.
 [python]
-python3 = buster, bullseye, bookworm, bionic, focal, jammy
+python3 = focal, bullseye, jammy, bookworm
-python3-pip = buster, bullseye, bookworm, bionic, focal, jammy
+python3-pip = focal, bullseye, jammy, bookworm
-python3-pika = buster, bullseye, bookworm, bionic, focal, jammy
 
 [liquidsoap]
 # https://github.com/savonet/liquidsoap/blob/main/CHANGES.md
-liquidsoap-plugin-alsa = bionic
+liquidsoap = focal, bullseye, jammy, bookworm
-liquidsoap-plugin-ao = bionic
-liquidsoap-plugin-ogg = bionic
-liquidsoap-plugin-portaudio = bionic
-# Already recommended packages in bionic
-# See `apt show liquidsoap`
-; liquidsoap-plugin-faad = bionic
-; liquidsoap-plugin-flac = bionic
-; liquidsoap-plugin-icecast = bionic
-; liquidsoap-plugin-lame = bionic
-; liquidsoap-plugin-mad = bionic
-; liquidsoap-plugin-pulseaudio = bionic
-; liquidsoap-plugin-taglib = bionic
-; liquidsoap-plugin-voaacenc = bionic
-; liquidsoap-plugin-vorbis = bionic
-liquidsoap = buster, bullseye, bookworm, bionic, focal, jammy
 
 [ffmpeg]
 # Detect duration, silences and replaygain
-ffmpeg = buster, bullseye, bookworm, bionic, focal, jammy
+ffmpeg = focal, bullseye, jammy, bookworm
 
 [=development]
 # Generate fixtures
-ffmpeg = buster, bullseye, bookworm, bionic, focal, jammy
+ffmpeg = focal, bullseye, jammy, bookworm

analyzer/pyproject.toml

@@ -1,3 +1,8 @@
+[tool.isort]
+profile = "black"
+combine_as_imports = true
+known_first_party = ["libretime_analyzer"]
+
 [tool.pylint.messages_control]
 extension-pkg-whitelist = "pydantic"
 disable = [
@@ -6,13 +11,13 @@ disable = [
   "missing-module-docstring",
 ]
 
-[tool.pylint.format]
-disable = "logging-fstring-interpolation"
-
 [tool.pytest.ini_options]
 log_cli = true
 log_cli_level = "DEBUG"
 
+[tool.coverage.run]
+source = ["libretime_analyzer"]
+
 [build-system]
 requires = ["setuptools", "wheel"]
 build-backend = "setuptools.build_meta"

analyzer/requirements.txt

@@ -1,6 +1,6 @@
 # Please do not edit this file, edit the setup.py file!
 # This file is auto-generated by tools/extract_requirements.py.
-mutagen>=1.45.1,<1.46
+mutagen>=1.45.1,<1.48
 pika>=1.0.0,<1.4
-requests>=2.25.1,<2.29
+requests>=2.32.2,<2.33
 typing_extensions

analyzer/setup.py

@@ -1,8 +1,10 @@
 from setuptools import find_packages, setup
 
+version = "4.2.0"  # x-release-please-version
+
 setup(
     name="libretime-analyzer",
-    version="3.0.0-beta.2",
+    version=version,
     description="Libretime Analyzer",
     author="LibreTime Contributors",
     url="https://github.com/libretime/libretime",
@@ -18,17 +20,20 @@ setup(
             "libretime-analyzer=libretime_analyzer.main:cli",
         ]
     },
-    python_requires=">=3.6",
+    python_requires=">=3.8",
     install_requires=[
-        "mutagen>=1.45.1,<1.46",
+        "mutagen>=1.45.1,<1.48",
         "pika>=1.0.0,<1.4",
-        "requests>=2.25.1,<2.29",
+        "requests>=2.32.2,<2.33",
         "typing_extensions",
     ],
     extras_require={
         "dev": [
-            "distro",
+            "distro>=1.8.0,<2",
-            "types-requests",
+            "types-requests>=2.31.0,<3",
+        ],
+        "sentry": [
+            "sentry-sdk>=1.15.0,<2",
         ],
     },
     zip_safe=False,

analyzer/tests/conftest.py

@@ -2,11 +2,11 @@ import shutil
 from pathlib import Path
 
 import pytest
-from libretime_shared.logging import TRACE, setup_logger
+from libretime_shared.logging import setup_logger
 
 from .fixtures import fixtures_path
 
-setup_logger(TRACE)
+setup_logger("debug")
 
 AUDIO_FILENAME = "s1-stereo-tagged.mp3"
 AUDIO_FILE = fixtures_path / AUDIO_FILENAME

analyzer/tests/fixtures/__init__.py

@@ -23,28 +23,28 @@ FILES = [
 # 8s  -> 9s: silence
 # 9s  -> 12s: musik
 # 12s -> 15s: pink noise fade out
-Fixture(here / "s1-jointstereo.mp3",    15.0,   6.0,    13.0,   -5.9    ),
+Fixture(here / "s1-jointstereo.mp3",    15.0,   1.4,    15.0,   -5.9    ),
-Fixture(here / "s1-mono.mp3",           15.0,   6.0,    13.0,   -2.0    ),
+Fixture(here / "s1-mono.mp3",           15.0,   1.5,    15.0,   -2.0    ),
-Fixture(here / "s1-stereo.mp3",         15.0,   6.0,    13.0,   -5.9    ),
+Fixture(here / "s1-stereo.mp3",         15.0,   1.4,    15.0,   -5.9    ),
-Fixture(here / "s1-mono-12.mp3",        15.0,   9.0,    12.0,   +7.0    ),
+Fixture(here / "s1-mono-12.mp3",        15.0,   1.2,    15.0,   +7.0    ),
-Fixture(here / "s1-stereo-12.mp3",      15.0,   9.0,    12.0,   +6.1    ),
+Fixture(here / "s1-stereo-12.mp3",      15.0,   1.2,    15.0,   +6.1    ),
-Fixture(here / "s1-mono+12.mp3",        15.0,   3.5,    13.0,   -17.0   ),
+Fixture(here / "s1-mono+12.mp3",        15.0,   1.2,    15.0,   -17.0   ),
-Fixture(here / "s1-stereo+12.mp3",      15.0,   3.5,    13.0,   -17.8   ),
+Fixture(here / "s1-stereo+12.mp3",      15.0,   1.2,    15.0,   -17.8   ),
-Fixture(here / "s1-mono.flac",          15.0,   6.0,    13.0,   -2.3    ),
+Fixture(here / "s1-mono.flac",          15.0,   1.4,    15.0,   -2.3    ),
-Fixture(here / "s1-stereo.flac",        15.0,   6.0,    13.0,   -6.0    ),
+Fixture(here / "s1-stereo.flac",        15.0,   1.4,    15.0,   -6.0    ),
-Fixture(here / "s1-mono-12.flac",       15.0,   9.0,    12.0,   +10.0   ),
+Fixture(here / "s1-mono-12.flac",       15.0,   2.0,    15.0,   +10.0   ),
-Fixture(here / "s1-stereo-12.flac",     15.0,   9.0,    12.0,   +5.9    ),
+Fixture(here / "s1-stereo-12.flac",     15.0,   1.8,    15.0,   +5.9    ),
-Fixture(here / "s1-mono+12.flac",       15.0,   3.5,    13.0,   -12.0   ),
+Fixture(here / "s1-mono+12.flac",       15.0,   0.0,    15.0,   -12.0   ),
-Fixture(here / "s1-stereo+12.flac",     15.0,   3.5,    13.0,   -14.9   ),
+Fixture(here / "s1-stereo+12.flac",     15.0,   0.0,    15.0,   -14.9   ),
-Fixture(here / "s1-mono.m4a",           15.0,   6.0,    13.0,   -4.5    ),
+Fixture(here / "s1-mono.m4a",           15.0,   1.4,    15.0,   -4.5    ),
-Fixture(here / "s1-stereo.m4a",         15.0,   6.0,    13.0,   -5.8    ),
+Fixture(here / "s1-stereo.m4a",         15.0,   1.4,    15.0,   -5.8    ),
-Fixture(here / "s1-mono.ogg",           15.0,   6.0,    13.0,   -4.9    ),
+Fixture(here / "s1-mono.ogg",           15.0,   1.4,    15.0,   -4.9    ),
-Fixture(here / "s1-stereo.ogg",         15.0,   6.0,    13.0,   -5.7    ),
+Fixture(here / "s1-stereo.ogg",         15.0,   1.4,    15.0,   -5.7    ),
-Fixture(here / "s1-stereo",             15.0,   6.0,    13.0,   -5.7    ),
+Fixture(here / "s1-stereo",             15.0,   1.4,    15.0,   -5.7    ),
-Fixture(here / "s1-mono.wav",           15.0,   6.0,    13.0,   -2.3    ),
+Fixture(here / "s1-mono.wav",           15.0,   1.5,    15.0,   -2.3    ),
-Fixture(here / "s1-stereo.wav",         15.0,   6.0,    13.0,   -6.0    ),
+Fixture(here / "s1-stereo.wav",         15.0,   1.4,    15.0,   -6.0    ),
 # sample 1 large (looped for 2 hours)
-Fixture(here / "s1-large.flac",         7200,   6.0,    7198,   -6.0    ),
+Fixture(here / "s1-large.flac",         7200,   1.4,    7200,   -6.0    ),
 # sample 2
 # 0s   -> 1.8s: silence
 # 1.8s        : noise
@@ -96,12 +96,18 @@ tags = {
     "comment": "Test Comment",
 }
 
+mp3Tags = {
+    **tags,
+    "comments": tags["comment"],
+    "description": tags["comment"],
+}
+
 FILES_TAGGED = [
     FixtureMeta(
         here / "s1-jointstereo-tagged.mp3",
         {
             **meta,
-            **tags,
+            **mp3Tags,
             "bit_rate": approx(128000, abs=1e2),
             "channels": 2,
             "mime": "audio/mp3",
@@ -111,7 +117,7 @@ FILES_TAGGED = [
         here / "s1-mono-tagged.mp3",
         {
             **meta,
-            **tags,
+            **mp3Tags,
             "bit_rate": approx(64000, abs=1e2),
             "channels": 1,
             "mime": "audio/mp3",
@@ -121,7 +127,7 @@ FILES_TAGGED = [
         here / "s1-stereo-tagged.mp3",
         {
             **meta,
-            **tags,
+            **mp3Tags,
             "bit_rate": approx(128000, abs=1e2),
             "channels": 2,
             "mime": "audio/mp3",
@@ -151,7 +157,7 @@ FILES_TAGGED = [
         here / "s1-mono-tagged.m4a",
         {
             **meta,
-            **tags,
+            **mp3Tags,
             "bit_rate": approx(65000, abs=5e4),
             "channels": 2,  # Weird
             "mime": "audio/mp4",
@@ -161,7 +167,7 @@ FILES_TAGGED = [
         here / "s1-stereo-tagged.m4a",
         {
             **meta,
-            **tags,
+            **mp3Tags,
             "bit_rate": approx(128000, abs=1e5),
             "channels": 2,
             "mime": "audio/mp4",
@@ -201,7 +207,7 @@ FILES_TAGGED = [
         here / "s1-mono-tagged.wav",
         {
             **meta,
-            "bit_rate": approx(96000, abs=1e2),
+            "bit_rate": approx(768000, abs=1e2),
             "channels": 1,
             "mime": "audio/wav",
         },
@@ -210,7 +216,7 @@ FILES_TAGGED = [
         here / "s1-stereo-tagged.wav",
         {
             **meta,
-            "bit_rate": approx(384000, abs=1e2),
+            "bit_rate": approx(1536000, abs=1e2),
             "channels": 2,
             "mime": "audio/wav",
         },
@@ -228,12 +234,18 @@ tags = {
     "comment": "Ł Ą Ż Ę Ć Ń Ś Ź",
 }
 
+mp3Tags = {
+    **tags,
+    "comments": tags["comment"],
+    "description": tags["comment"],
+}
+
 FILES_TAGGED += [
     FixtureMeta(
         here / "s1-jointstereo-tagged-utf8.mp3",
         {
             **meta,
-            **tags,
+            **mp3Tags,
             "bit_rate": approx(128000, abs=1e2),
             "channels": 2,
             "mime": "audio/mp3",
@@ -243,7 +255,7 @@ FILES_TAGGED += [
         here / "s1-mono-tagged-utf8.mp3",
         {
             **meta,
-            **tags,
+            **mp3Tags,
             "bit_rate": approx(64000, abs=1e2),
             "channels": 1,
             "mime": "audio/mp3",
@@ -253,7 +265,7 @@ FILES_TAGGED += [
         here / "s1-stereo-tagged-utf8.mp3",
         {
             **meta,
-            **tags,
+            **mp3Tags,
             "bit_rate": approx(128000, abs=1e2),
             "channels": 2,
             "mime": "audio/mp3",
@@ -283,7 +295,7 @@ FILES_TAGGED += [
         here / "s1-mono-tagged-utf8.m4a",
         {
             **meta,
-            **tags,
+            **mp3Tags,
             "bit_rate": approx(65000, abs=5e4),
             "channels": 2,  # Weird
             "mime": "audio/mp4",
@@ -293,7 +305,7 @@ FILES_TAGGED += [
         here / "s1-stereo-tagged-utf8.m4a",
         {
             **meta,
-            **tags,
+            **mp3Tags,
             "bit_rate": approx(128000, abs=1e5),
             "channels": 2,
             "mime": "audio/mp4",
@@ -333,7 +345,7 @@ FILES_TAGGED += [
         here / "s1-mono-tagged-utf8.wav",
         {
             **meta,
-            "bit_rate": approx(96000, abs=1e2),
+            "bit_rate": approx(768000, abs=1e2),
             "channels": 1,
             "mime": "audio/wav",
         },
@@ -342,7 +354,7 @@ FILES_TAGGED += [
         here / "s1-stereo-tagged-utf8.wav",
         {
             **meta,
-            "bit_rate": approx(384000, abs=1e2),
+            "bit_rate": approx(1536000, abs=1e2),
             "channels": 2,
             "mime": "audio/wav",
         },

analyzer/tests/pipeline/analyze_cuepoint_test.py

@@ -1,7 +1,9 @@
-import distro
 import pytest
 
-from libretime_analyzer.pipeline.analyze_cuepoint import analyze_cuepoint
+from libretime_analyzer.pipeline.analyze_cuepoint import (
+    analyze_cuepoint,
+    analyze_duration,
+)
 
 from ..fixtures import FILES
 
@@ -16,11 +18,8 @@ from ..fixtures import FILES
     ),
 )
 def test_analyze_cuepoint(filepath, length, cuein, cueout):
-    metadata = analyze_cuepoint(filepath, {})
+    metadata = analyze_duration(filepath, {})
-
+    metadata = analyze_cuepoint(filepath, metadata)
-    # On bionic, large file duration is a wrong.
-    if distro.codename() == "bionic" and str(filepath).endswith("s1-large.flac"):
-        return
 
     assert metadata["length_seconds"] == pytest.approx(length, abs=0.1)
     assert float(metadata["cuein"]) == pytest.approx(float(cuein), abs=1)

analyzer/tests/pipeline/analyze_metadata_test.py

@@ -27,8 +27,8 @@ def test_analyze_metadata(filepath: Path, metadata: dict):
     del metadata["length"]
     del found["length"]
 
-    # mp3,ogg,flac files does not support comments yet
+    # ogg,flac files does not support comments yet
-    if not filepath.suffix == ".m4a":
+    if not filepath.suffix == ".m4a" and not filepath.suffix == ".mp3":
         if "comment" in metadata:
             del metadata["comment"]
 

analyzer/tests/pipeline/analyze_playability_test.py

@@ -33,8 +33,8 @@ def test_analyze_playability_invalid_filepath():
 
 
 def test_analyze_playability_invalid_wma():
-    # Liquisoap does not fail with wma files on buster, bullseye, focal, jammy
+    # Liquisoap does not fail with wma files on focal, bullseye, jammy
-    if distro.codename() in ("buster", "bullseye", "focal", "jammy"):
+    if distro.codename() in ("focal", "bullseye", "jammy"):
         return
 
     with pytest.raises(UnplayableFileError):

analyzer/tests/pipeline/analyze_replaygain_test.py

@@ -1,4 +1,3 @@
-import distro
 import pytest
 
 from libretime_analyzer.pipeline.analyze_replaygain import analyze_replaygain
@@ -13,10 +12,5 @@ from ..fixtures import FILES
 def test_analyze_replaygain(filepath, replaygain):
     tolerance = 0.8
 
-    # On bionic, replaygain is a bit higher for loud mp3 files.
-    # This huge tolerance makes the test pass, with values devianting from ~-17 to ~-13
-    if distro.codename() == "bionic" and str(filepath).endswith("+12.mp3"):
-        tolerance = 5
-
     metadata = analyze_replaygain(filepath, {})
     assert metadata["replay_gain"] == pytest.approx(replaygain, abs=tolerance)

analyzer/tests/pipeline/ffmpeg_test.py

@@ -1,6 +1,3 @@
-from math import inf
-
-import distro
 import pytest
 
 from libretime_analyzer.pipeline._ffmpeg import (
@@ -30,11 +27,6 @@ def test_probe_replaygain(filepath, replaygain):
 def test_compute_replaygain(filepath, replaygain):
     tolerance = 0.8
 
-    # On bionic, replaygain is a bit higher for loud mp3 files.
-    # This huge tolerance makes the test pass, with values devianting from ~-17 to ~-13
-    if distro.codename() == "bionic" and str(filepath).endswith("+12.mp3"):
-        tolerance = 5
-
     assert compute_replaygain(filepath) == pytest.approx(replaygain, abs=tolerance)
 
 
@@ -86,10 +78,6 @@ def test_silence_detect_re(line, expected):
 def test_compute_silences(filepath, length, cuein, cueout):
     result = compute_silences(filepath)
 
-    # On bionic, large file duration is a wrong.
-    if distro.codename() == "bionic" and str(filepath).endswith("s1-large.flac"):
-        return
-
     if cuein != 0.0:
         assert len(result) > 0
         first = result.pop(0)
@@ -97,11 +85,6 @@ def test_compute_silences(filepath, length, cuein, cueout):
         assert first[1] == pytest.approx(cuein, abs=1)
 
     if cueout != length:
-        # ffmpeg v3 (bionic) does not warn about silence end when the track ends.
-        # Check for infinity on last silence ending
-        if distro.codename() == "bionic":
-            length = inf
-
         assert len(result) > 0
         last = result.pop()
         assert last[0] == pytest.approx(cueout, abs=1)
@@ -113,8 +96,4 @@ def test_compute_silences(filepath, length, cuein, cueout):
     map(lambda i: pytest.param(i.path, i.length, id=i.path.name), FILES),
 )
 def test_probe_duration(filepath, length):
-    # On bionic, large file duration is a wrong.
-    if distro.codename() == "bionic" and str(filepath).endswith("s1-large.flac"):
-        return
-
     assert probe_duration(filepath) == pytest.approx(length, abs=0.05)

analyzer/tests/pipeline/pipeline_test.py

@@ -4,7 +4,7 @@ from queue import Queue
 
 import pytest
 
-from libretime_analyzer.pipeline import Pipeline
+from libretime_analyzer.pipeline import Pipeline, PipelineOptions
 
 from ..conftest import AUDIO_FILENAME, AUDIO_IMPORT_DEST
 
@@ -16,6 +16,7 @@ def test_run_analysis(src_dir: Path, dest_dir: Path):
         str(src_dir / AUDIO_FILENAME),
         str(dest_dir),
         AUDIO_FILENAME,
+        PipelineOptions(),
     )
     metadata = queue.get()
 

api-client/Makefile

@@ -5,12 +5,12 @@ include ../tools/python.mk
 PIP_INSTALL := \
 	--editable ../shared \
 	--editable .[dev]
-PYLINT_ARG := libretime_api_client tests || true
+PYLINT_ARG := libretime_api_client tests
-MYPY_ARG := libretime_api_client tests || true
+MYPY_ARG := libretime_api_client tests
-BANDIT_ARG := libretime_api_client || true
+BANDIT_ARG := libretime_api_client
-PYTEST_ARG := --cov=libretime_api_client tests
 
 format: .format
 lint: .format-check .pylint .mypy .bandit
 test: .pytest
+test-coverage: .coverage
 clean: .clean

api-client/libretime_api_client/__init__.py

@@ -0,0 +1,4 @@
+from importlib.metadata import version as get_version
+
+PACKAGE = __name__
+VERSION = get_version(__name__)

api-client/libretime_api_client/_client.py

@@ -1,11 +1,13 @@
+import logging
 from typing import Optional
 
-from loguru import logger
 from requests import Response, Session as BaseSession
 from requests.adapters import HTTPAdapter
 from requests.exceptions import RequestException
 from urllib3.util import Retry
 
+logger = logging.getLogger(__name__)
+
 DEFAULT_TIMEOUT = 5
 
 
@@ -24,20 +26,26 @@ class TimeoutHTTPAdapter(HTTPAdapter):
         return super().send(request, *args, **kwargs)
 
 
-class Session(BaseSession):
+def default_retry(max_retries: int = 5):
-    base_url: Optional[str]
+    return Retry(
-
+        total=max_retries,
-    def __init__(self, base_url: Optional[str] = None):
-        super().__init__()
-        self.base_url = base_url
-
-        retry_strategy = Retry(
-            total=5,
         backoff_factor=2,
         status_forcelist=[413, 429, 500, 502, 503, 504],
     )
 
-        adapter = TimeoutHTTPAdapter(max_retries=retry_strategy)
+
+class Session(BaseSession):
+    base_url: Optional[str]
+
+    def __init__(
+        self,
+        base_url: Optional[str] = None,
+        retry: Optional[Retry] = None,
+    ):
+        super().__init__()
+        self.base_url = base_url
+
+        adapter = TimeoutHTTPAdapter(max_retries=retry)
 
         self.mount("http://", adapter)
         self.mount("https://", adapter)
@@ -59,9 +67,16 @@ class AbstractApiClient:
     session: Session
     base_url: str
 
-    def __init__(self, base_url: str):
+    def __init__(
+        self,
+        base_url: str,
+        retry: Optional[Retry] = None,
+    ):
         self.base_url = base_url
-        self.session = Session(base_url=base_url)
+        self.session = Session(
+            base_url=base_url,
+            retry=retry,
+        )
 
     def _request(
         self,

api-client/libretime_api_client/_utils.py

@@ -1,167 +0,0 @@
-import logging
-from time import sleep
-
-import requests
-from requests.auth import AuthBase
-
-
-class UrlParamDict(dict):
-    def __missing__(self, key):
-        return "{" + key + "}"
-
-
-class UrlException(Exception):
-    pass
-
-
-class IncompleteUrl(UrlException):
-    def __init__(self, url):
-        super().__init__()
-        self.url = url
-
-    def __str__(self):
-        return f"Incomplete url: '{self.url}'"
-
-
-class UrlBadParam(UrlException):
-    def __init__(self, url, param):
-        super().__init__()
-        self.url = url
-        self.param = param
-
-    def __str__(self):
-        return f"Bad param '{self.param}' passed into url: '{self.url}'"
-
-
-# pylint: disable=too-few-public-methods
-class KeyAuth(AuthBase):
-    def __init__(self, key):
-        self.key = key
-
-    def __call__(self, r):
-        r.headers["Authorization"] = f"Api-Key {self.key}"
-        return r
-
-
-class ApcUrl:
-    """A safe abstraction and testable for filling in parameters in
-    api_client.cfg"""
-
-    def __init__(self, base_url):
-        self.base_url = base_url
-
-    def params(self, **params):
-        temp_url = self.base_url
-        for k in params:
-            wrapped_param = "{" + k + "}"
-            if wrapped_param not in temp_url:
-                raise UrlBadParam(self.base_url, k)
-        temp_url = temp_url.format_map(UrlParamDict(**params))
-        return ApcUrl(temp_url)
-
-    def url(self):
-        if "{" in self.base_url:
-            raise IncompleteUrl(self.base_url)
-        return self.base_url
-
-
-class ApiRequest:
-    API_HTTP_REQUEST_TIMEOUT = 30  # 30 second HTTP request timeout
-
-    def __init__(self, name, url, logger=None, api_key=None):
-        self.name = name
-        self.url = url
-        self.__req = None
-        if logger is None:
-            self.logger = logging
-        else:
-            self.logger = logger
-        self.auth = KeyAuth(api_key)
-
-    def __call__(self, *, _post_data=None, _put_data=None, params=None, **kwargs):
-        final_url = self.url.params(**kwargs).url()
-        self.logger.debug(final_url)
-        try:
-            if _post_data is not None:
-                res = requests.post(
-                    final_url,
-                    data=_post_data,
-                    auth=self.auth,
-                    timeout=ApiRequest.API_HTTP_REQUEST_TIMEOUT,
-                )
-            elif _put_data is not None:
-                res = requests.put(
-                    final_url,
-                    data=_put_data,
-                    auth=self.auth,
-                    timeout=ApiRequest.API_HTTP_REQUEST_TIMEOUT,
-                )
-            else:
-                res = requests.get(
-                    final_url,
-                    params=params,
-                    auth=self.auth,
-                    timeout=ApiRequest.API_HTTP_REQUEST_TIMEOUT,
-                )
-
-            # Check for bad HTTP status code
-            res.raise_for_status()
-
-            if "application/json" in res.headers["content-type"]:
-                return res.json()
-            return res
-        except requests.exceptions.Timeout:
-            self.logger.error("HTTP request to %s timed out", final_url)
-            raise
-        except requests.exceptions.HTTPError:
-            self.logger.error(
-                f"{res.request.method} {res.request.url} request failed '{res.status_code}':"
-                f"\nPayload: {res.request.body}"
-                f"\nResponse: {res.text}"
-            )
-            raise
-
-    def req(self, *args, **kwargs):
-        self.__req = lambda: self(*args, **kwargs)
-        return self
-
-    def retry(self, count, delay=5):
-        """Try to send request n times. If after n times it fails then
-        we finally raise exception"""
-        for _ in range(0, count - 1):
-            try:
-                return self.__req()
-            except requests.exceptions.RequestException:
-                sleep(delay)
-        return self.__req()
-
-
-class RequestProvider:
-    """
-    Creates the available ApiRequest instance
-    """
-
-    def __init__(self, base_url: str, api_key: str, endpoints: dict):
-        self.requests = {}
-        self.url = ApcUrl(base_url + "/{action}")
-
-        # Now we must discover the possible actions
-        for action_name, action_value in endpoints.items():
-            new_url = self.url.params(action=action_value)
-            if "{api_key}" in action_value:
-                new_url = new_url.params(api_key=api_key)
-            self.requests[action_name] = ApiRequest(
-                action_name, new_url, api_key=api_key
-            )
-
-    def available_requests(self):
-        return list(self.requests.keys())
-
-    def __contains__(self, request):
-        return request in self.requests
-
-    def __getattr__(self, attr):
-        if attr in self:
-            return self.requests[attr]
-
-        return super().__getattribute__(attr)

api-client/libretime_api_client/v1.py

@@ -1,124 +1,150 @@
 import json
 import logging
-import time
+from functools import wraps
-import urllib.parse
+from time import sleep
 
-import requests
+from requests.exceptions import RequestException
-from libretime_shared.config import BaseConfig, GeneralConfig
 
-from ._utils import ApiRequest, RequestProvider
+from ._client import AbstractApiClient, Response
+
+logger = logging.getLogger(__name__)
 
 
-class Config(BaseConfig):
+def retry_decorator(max_retries: int = 5):
-    general: GeneralConfig
+    def retry_request(func):
+        @wraps(func)
+        def wrapper(*args, **kwargs):
+            retries = max_retries
+            while True:
+                try:
+                    return func(*args, **kwargs)
+                except RequestException as exception:
+                    logger.warning(exception)
+
+                    retries -= 1
+                    if retries <= 0:
+                        break
+
+                    sleep(2.0)
+
+            return None
+
+        return wrapper
+
+    return retry_request
 
 
-AIRTIME_API_VERSION = "1.1"
+class BaseApiClient(AbstractApiClient):
+    def __init__(self, base_url: str, api_key: str):
+        super().__init__(base_url=base_url)
+        self.session.headers.update({"Authorization": f"Api-Key {api_key}"})
+        self.session.params.update({"format": "json"})  # type: ignore[union-attr]
 
+    def version(self, **kwargs) -> Response:
+        return self._request(
+            "GET",
+            "/api/version",
+            **kwargs,
+        )
 
-api_endpoints = {}
+    def register_component(self, component: str, **kwargs) -> Response:
+        return self._request(
+            "GET",
+            "/api/register-component",
+            params={"component": component},
+            **kwargs,
+        )
 
-# URL to get the version number of the server API
+    def notify_media_item_start_play(self, media_id, **kwargs) -> Response:
-api_endpoints["version_url"] = "version/api_key/{api_key}"
+        return self._request(
-# URL to register a components IP Address with the central web server
+            "GET",
-api_endpoints[
+            "/api/notify-media-item-start-play",
-    "register_component"
+            params={"media_id": media_id},
-] = "register-component/format/json/api_key/{api_key}/component/{component}"
+            **kwargs,
+        )
 
-# media-monitor
+    def update_liquidsoap_status(self, msg, stream_id, boot_time, **kwargs) -> Response:
-api_endpoints[
+        return self._request(
-    "upload_recorded"
+            "POST",
-] = "upload-recorded/format/json/api_key/{api_key}/fileid/{fileid}/showinstanceid/{showinstanceid}"
+            "/api/update-liquidsoap-status",
-# show-recorder
+            params={"stream_id": stream_id, "boot_time": boot_time},
-api_endpoints["show_schedule_url"] = "recorded-shows/format/json/api_key/{api_key}"
+            data={"msg_post": msg},
-api_endpoints["upload_file_url"] = "rest/media"
+            **kwargs,
-# pypo
+        )
-api_endpoints[
+
-    "update_start_playing_url"
+    def update_source_status(self, sourcename, status, **kwargs) -> Response:
-] = "notify-media-item-start-play/api_key/{api_key}/media_id/{media_id}/"
+        return self._request(
-api_endpoints[
+            "GET",
-    "get_stream_setting"
+            "/api/update-source-status",
-] = "get-stream-setting/format/json/api_key/{api_key}/"
+            params={"sourcename": sourcename, "status": status},
-api_endpoints[
+            **kwargs,
-    "update_liquidsoap_status"
+        )
-] = "update-liquidsoap-status/format/json/api_key/{api_key}/msg/{msg}/stream_id/{stream_id}/boot_time/{boot_time}"
+
-api_endpoints[
+    def check_live_stream_auth(self, username, password, djtype, **kwargs) -> Response:
-    "update_source_status"
+        return self._request(
-] = "update-source-status/format/json/api_key/{api_key}/sourcename/{sourcename}/status/{status}"
+            "GET",
-api_endpoints[
+            "/api/check-live-stream-auth",
-    "check_live_stream_auth"
+            params={"username": username, "password": password, "djtype": djtype},
-] = "check-live-stream-auth/format/json/api_key/{api_key}/username/{username}/password/{password}/djtype/{djtype}"
+            **kwargs,
-api_endpoints["get_bootstrap_info"] = "get-bootstrap-info/format/json/api_key/{api_key}"
+        )
-api_endpoints[
+
-    "notify_webstream_data"
+    def notify_webstream_data(self, media_id, data, **kwargs) -> Response:
-] = "notify-webstream-data/api_key/{api_key}/media_id/{media_id}/format/json"
+        return self._request(
-api_endpoints[
+            "POST",
-    "notify_liquidsoap_started"
+            "/api/notify-webstream-data",
-] = "rabbitmq-do-push/api_key/{api_key}/format/json"
+            params={"media_id": media_id},
-api_endpoints[
+            data={"data": data},  # Data is already a json formatted string
-    "get_stream_parameters"
+            **kwargs,
-] = "get-stream-parameters/api_key/{api_key}/format/json"
+        )
-api_endpoints["push_stream_stats"] = "push-stream-stats/api_key/{api_key}/format/json"
+
-api_endpoints[
+    def rabbitmq_do_push(self, **kwargs) -> Response:
-    "update_stream_setting_table"
+        return self._request(
-] = "update-stream-setting-table/api_key/{api_key}/format/json"
+            "GET",
-api_endpoints[
+            "/api/rabbitmq-do-push",
-    "update_metadata_on_tunein"
+            **kwargs,
-] = "update-metadata-on-tunein/api_key/{api_key}"
+        )
+
+    def push_stream_stats(self, data, **kwargs) -> Response:
+        return self._request(
+            "POST",
+            "/api/push-stream-stats",
+            data={"data": json.dumps(data)},
+            **kwargs,
+        )
+
+    def update_stream_setting_table(self, data, **kwargs) -> Response:
+        return self._request(
+            "POST",
+            "/api/update-stream-setting-table",
+            data={"data": json.dumps(data)},
+            **kwargs,
+        )
+
+    def update_metadata_on_tunein(self, **kwargs) -> Response:
+        return self._request(
+            "GET",
+            "/api/update-metadata-on-tunein",
+            **kwargs,
+        )
 
 
 class ApiClient:
-    API_BASE = "/api"
+    def __init__(self, base_url: str, api_key: str):
-    UPLOAD_RETRIES = 3
+        self._base_client = BaseApiClient(base_url=base_url, api_key=api_key)
-    UPLOAD_WAIT = 60
 
-    def __init__(self, logger=None, config_path="/etc/libretime/config.yml"):
+    def version(self):
-        self.logger = logger or logging
-
-        config = Config(config_path)
-        self.base_url = config.general.public_url
-        self.api_key = config.general.api_key
-
-        self.services = RequestProvider(
-            base_url=self.base_url + self.API_BASE,
-            api_key=self.api_key,
-            endpoints=api_endpoints,
-        )
-
-    def __get_api_version(self):
         try:
-            return self.services.version_url()["api_version"]
+            resp = self._base_client.version()
-        except Exception as exception:
+            payload = resp.json()
-            self.logger.exception(exception)
+            return payload["api_version"]
+        except RequestException:
             return -1
 
-    def is_server_compatible(self, verbose=True):
-        api_version = self.__get_api_version()
-        if api_version == -1:
-            if verbose:
-                self.logger.info("Unable to get Airtime API version number.\n")
-            return False
-
-        if api_version[0:3] != AIRTIME_API_VERSION[0:3]:
-            if verbose:
-                self.logger.info("Airtime API version found: " + str(api_version))
-                self.logger.info(
-                    "pypo is only compatible with API version: " + AIRTIME_API_VERSION
-                )
-            return False
-
-        if verbose:
-            self.logger.info("Airtime API version found: " + str(api_version))
-            self.logger.info(
-                "pypo is only compatible with API version: " + AIRTIME_API_VERSION
-            )
-        return True
-
     def notify_liquidsoap_started(self):
         try:
-            self.services.notify_liquidsoap_started()
+            self._base_client.rabbitmq_do_push()
-        except Exception as exception:
+        except RequestException:
-            self.logger.exception(exception)
+            pass
 
     def notify_media_item_start_playing(self, media_id):
         """
@@ -127,96 +153,20 @@ class ApiClient:
         which we handed to liquidsoap in get_liquidsoap_data().
         """
         try:
-            return self.services.update_start_playing_url(media_id=media_id)
+            return self._base_client.notify_media_item_start_play(media_id=media_id)
-        except Exception as exception:
+        except RequestException:
-            self.logger.exception(exception)
             return None
 
-    def get_shows_to_record(self):
-        try:
-            return self.services.show_schedule_url()
-        except Exception as exception:
-            self.logger.exception(exception)
-            return None
-
-    def upload_recorded_show(self, files, show_id):
-        response = ""
-
-        retries = self.UPLOAD_RETRIES
-        retries_wait = self.UPLOAD_WAIT
-
-        url = self.construct_rest_url("upload_file_url")
-
-        self.logger.debug(url)
-
-        for i in range(0, retries):
-            self.logger.debug("Upload attempt: %s", i + 1)
-            self.logger.debug(files)
-            self.logger.debug(ApiRequest.API_HTTP_REQUEST_TIMEOUT)
-
-            try:
-                request = requests.post(
-                    url, files=files, timeout=float(ApiRequest.API_HTTP_REQUEST_TIMEOUT)
-                )
-                response = request.json()
-                self.logger.debug(response)
-
-                # FIXME: We need to tell LibreTime that the uploaded track was recorded
-                # for a specific show
-                #
-                # My issue here is that response does not yet have an id. The id gets
-                # generated at the point where analyzer is done with it's work. We
-                # probably need to do what is below in analyzer and also make sure that
-                # the show instance id is routed all the way through.
-                #
-                # It already gets uploaded by this but the RestController does not seem
-                # to care about it. In the end analyzer doesn't have the info in it's
-                # rabbitmq message and imports the show as a regular track.
-                #
-                # logger.info("uploaded show result as file id %s", response.id)
-                #
-                # url = self.construct_url("upload_recorded") url =
-                # url.replace('%%fileid%%', response.id) url =
-                # url.replace('%%showinstanceid%%', show_id) request.get(url)
-                # logger.info("associated uploaded file %s with show instance %s",
-                # response.id, show_id)
-                break
-
-            except requests.exceptions.HTTPError as exception:
-                self.logger.error(f"Http error code: {exception.response.status_code}")
-                self.logger.exception(exception)
-
-            except requests.exceptions.ConnectionError as exception:
-                self.logger.exception(f"Server is down: {exception}")
-
-            except Exception as exception:
-                self.logger.exception(exception)
-
-            # wait some time before next retry
-            time.sleep(retries_wait)
-
-        return response
-
     def check_live_stream_auth(self, username, password, dj_type):
         try:
-            return self.services.check_live_stream_auth(
+            return self._base_client.check_live_stream_auth(
-                username=username, password=password, djtype=dj_type
+                username=username,
+                password=password,
+                djtype=dj_type,
             )
-        except Exception as exception:
+        except RequestException:
-            self.logger.exception(exception)
             return {}
 
-    def construct_rest_url(self, action_key):
-        """
-        Constructs the base url for RESTful requests
-        """
-        url = urllib.parse.urlsplit(self.base_url)
-        url.username = self.api_key
-        return f"{url.geturl()}/{api_endpoints[action_key]}"
-
-    def get_stream_setting(self):
-        return self.services.get_stream_setting()
-
     def register_component(self, component):
         """
         Purpose of this method is to contact the server with a "Hey its
@@ -225,71 +175,45 @@ class ApiClient:
         to query monit via monit's http service, or download log files via a
         http server.
         """
-        return self.services.register_component(component=component)
+        return self._base_client.register_component(component=component)
 
+    @retry_decorator()
     def notify_liquidsoap_status(self, msg, stream_id, time):
-        try:
+        self._base_client.update_liquidsoap_status(
-            # encoded_msg is no longer used server_side!!
+            msg=msg,
-            encoded_msg = urllib.parse.quote("dummy")
-
-            self.services.update_liquidsoap_status.req(
-                _post_data={"msg_post": msg},
-                msg=encoded_msg,
             stream_id=stream_id,
             boot_time=time,
-            ).retry(5)
+        )
-        except Exception as exception:
-            self.logger.exception(exception)
 
+    @retry_decorator()
     def notify_source_status(self, sourcename, status):
-        try:
+        return self._base_client.update_source_status(
-            return self.services.update_source_status.req(
+            sourcename=sourcename,
-                sourcename=sourcename, status=status
+            status=status,
-            ).retry(5)
+        )
-        except Exception as exception:
-            self.logger.exception(exception)
-
-    def get_bootstrap_info(self):
-        """
-        Retrieve infomations needed on bootstrap time.
-        """
-        return self.services.get_bootstrap_info()
 
+    @retry_decorator()
     def notify_webstream_data(self, data, media_id):
         """
         Update the server with the latest metadata we've received from the
         external webstream
         """
-        self.logger.info(
+        return self._base_client.notify_webstream_data(
-            self.services.notify_webstream_data.req(
+            data=data,
-                _post_data={"data": data}, media_id=str(media_id)
+            media_id=str(media_id),
-            ).retry(5)
         )
 
-    def get_stream_parameters(self):
-        response = self.services.get_stream_parameters()
-        self.logger.debug(response)
-        return response
-
     def push_stream_stats(self, data):
-        # TODO : users of this method should do their own error handling
+        return self._base_client.push_stream_stats(data=data)
-        response = self.services.push_stream_stats(
-            _post_data={"data": json.dumps(data)}
-        )
-        return response
 
     def update_stream_setting_table(self, data):
         try:
-            response = self.services.update_stream_setting_table(
+            return self._base_client.update_stream_setting_table(data=data)
-                _post_data={"data": json.dumps(data)}
+        except RequestException:
-            )
+            return None
-            return response
-        except Exception as exception:
-            self.logger.exception(exception)
 
     def update_metadata_on_tunein(self):
-        self.services.update_metadata_on_tunein()
+        self._base_client.update_metadata_on_tunein()
 
-
+    def trigger_task_manager(self):
-class InvalidContentType(Exception):
+        self._base_client.version()
-    pass

api-client/libretime_api_client/v2.py

@@ -1,11 +1,14 @@
-from ._client import AbstractApiClient, Response
+from ._client import AbstractApiClient, Response, default_retry
 
 
 class ApiClient(AbstractApiClient):
     VERSION = "2.0"
 
     def __init__(self, base_url: str, api_key: str):
-        super().__init__(base_url=base_url)
+        super().__init__(
+            base_url=base_url,
+            retry=default_retry(),
+        )
         self.session.headers.update({"Authorization": f"Api-Key {api_key}"})
 
     def get_info(self, **kwargs) -> Response:

api-client/pyproject.toml

@@ -1,3 +1,8 @@
+[tool.isort]
+profile = "black"
+combine_as_imports = true
+known_first_party = ["libretime_api_client"]
+
 [tool.pylint.messages_control]
 extension-pkg-whitelist = "pydantic"
 disable = [
@@ -6,13 +11,13 @@ disable = [
   "missing-module-docstring",
 ]
 
-[tool.pylint.format]
-disable = "logging-fstring-interpolation"
-
 [tool.pytest.ini_options]
 log_cli = true
 log_cli_level = "DEBUG"
 
+[tool.coverage.run]
+source = ["libretime_api_client"]
+
 [build-system]
 requires = ["setuptools", "wheel"]
 build-backend = "setuptools.build_meta"

api-client/requirements.txt

@@ -1,4 +1,4 @@
 # Please do not edit this file, edit the setup.py file!
 # This file is auto-generated by tools/extract_requirements.py.
-python-dateutil>=2.8.1,<2.9
+python-dateutil>=2.8.1,<2.10
-requests>=2.25.1,<2.29
+requests>=2.32.2,<2.33

api-client/setup.py

@@ -1,8 +1,10 @@
 from setuptools import find_packages, setup
 
+version = "4.2.0"  # x-release-please-version
+
 setup(
     name="libretime-api-client",
-    version="3.0.0-beta.2",
+    version=version,
     description="LibreTime API Client",
     author="LibreTime Contributors",
     url="https://github.com/libretime/libretime",
@@ -14,16 +16,16 @@ setup(
     license="AGPLv3",
     packages=find_packages(exclude=["*tests*", "*fixtures*"]),
     package_data={"": ["py.typed"]},
-    python_requires=">=3.6",
+    python_requires=">=3.8",
     install_requires=[
-        "python-dateutil>=2.8.1,<2.9",
+        "python-dateutil>=2.8.1,<2.10",
-        "requests>=2.25.1,<2.29",
+        "requests>=2.32.2,<2.33",
     ],
     extras_require={
         "dev": [
-            "requests-mock",
+            "requests-mock>=1.10.0,<2",
-            "types-python-dateutil",
+            "types-python-dateutil>=2.8.1,<3",
-            "types-requests",
+            "types-requests>=2.31.0,<3",
         ],
     },
     zip_safe=False,

api-client/tests/utils_test.py

@@ -1,95 +0,0 @@
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from libretime_api_client._utils import (
-    ApcUrl,
-    ApiRequest,
-    IncompleteUrl,
-    RequestProvider,
-    UrlBadParam,
-)
-
-
-@pytest.mark.parametrize(
-    "url, params, expected",
-    [
-        ("one/two/three", {}, "one/two/three"),
-        ("/testing/{key}", {"key": "aaa"}, "/testing/aaa"),
-        (
-            "/more/{key_a}/{key_b}/testing",
-            {"key_a": "aaa", "key_b": "bbb"},
-            "/more/aaa/bbb/testing",
-        ),
-    ],
-)
-def test_apc_url(url: str, params: dict, expected: str):
-    found = ApcUrl(url)
-    assert found.base_url == url
-    assert found.params(**params).url() == expected
-
-
-def test_apc_url_bad_param():
-    url = ApcUrl("/testing/{key}")
-    with pytest.raises(UrlBadParam):
-        url.params(bad_key="testing")
-
-
-def test_apc_url_incomplete():
-    url = ApcUrl("/{one}/{two}/three").params(two="testing")
-    with pytest.raises(IncompleteUrl):
-        url.url()
-
-
-def test_api_request_init():
-    req = ApiRequest("request_name", ApcUrl("/test/ing"))
-    assert req.name == "request_name"
-
-
-def test_api_request_call_json():
-    return_value = {"ok": "ok"}
-
-    read = MagicMock()
-    read.headers = {"content-type": "application/json"}
-    read.json = MagicMock(return_value=return_value)
-
-    with patch("requests.get") as mock_method:
-        mock_method.return_value = read
-        request = ApiRequest("mm", ApcUrl("http://localhost/testing"))()
-        assert request == return_value
-
-
-def test_api_request_call_html():
-    return_value = "<html><head></head><body></body></html>"
-
-    read = MagicMock()
-    read.headers = {"content-type": "application/html"}
-    read.text = MagicMock(return_value=return_value)
-
-    with patch("requests.get") as mock_method:
-        mock_method.return_value = read
-        request = ApiRequest("mm", ApcUrl("http://localhost/testing"))()
-        assert request.text() == return_value
-
-
-def test_request_provider_init():
-    request_provider = RequestProvider(
-        base_url="http://localhost/test",
-        api_key="test_key",
-        endpoints={},
-    )
-    assert len(request_provider.available_requests()) == 0
-
-
-def test_request_provider_contains():
-    endpoints = {
-        "upload_recorded": "/1/",
-    }
-    request_provider = RequestProvider(
-        base_url="http://localhost/test",
-        api_key="test_key",
-        endpoints=endpoints,
-    )
-
-    for endpoint in endpoints:
-        assert endpoint in request_provider.requests

api-client/tests/v1_test.py

@@ -0,0 +1,21 @@
+import pytest
+
+from libretime_api_client.v1 import ApiClient
+
+
+@pytest.mark.parametrize(
+    "base_url",
+    [
+        ("http://localhost:8080"),
+        ("http://localhost:8080/base"),
+    ],
+)
+def test_api_client(requests_mock, base_url):
+    api_client = ApiClient(base_url=base_url, api_key="test-key")
+
+    requests_mock.get(
+        f"{base_url}/api/version",
+        json={"api_version": "1.0.0"},
+    )
+
+    assert api_client.version() == "1.0.0"

api/Makefile

@@ -4,32 +4,22 @@ include ../tools/python.mk
 
 PIP_INSTALL := \
 	--editable ../shared \
-	--editable .[dev]
+	--editable .[dev,sentry]
 PYLINT_ARG := libretime_api
 MYPY_ARG := libretime_api
 BANDIT_ARG := --exclude '*/tests/*' libretime_api || true
 
+export DJANGO_SETTINGS_MODULE=libretime_api.settings.testing
+
 format: .format
 lint: .format-check .pylint .mypy .bandit
+test: .pytest
+test-coverage: .coverage
 clean: .clean
 
-test: $(VENV)
-	source $(VENV)/bin/activate
-	export DJANGO_SETTINGS_MODULE=libretime_api.settings.testing
-	pytest -v \
-		--numprocesses=$(CPU_CORES) \
-		--color=yes \
-		--cov-config=pyproject.toml \
-		--cov-report=term \
-		--cov-report=xml:./coverage.xml \
-		--cov=libretime_api \
-		libretime_api
-
 SCHEMA_FILE ?= schema.yml
 schema: $(VENV)
-	source $(VENV)/bin/activate
+	$(VENV)/bin/libretime-api spectacular --file $(SCHEMA_FILE)
-	export DJANGO_SETTINGS_MODULE=libretime_api.settings.testing
-	libretime-api spectacular --file $(SCHEMA_FILE)
 	if command -v npx > /dev/null; then npx prettier --write $(SCHEMA_FILE); fi
 
 schema-foreach-commit:

api/install/systemd/libretime-api.service

@@ -11,7 +11,6 @@ PrivateTmp=true
 PrivateUsers=true
 ProtectClock=true
 ProtectControlGroups=true
-ProtectHome=true
 ProtectHostname=true
 ProtectKernelLogs=true
 ProtectKernelModules=true
@@ -19,14 +18,15 @@ ProtectKernelTunables=true
 ProtectProc=invisible
 ProtectSystem=full
 
+Environment=PYTHONOPTIMIZE=2
 Environment=LIBRETIME_CONFIG_FILEPATH=@@CONFIG_FILEPATH@@
 Environment=LIBRETIME_LOG_FILEPATH=@@LOG_DIR@@/api.log
 
 Type=notify
 KillMode=mixed
-ExecStart=/usr/bin/gunicorn \
+ExecStart=@@VENV_DIR@@/bin/gunicorn \
         --workers 4 \
-        --worker-class uvicorn.workers.UvicornWorker \
+        --worker-class libretime_api.gunicorn.Worker \
         --log-file - \
         --bind unix:/run/libretime-api.sock \
         libretime_api.asgi

api/libretime_api/__init__.py

@@ -0,0 +1,4 @@
+from importlib.metadata import version as get_version
+
+PACKAGE = __name__
+VERSION = get_version(__name__)

api/libretime_api/core/models/preference.py

@@ -18,12 +18,13 @@ class StreamPreferences(BaseModel):
     input_fade_transition: float
     message_format: MessageFormatKind
     message_offline: str
+    replay_gain_enabled: bool
+    replay_gain_offset: float
+
     # input_auto_switch_off: bool
     # input_auto_switch_on: bool
     # input_main_user: str
     # input_main_password: str
-    # replay_gain_enabled: bool
-    # replay_gain_offset: float
     # track_fade_in: float
     # track_fade_out: float
     # track_fade_transition: float
@@ -78,8 +79,12 @@ class Preference(models.Model):
         entries = dict(cls.site.values_list("key", "value"))
         return StreamPreferences(
             input_fade_transition=float(entries.get("default_transition_fade") or 0.0),
-            message_format=int(entries.get("stream_label_format") or 0),
+            message_format=MessageFormatKind(
+                int(entries.get("stream_label_format") or 0)
+            ),
             message_offline=entries.get("off_air_meta") or "Offline",
+            replay_gain_enabled=entries.get("enable_replay_gain") == "1",
+            replay_gain_offset=float(entries.get("replay_gain_modifier") or 0.0),
         )
 
     @classmethod

api/libretime_api/core/models/user.py

@@ -8,6 +8,7 @@ from .role import Role
 
 
 class UserManager(BaseUserManager):
+    # pylint: disable=too-many-positional-arguments
     def create_user(self, role, username, password, email, first_name, last_name):
         user = self.model(
             role=role,
@@ -20,6 +21,7 @@ class UserManager(BaseUserManager):
         user.save(using=self._db)
         return user
 
+    # pylint: disable=too-many-positional-arguments
     def create_superuser(self, username, password, email, first_name, last_name):
         return self.create_user(
             Role.ADMIN,

api/libretime_api/core/serializers/stream.py

@@ -6,6 +6,8 @@ class StreamPreferencesSerializer(serializers.Serializer):
     input_fade_transition = serializers.FloatField(read_only=True)
     message_format = serializers.IntegerField(read_only=True)
     message_offline = serializers.CharField(read_only=True)
+    replay_gain_enabled = serializers.BooleanField(read_only=True)
+    replay_gain_offset = serializers.FloatField(read_only=True)
 
 
 # pylint: disable=abstract-method

api/libretime_api/core/tests/models/test_preference.py

@@ -4,7 +4,7 @@ from libretime_api.core.models.preference import Preference
 # pylint: disable=invalid-name,unused-argument
 def test_preference_get_site_preferences(db):
     result = Preference.get_site_preferences()
-    assert result.dict() == {
+    assert result.model_dump() == {
         "station_name": "LibreTime",
     }
 
@@ -12,17 +12,19 @@ def test_preference_get_site_preferences(db):
 # pylint: disable=invalid-name,unused-argument
 def test_preference_get_stream_preferences(db):
     result = Preference.get_stream_preferences()
-    assert result.dict() == {
+    assert result.model_dump() == {
         "input_fade_transition": 0.0,
         "message_format": 0,
         "message_offline": "LibreTime - offline",
+        "replay_gain_enabled": True,
+        "replay_gain_offset": 0.0,
     }
 
 
 # pylint: disable=invalid-name,unused-argument
 def test_preference_get_stream_state(db):
     result = Preference.get_stream_state()
-    assert result.dict() == {
+    assert result.model_dump() == {
         "input_main_connected": False,
         "input_main_streaming": False,
         "input_show_connected": False,

api/libretime_api/core/tests/views/test_stream.py

@@ -9,6 +9,8 @@ def test_stream_preferences_get(db, api_client: APIClient):
         "input_fade_transition": 0.0,
         "message_format": 0,
         "message_offline": "LibreTime - offline",
+        "replay_gain_enabled": True,
+        "replay_gain_offset": 0.0,
     }
 
 

api/libretime_api/core/views/info.py

@@ -22,7 +22,7 @@ class InfoView(APIView):
     def get(self, request):
         data = Preference.get_site_preferences()
         return Response(
-            data.dict(
+            data.model_dump(
                 include={
                     "station_name",
                 }

api/libretime_api/core/views/stream.py

@@ -14,11 +14,13 @@ class StreamPreferencesView(views.APIView):
     def get(self, request):
         data = Preference.get_stream_preferences()
         return Response(
-            data.dict(
+            data.model_dump(
                 include={
                     "input_fade_transition",
                     "message_format",
                     "message_offline",
+                    "replay_gain_enabled",
+                    "replay_gain_offset",
                 }
             )
         )
@@ -32,7 +34,7 @@ class StreamStateView(views.APIView):
     def get(self, request):
         data = Preference.get_stream_state()
         return Response(
-            data.dict(
+            data.model_dump(
                 include={
                     "input_main_connected",
                     "input_main_streaming",

api/libretime_api/gunicorn.py

@@ -0,0 +1,5 @@
+from uvicorn.workers import UvicornWorker  # pylint: disable=import-error
+
+
+class Worker(UvicornWorker):
+    CONFIG_KWARGS = {"lifespan": "off"}

api/libretime_api/legacy/migrations/0006_2_5_5.py

@@ -8,8 +8,6 @@ UP = """
 -- DELETE FROM cc_pref WHERE keystr = 'system_version';
 -- INSERT INTO cc_pref (keystr, valstr) VALUES ('system_version', '2.5.5');
 
-ALTER TABLE cc_show ADD COLUMN image_path varchar(255) DEFAULT '';
-ALTER TABLE cc_show_instances ADD COLUMN description varchar(255) DEFAULT '';
 """
 
 DOWN = None

api/libretime_api/legacy/migrations/0015_2_5_17.py

@@ -5,7 +5,7 @@ from django.db import migrations
 from ._migrations import legacy_migration_factory
 
 UP = """
-ALTER TABLE cc_files ADD COLUMN artwork TYPE character varying(255);
+ALTER TABLE cc_files ADD COLUMN artwork VARCHAR(255);
 """
 
 DOWN = None

api/libretime_api/legacy/migrations/0023_3_0_0_alpha_9_1.py

@@ -4,12 +4,18 @@ from django.db import migrations
 
 from ._migrations import legacy_migration_factory
 
+# This migration is currently a placeholder for 3.0.0-alpha.9.1.
+# Please do not remove it.  There are currently no actions, but it
+# needs to remain intact so it does not fail when called from the
+# migrations script. Any future migrations that may apply to
+# 3.0.0-alpha.9.1 will be added to this file.
+
 UP = """
-ALTER TABLE cc_files ADD COLUMN artwork VARCHAR(4096);
+
 """
 
 DOWN = """
-ALTER TABLE cc_files DROP COLUMN IF EXISTS artwork;
+
 """
 
 

api/libretime_api/legacy/migrations/0041_drop_stream_setting_table.py

@@ -0,0 +1,33 @@
+# pylint: disable=invalid-name
+
+from django.db import migrations
+
+from ._migrations import legacy_migration_factory
+
+UP = """
+drop table if exists "cc_stream_setting" cascade;
+"""
+
+DOWN = """
+create table "cc_stream_setting"
+(
+    "keyname" varchar(64) not null,
+    "value" varchar(255),
+    "type" varchar(16) not null,
+    primary key ("keyname")
+);
+"""
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("legacy", "0040_bump_legacy_schema_version"),
+    ]
+    operations = [
+        migrations.RunPython(
+            code=legacy_migration_factory(
+                target="41",
+                sql=UP,
+            )
+        )
+    ]

api/libretime_api/legacy/migrations/0042_remove_stream_preferences.py

@@ -0,0 +1,46 @@
+# pylint: disable=invalid-name
+
+from django.db import migrations
+
+from ._migrations import legacy_migration_factory
+
+UP = """
+delete from cc_pref
+where "keystr" in (
+    'default_icecast_password',
+    'default_stream_mount_point',
+    'live_dj_connection_url_override',
+    'live_dj_source_connection_url',
+    'master_dj_connection_url_override',
+    'master_dj_source_connection_url',
+    'max_bitrate',
+    'num_of_streams',
+    'stream_bitrate',
+    'stream_type'
+);
+"""
+
+DOWN = """
+insert into
+    cc_pref ("keystr", "valstr")
+values
+    ('default_stream_mount_point', 'main'),
+    ('max_bitrate', '320'),
+    ('num_of_streams', '3'),
+    ('stream_bitrate', '24, 32, 48, 64, 96, 128, 160, 192, 224, 256, 320'),
+    ('stream_type', 'ogg, mp3, opus, aac');
+"""
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("legacy", "0041_drop_stream_setting_table"),
+    ]
+    operations = [
+        migrations.RunPython(
+            code=legacy_migration_factory(
+                target="42",
+                sql=UP,
+            )
+        )
+    ]

api/libretime_api/legacy/migrations/0043_remove_cors_preference.py

@@ -0,0 +1,26 @@
+# pylint: disable=invalid-name
+
+from django.db import migrations
+
+from ._migrations import legacy_migration_factory
+
+UP = """
+delete from cc_pref
+where "keystr" = 'allowed_cors_urls';
+"""
+
+DOWN = """"""
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("legacy", "0042_remove_stream_preferences"),
+    ]
+    operations = [
+        migrations.RunPython(
+            code=legacy_migration_factory(
+                target="43",
+                sql=UP,
+            )
+        )
+    ]

api/libretime_api/legacy/migrations/0044_add_track_types_analyzer_options.py

@@ -0,0 +1,29 @@
+# pylint: disable=invalid-name
+
+from django.db import migrations
+
+from ._migrations import legacy_migration_factory
+
+UP = """
+alter table "cc_track_types" add column "analyze_cue_points" boolean default 'f' not null;
+
+update "cc_track_types" set "analyze_cue_points" = 't';
+"""
+
+DOWN = """
+alter table "cc_track_types" drop column if exists "analyze_cue_points";
+"""
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("legacy", "0043_remove_cors_preference"),
+    ]
+    operations = [
+        migrations.RunPython(
+            code=legacy_migration_factory(
+                target="44",
+                sql=UP,
+            )
+        )
+    ]

api/libretime_api/legacy/migrations/0045_add_sessions_table.py

@@ -0,0 +1,34 @@
+# pylint: disable=invalid-name
+
+from django.db import migrations
+
+from ._migrations import legacy_migration_factory
+
+UP = """
+CREATE TABLE "sessions"
+(
+    "id" CHAR(32) NOT NULL,
+    "modified" INTEGER,
+    "lifetime" INTEGER,
+    "data" TEXT,
+    PRIMARY KEY ("id")
+);
+"""
+
+DOWN = """
+DROP TABLE IF EXISTS "sessions" CASCADE;
+"""
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("legacy", "0044_add_track_types_analyzer_options"),
+    ]
+    operations = [
+        migrations.RunPython(
+            code=legacy_migration_factory(
+                target="45",
+                sql=UP,
+            )
+        )
+    ]

api/libretime_api/legacy/migrations/0046_add_override_intro_outro_playlists.py

@@ -0,0 +1,37 @@
+# pylint: disable=invalid-name
+
+from django.db import migrations
+
+from ._migrations import legacy_migration_factory
+
+UP = """
+ALTER TABLE cc_show ADD COLUMN override_intro_playlist boolean default 'f' NOT NULL;
+ALTER TABLE cc_show ADD COLUMN intro_playlist_id integer DEFAULT NULL;
+ALTER TABLE cc_show ADD CONSTRAINT cc_playlist_intro_playlist_fkey FOREIGN KEY (intro_playlist_id) REFERENCES cc_playlist (id) ON DELETE SET NULL;
+ALTER TABLE cc_show ADD COLUMN override_outro_playlist boolean default 'f' NOT NULL;
+ALTER TABLE cc_show ADD COLUMN outro_playlist_id integer DEFAULT NULL;
+ALTER TABLE cc_show ADD CONSTRAINT cc_playlist_outro_playlist_fkey FOREIGN KEY (outro_playlist_id) REFERENCES cc_playlist (id) ON DELETE SET NULL;
+"""
+
+DOWN = """
+ALTER TABLE cc_show DROP COLUMN IF EXISTS override_intro_playlist;
+ALTER TABLE cc_show DROP COLUMN IF EXISTS intro_playlist_id;
+ALTER TABLE cc_show DROP CONSTRAINT IF EXISTS cc_playlist_intro_playlist_fkey;
+ALTER TABLE cc_show DROP COLUMN IF EXISTS override_outro_playlist;
+ALTER TABLE cc_show DROP COLUMN IF EXISTS outro_playlist_id;
+ALTER TABLE cc_show DROP CONSTRAINT IF EXISTS cc_playlist_outro_playlist_fkey;
+"""
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("legacy", "0045_add_sessions_table"),
+    ]
+    operations = [
+        migrations.RunPython(
+            code=legacy_migration_factory(
+                target="46",
+                sql=UP,
+            )
+        )
+    ]

api/libretime_api/legacy/migrations/__init__.py

@@ -1 +1,2 @@
-LEGACY_SCHEMA_VERSION = "3.0.0-beta.0.1"
+# The schema version is defined using the migration file prefix number
+LEGACY_SCHEMA_VERSION = "46"