Congegni/sintonia
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

docs: split developer and contributor manual

Browse source
This commit is contained in:
jo 2023-03-16 23:20:20 +01:00 • committed by Kyle Robbertze
parent bb755f7a62
commit a9b7513bc0
12 changed files with 23 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/developer-manual/README.md
View file

@ -2,13 +2,8 @@
title: Developer manual
---
Welcome to the **LibreTime developer manual**, you should find guides to integrate LibreTime and tools to improve and contribute to LibreTime.
Welcome to the **LibreTime developer manual**, you should find guides to integrate LibreTime.
## Integrate LibreTime
- :construction: Work in progress
## Improve and contribute to LibreTime
- Learn about the [architecture of LibreTime](./design/architecture.md)
- Learn about the [database migrations](./design/database-migrations.md)

1
docs/developer-manual/design/_category_.yml
View file

@ -1 +0,0 @@
label: Design

108
docs/developer-manual/design/architecture.md
View file

@ -1,108 +0,0 @@
---
title: Architecture
---
This document explains the design details and goals for the architecture of LibreTime. It describes the result of a [discussion that happened on Github](https://github.com/libretime/libretime/issues/1610).
## Previous architecture
The previous architecture of LibreTime (based on AirTime) was missing a proper separation of concerns. It was build around a legacy MVC app written in PHP, and services in Python to accomplish specific tasks.
## New architecture
Below is the new architecture goal of LibreTime, with a proper separation of concerns.
```mermaid
flowchart TD
users([Users])
public([Public])
subgraph create_schedule[Create the schedule]
webapp[Web app]
subgraph core[Backend]
message_api[Message API]
api[Web API]
worker[Worker]
end
end
subgraph play_schedule[Play the schedule]
playout[Playout]
liquidsoap[[Liquidsoap]]
icecast[[Icecast]]
hls[[HLS]]
end
message_queue[[Message Queue]]
database[[Database]]
storage[[Storage]]
users --> |Edit| webapp
webapp --> api
api --> database
api --> storage
api --> message_queue
message_queue <--> worker
worker --> database
worker --> storage
message_queue <--> message_api
message_api --> database
message_queue <--> playout
playout <-. via message queue .-> message_api
playout --> |e.g. download file| api
playout <--> liquidsoap
liquidsoap --> icecast
liquidsoap --> hls
public --> webapp
public --> |Listen| icecast
public --> |Listen| hls
```
The LibreTime architecture is split into 2 main monolithic blocks `Create the schedule` and `Play the schedule`. Both blocks must be able to scale horizontally.
:::note
A microservice architecture was rejected as it won't fix or improve any aspect of LibreTime.
:::
:::warning
This document tries to focus on creating and playing a schedule, it doesn't consider features such as monitoring, logging or archiving.
:::
### Create the schedule
This block contains the following components:
- a web API,
- a worker to run background tasks,
- a message API to communicate with the `Play the schedule` block, and other services,
- a web app to interface with the users.
The web API, the worker and the message API rely on the [Django framework](https://www.djangoproject.com/) to handle database, message queue and storage access.
### Play the schedule
Since the `Play the schedule` has its own requirements in terms of logic and uptime, it's handled separately from the `Create the schedule` block. This block needs to be able to be duplicated in a high availability context.
This block contains the following components:
- a Playout app that communicates with the `Play the schedule` block to gather the schedule,
- a Liquisoap app that plays and mixes the scheduled items, and dispatch them to the delivery services,
- an Icecast server that delivers a legacy audio stream to the public,
- a HLS stream that delivers a modern audio stream to the public.
### One setup per radio station
LibreTime isn't meant to be used in a multi-tenant architecture, and an entire LibreTime installation should be dedicated to a single radio station. Previous SAAS or multi-tenant features from Airtime should be deprecated or removed.
### Separation of concerns
The `Create the schedule` block must only prepare a schedule, and the `Play the schedule` must only play that schedule. A strong separation of concerns is required between the 2 blocks to allow the `Play the schedule` block to meet its uptime requirements while not depending on the `Create the schedule` in case of a failure. Development will be simplified if both blocks share a single and properly defined protocol.

51
docs/developer-manual/design/database-migrations.md
View file

@ -1,51 +0,0 @@
# Database schema creation and migrations
The method to maintain the database schema, is to write both a migration file for already installed databases and to update a `schema.sql` file for fresh databases. On fresh installation, the database is filled with the `schema.sql` and `data.sql` files and LibreTime won't run any sql migration on top of it. Previously, when LibreTime was upgraded, the missing migrations were run using a custom php based migration tool, those migrations are now handled by Django. The missing migrations are tracked using both a `schema_version` field in the `cc_pref` table and a Django migration id.
:::note
Since LibreTime forked, the `schema_version` in the `schema.sql` was locked on `3.0.0-alpha` and all the migrations were run during the first user connection. This has been fixed during the move to the Django based migrations.
:::
Django doesn't maintain a `schema.sql` file, it applies every migrations until it reaches the targeted schema represented by the code. The legacy `schema_version` has to be tracked until we remove the Propel schema generation and let Django handle all the schema migrations. Until then Propel generate the schema and Django handle migrations from already installed databases.
:::info
The first Django migration is the initial schema creation using the `schema.sql` and `data.sql` files.
:::
```mermaid
stateDiagram-v2
state is_django_migration_applied <<choice>>
[*] --> is_django_migration_applied: Is the django migration ID in the DB ?
is_django_migration_applied --> [*]: Yes, ignoring...
state "Apply django migration" as apply_django_migration
is_django_migration_applied --> apply_django_migration: No
state apply_django_migration {
state is_legacy_migration <<choice>>
[*] --> is_legacy_migration: Is it a legacy migration ?
state "Run django migration" as run_django_migration
state "Apply changes" as run_django_migration
state "Save migration ID in DB" as run_django_migration
is_legacy_migration --> run_django_migration: No
run_legacy_migration --> run_django_migration
run_django_migration --> [*]
state is_legacy_migration_applied <<choice>>
is_legacy_migration_applied --> [*]: Yes, ignoring...
state "Run legacy migration" as run_legacy_migration
state "Apply changes" as run_legacy_migration
state "Bump legacy schema version" as run_legacy_migration
is_legacy_migration_applied --> run_legacy_migration: No
is_legacy_migration --> is_legacy_migration_applied: Yes, is the DB schema version >= legacy migration schema version ?
}
apply_django_migration --> [*]
```

1
docs/developer-manual/development/_category_.yml
View file

@ -1 +0,0 @@
label: Development

149
docs/developer-manual/development/environment.md
View file

@ -1,149 +0,0 @@
---
title: Development environment
---
## Docker-compose
To setup a docker-compose development environment, run the following commands:
```bash
# Clean and build
make clean
cp .env.dev .env
DOCKER_BUILDKIT=1 docker-compose build
# Setup
docker-compose run --rm legacy make build
docker-compose run --rm api libretime-api migrate
# Run
docker-compose up -d
docker-compose logs -f
```
## Vagrant
To use Vagrant, you need to install a virtualization engine: [VirtualBox](https://www.virtualbox.org) or Libvirt. The [vagrant-vbguest] package on Github can help maintain guest extensions on host systems using VirtualBox.
:::tip
If you try run a libvirt provided box after using a VirtualBox one, you will receive an
error:
```
Error while activating network:
Call to virNetworkCreate failed: internal error: Network is already in use by interface vboxnet0.
```
This is fixed by stopping virtualbox and re-creating the vagrant box:
```bash
sudo systemctl stop virtualbox
vagrant destroy bionic
vagrant up bionic --provider=libvirt
```
:::
### Installing Libvirt
On Debian and Ubuntu:
1. Install Vagrant
```bash
sudo apt install vagrant vagrant-libvirt libvirt-daemon-system vagrant-mutate libvirt-dev
sudo usermod -aG libvirt $USER
```
2. Reboot your computer, and then run
```bash
vagrant box add bento/ubuntu-18.04 --provider=virtualbox
vagrant mutate bento/ubuntu-18.04 libvirt
vagrant up bionic --provider=libvirt
```
On other distributions, you will need to install [libvirt](https://libvirt.org/) and `vagrant-mutate` and then run
```bash
vagrant plugin install vagrant-libvirt
sudo usermod -a -G libvirt $USER
# Reboot
vagrant plugin install vagrant-mutate
vagrant box fetch bento/ubuntu-18.04
vagrant mutate bento/ubuntu-18.04 libvirt
vagrant up bionic --provider=libvirt
```
### Starting LibreTime Vagrant
To get started you clone the repo and run `vagrant up`. The command accepts a parameter to
change the default provider if you have multiple installed. This can be done by appending
`--provider=virtualbox` or `--provider=libvirt` as applicable.
```bash
git clone https://github.com/libretime/libretime
cd libretime
vagrant up bionic
```
If everything works out, you will find LibreTime on [port 8080](http://localhost:8080)
and Icecast on [port 8000](http://localhost:8000).
Once you reach the web setup GUI you can click through it using the default values. To
connect to the vagrant machine you can run `vagrant ssh bionic` in the libretime
directory.
### Alternative OS installations
With the above instructions LibreTime is installed on Ubuntu Bionic. The Vagrant setup
offers the option to choose a different operation system according to you needs.
| OS | Command | Comment |
| ------------ | --------------------- | -------------------------------- |
| Debian 10 | `vagrant up buster` | Install on Debian Buster. |
| Debian 11 | `vagrant up bullseye` | Install on Debian Bullseye. |
| Ubuntu 18.04 | `vagrant up bionic` | Install on Ubuntu Bionic Beaver. |
| Ubuntu 20.04 | `vagrant up focal` | Install on Ubuntu Focal Fossa. |
### Troubleshooting
If anything fails during the initial provisioning step you can try running `vagrant provision`
to re-run the installer.
If you only want to re-run parts of the installer, use `--provision-with $step`. The
supported steps are `prepare` and `install`.
## Multipass
[Multipass](https://multipass.run) is a tool for easily setting up Ubuntu VMs on Windows, Mac, and Linux.
Similar to Docker, Multipass works through a CLI. To use, clone this repo and then create a new Multipass VM.
```
git clone https://github.com/libretime/libretime
cd libretime
multipass launch bionic -n ltTEST --cloud-init cloud-init.yaml
multipass shell ltTEST
```
Multipass isn't currently able to do an automated install from the cloud-init script.
After you enter the shell for the first time, you will still need to [run the LibreTime installer](../../admin-manual/setup/install.md).
The IP address of your new VM can be found by running `multipass list`. Copy and paste it into your web browser to access the LibreTime interface and complete the setup wizard.
You can stop the VM with `multipass stop ltTEST` and restart with `multipass start ltTEST`.
If you want to delete the image and start again, run `multipass delete ltTEST && multipass purge`.
### Cloud-init options in cloud-init.yaml
You may wish to change the below fields as per your location.
```yaml
timezone: America/New York # change as needed
ntp:
pools: ["north-america.pool.ntp.org"]
servers: ["0.north-america.pool.ntp.org", "0.pool.ntp.org"]
```

143
docs/developer-manual/development/releases.md
View file

@ -1,143 +0,0 @@
---
title: Releases
---
## Distributions releases support
New releases target the current stable distributions release, and development should prepare for future stable distributions releases.
- We recommend installing LibreTime on the current stable distributions.
- Maintenance only releases will provide bug and security fixes for stable and old stable distributions.
| | Ubuntu 18.04 | Debian 10 | Ubuntu 20.04 | Debian 11 |
| ------------ | :----------: | :---------: | :----------: | :---------: |
| Release date | 2018-04-26 | 2019-07-06 | 2020-04-23 | 2021-08-14 |
| End of life | 2023-04 | 2024-06 | 2025-04 | 2026-06 |
| Versions | | | | |
| 3.0.x | maintenance | maintenance | recommended | recommended |
## Versioning schema
We follow the [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standards.
In a nutshell, given a version number `MAJOR.MINOR.PATCH` we increment the:
1. `MAJOR` version when we make incompatible API changes,
2. `MINOR` version when we add functionality in a backwards-compatible manner, and
3. `PATCH` version when we make backwards-compatible bug fixes.
## Releasing a new version
This guide walks you through the steps required to release a new version of LibreTime.
:::caution
This guide is still a work in progress, and doesn't cover every use cases. Depending on
the version bump, some steps might be wrong. For example, in case of a patch release,
the documentation requires different changes.
:::
Before releasing a new version, make sure linter don't fail and tests are passing.
Start by cleaning the repository and make sure you don't have uncommitted changes:
```
git checkout main
make clean
git status
```
Choose the next version based the our [versioning schema](#versioning-schema):
```bash
export VERSION=3.0.0-beta.0
```
Create a new `release-$VERSION` branch and release commit to prepare a release pull request:
```bash
git checkout -b "release-$VERSION"
export COMMIT_MESSAGE="chore: release $VERSION"
git commit --allow-empty --message="$COMMIT_MESSAGE"
```
### 1. Version bump
Write the new `$VERSION` to the VERSION file, and bump the python packages version:
```bash
bash tools/bump-python-version.sh "$VERSION"
git add .
git commit --fixup ":/$COMMIT_MESSAGE"
```
### 2. Release note
Prepare a new release note based on the `docs/releases/unreleased.md` file. Be sure that
the filename match the releases notes naming conventions:
```bash
ls -l docs/releases/
cp docs/releases/unreleased.md docs/releases/$VERSION.md
```
The release note file must be updated with:
- the version and date of this release,
- an auto generated features and bug fixes changelog,
- instructions for upgrading,
- deprecation notices,
- remove empty sections.
Reset and clean the `docs/releases/unreleased.md` file for a future version.
Commit the release note changes:
```bash
git add .
git commit --fixup ":/$COMMIT_MESSAGE"
```
### 3. Create a new pull request
Squash the changes and open a pull request for others to review:
```bash
git rebase --autosquash --interactive main
```
Merge the pull request when it's reviewed and ready.
### 4. Create and push a tag
Pull the merged release commit:
```bash
git checkout main
git pull upstream main
```
Make sure `HEAD` is the previously merged release commit and tag it with the new version:
```bash
git show --quiet
git tag -a -m "$VERSION" "$VERSION"
```
Generate the changelog for the newly tagged version:
```bash
make changelog
git add .
git commit -m "chore: generate changelog for $VERSION"
```
Push the tag upstream to finalize the release process:
```bash
git push upstream main --follow-tags
```