Commit graph

239 commits

Author SHA1 Message Date
Hans-Joachim
aa64b8eab8 get rid of SaaS definitions 2017-03-21 22:56:43 +01:00
Lucas Bickel
c29285ae48 Vagrant Debian support (and experimental CentOS)
This changes the Vagrant setup to support multiple installations as multiple
boxes. In addition to Ubuntu Vagrant can now be used to install on Debian
as well as on CentOS.

I took the chance to clean up the .deb install a bit and backported analyzer
and celery to SysV proper so it runs there. Some of the distro specfics were
moved to the install script from the python setup scripts to acheive this.

For the CentOS support I added a rather involved OS prepare script. In the
long term this will be added to the preparing-the-server docs we already have.

I had to switch the default port to http-alt (8080). On CentOS 9080 is registered
for ocsp and getting it to work for apache without hacking SELinux is hard. I
think 8080 is the RFC way to go anyhow. If anyone want to override this it
should be rather easy using the --web-port arg and by hacking Vagrantfile.

The PyOpenSSL code has been refactored for all the distros that the Vagrantfile
now supports.

As far as my checks go, I tried this code with all the distros, uploaded a track
and downloaded a unicode and a ssl podcast and was able to listen to them
in each case.

In the experimental CentOS case, the UI is not up to spec since services
need to get scheduled through systemctl and the status overview (ie. on the /?config page)
do not work properly. They need to be as follows:

```
sudo systemctl start airtime-playout
sudo systemctl start airtime-liquidsoap
sudo systemctl start airtime_analyzer.service
sudo systemctl start airtime-celery.service
```
2017-03-12 13:42:22 +01:00
Lucas Bickel
188c127057 Merge branch 'master' into feature/analyzer-on-travis
# Conflicts:
#	.travis.yml
2017-03-04 16:35:11 +01:00
Lucas Bickel
9b3d0c90da mostly run analyzer tests on travis
* [x] regonfigured the build matrix with more php jobs and a separate python job (we can add more python jobs later)
* [x] run tests on travis' trusty beta container (it's closer to what we need anyway)
* [x] install packages needed for analyzer tests in build env
* [x] added docs on how to run nosetests locally
* [x] don't run initctl in analyzer setup so setup can also be used on travis (and add it to the install script directly)
* [x] ignore replaygain checks on travis (it has proven quite impossible to get the needed python-gi module to work in the provided virtualenv)

I tried a lot of solutions to get the replaygain checks to run. I needed to decide that this has gone far enough, maybe someone who is more of a pythonista than me can take a crack at it and get it solved. Even without running those tests on CI/CD there are still plenty others.

This PR only has parts of what are needed for getting python tests running on travis as per #15. I only took a quick shot at anything not analyzer and figured I would not be able to "fix" them without digging a bit deeper (ie. also getting rid of std_err_override).
2017-03-03 20:38:27 +01:00
Lucas Bickel
e28ad471f9 Rewrite config from /etc/airtime-saas to plain /etc/airtime
This is the results of sed -i -e 's|/etc/airtime-saas/|/etc/airtime/|' `grep -irl 'airtime-saas' airtime_mvc/ python_apps/` :P

It might need more testing, the airtime-saas part never really made sense, zf1 has environments for that, ie you would create a saas env based on production for instance.

I beleive legacy upstream was using this to share configuration between customers (ie. analyser runs only once and writes to a shared S3 bucket). I assume they mount the airtime-saas folder onto individual customers instances with a global config. Like I said, I don't feel that this makes sense since all it does is make hacking at the configs in airtime-saas a bit easier. A serious SaaS operation should be using something like puppet or ansible to achieve this.
2017-03-03 15:57:41 +01:00
Albert Santoni
965a78c54a Merge branch 'saas-dev' into saas-dev-publishing
Conflicts:
	airtime_mvc/application/controllers/ApiController.php
2015-11-18 12:38:18 -05:00
Albert Santoni
c85944785b Comments 2015-10-30 18:09:56 -04:00
Albert Santoni
b49bb2e262 Loosen up Mutagen vs. Silan cue out length threshold a bit 2015-10-30 17:54:24 -04:00
Albert Santoni
7be548f30e Fixed the airtime_analyzer unit tests 2015-10-30 17:15:33 -04:00
Albert Santoni
fc51e63214 Fix typo in airtime_analyzer setup.py 2015-10-30 17:13:12 -04:00
Albert Santoni
76f202106b Defensive coding against Silan bugs and bump to Mutagen 1.31 2015-10-30 17:12:13 -04:00
Duncan Sommerville
a78a0d0cc8 * Fix track number bug in analyzer
* Move library search bar out of header
2015-09-16 17:50:21 -04:00
Albert Santoni
67c5b73db4 airtime_analyzer unit test backwards compatibility with older versions of requests 2015-07-08 10:32:35 -04:00
Duncan Sommerville
4c797cf100 Update installer to work with saas branch 2015-06-23 19:02:55 -04:00
Albert Santoni
c7aa6b3e5e airtime_analyzer: use requests >= 2.7.0 2015-05-20 10:36:55 -04:00
Albert Santoni
3d8cd8e927 Added daemon dependency back in for airtime_analyzer 2015-05-19 13:45:08 -04:00
Albert Santoni
e1c50ecc7b Squash urllib3 warnings to see if it helps with SAAS-714 2015-05-19 13:09:43 -04:00
Albert Santoni
add211d13c Cleanup to help with SAAS-714 2015-05-19 12:02:19 -04:00
Albert Santoni
b3a19ae49e Bugfix for embedded player sending too many requests on load, quieter logging for boto 2015-04-29 11:46:44 -04:00
Albert Santoni
28fc1cf2fc Fixed some airtime_analyzer unit tests 2015-04-10 15:08:44 -04:00
Albert Santoni
e3390b99ef Disable analyzer log file rotation (already handed by Upstart), clean up 2015-04-10 15:03:58 -04:00
Albert Santoni
d5012c25cb Another small bugfix for error handling in the analyzer 2015-04-06 17:33:08 -04:00
Albert Santoni
492a7f329a Minor airtime_analyzer error handling improvements and documentation 2015-04-06 17:22:13 -04:00
Albert Santoni
7fd58ac51b Merge branch 'saas-s3proxy' into saas 2015-04-01 13:49:47 -04:00
Albert Santoni
ae02fe7816 Log traceback in analyzer status_reporter.py exception handler 2015-03-30 13:21:26 -04:00
Albert Santoni
271dc266fa S3 proxy cache support + 1 minor analyzer bugfix 2015-03-26 12:08:52 -04:00
Albert Santoni
d1b28fd564 Fixed a bug in FTP upload hook, other minor improvements
* Ensure we have write permissions on any newly uploaded files by FTP
* Don't silence warnings in moveFileToStor()
* Reduced the FTP upload max retry count
* Log E_NOTICE and E_WARNING levels to Sentry
2015-03-24 14:56:03 -04:00
Albert Santoni
e182e73626 SAAS-602: airtime_analyzer deadlocks in S3 hostname lookup
* Stopped using multiprocess.subprocess because it's dangerous and
  unreliable due to Python bug 6721: http://bugs.python.org/issue6721
2015-03-11 18:33:08 -04:00
Albert Santoni
6cecb7c239 Simplified the FTP upload hook 2015-03-06 11:06:17 -05:00
Albert Santoni
ab6d83f49d Merge branch 'saas' into saas-media-refactor
Conflicts:
	airtime_mvc/application/Bootstrap.php
	airtime_mvc/application/modules/rest/controllers/MediaController.php
2015-02-26 13:10:13 -05:00
Albert Santoni
d82f461f35 Fix missing filesize and md5 for WAVE imports 2015-02-26 13:06:29 -05:00
Albert Santoni
edd07c18ae Fixed invalid detection of S3 backend in analyzer 2015-02-26 12:02:02 -05:00
Albert Santoni
3a1141d4c8 Merge branch 'saas' into saas-media-refactor
Conflicts:
	airtime_mvc/application/cloud_storage/ProxyStorageBackend.php
	airtime_mvc/application/controllers/ApiController.php
2015-02-25 13:02:11 -05:00
Albert Santoni
37f9f273c7 Fixed the airtime_analyzer unit tests 2015-02-24 17:03:38 -05:00
Albert Santoni
702de0f5f5 SAAS-602: Fix getaddrinfo deadlock (again)
Conflicts:
	python_apps/airtime_analyzer/airtime_analyzer/cloud_storage_uploader.py
2015-02-24 13:14:21 -05:00
Albert Santoni
8aaf300491 Attempt a workaround for unicode hostname resolution deadlock in the
analyzer
2015-02-23 18:22:16 -05:00
Albert Santoni
51aa846a4c Merge branch 'cc-5709-airtime-analyzer-refactor' into saas-media-refactor
Conflicts:
	airtime_mvc/application/models/RabbitMq.php
	python_apps/airtime_analyzer/airtime_analyzer/analyzer_pipeline.py
	python_apps/airtime_analyzer/airtime_analyzer/message_listener.py
2015-02-20 18:25:38 -05:00
Albert Santoni
c868136d26 Make airtime_analyzer respect the storage_backend setting passed to it
by Airtime
2015-02-20 18:21:49 -05:00
drigato
58f9ffd0eb Merge branch 'cc-5709-airtime-analyzer-store-file-size-and-hash-in-db' into saas-store-file-size-and-hash-in-db 2015-02-19 15:40:27 -05:00
drigato
b6acfb2dce SAAS-596: Store file size and hash in database
Renamed metadata md5 value to match cc_files column
2015-02-19 15:39:55 -05:00
drigato
6ef085f460 Merge branch 'saas' into saas-store-file-size-and-hash-in-db 2015-02-19 11:43:34 -05:00
drigato
0f921cb481 Merge branch 'cc-5709-airtime-analyzer-store-file-size-and-hash-in-db' into saas-store-file-size-and-hash-in-db 2015-02-19 10:51:53 -05:00
drigato
cc9e6efbca SAAS-596: Store file size and hash in database
Removed try/except while getting file size and hash
2015-02-19 10:38:30 -05:00
Albert Santoni
70f5bd3748 Fixed the unit tests 2015-02-19 10:17:42 -05:00
Albert Santoni
d0bd4bfe81 Merge branch 'saas' into saas-store-file-size-and-hash-in-db
Conflicts:
	airtime_mvc/application/modules/rest/controllers/MediaController.php
2015-02-18 19:29:45 -05:00
drigato
5bd3371ec2 SAAS-596: Store file size and hash in database
Removed getFileSize from CloudFile.php since it can call the parent
function.
Moved setting filesize metadata from cloud_storage_uploader.
2015-02-17 16:51:25 -05:00
drigato
c1086842d3 Merge branch 'cc-5709-airtime-analyzer-store-file-size-and-hash-in-db' into saas-store-file-size-and-hash-in-db
Conflicts:
	airtime_mvc/application/Bootstrap.php

airtime_mvc/application/modules/rest/controllers/MediaController.php
2015-02-17 16:19:58 -05:00
drigato
66a6a8f985 SAAS-596: Store file size and hash in database
Added file size and hash to the metadata dict in the analyzer
2015-02-17 16:02:49 -05:00
Albert Santoni
d8f5e41fca Set a connection timeout for Boto to prevent cloud storage deadlock 2015-02-17 14:58:19 -05:00
drigato
cee0ff4881 SAAS-560: Deploy separate cloud storage config files for each development environment
Default to production config files if dev env specific files are not
found.
Fix analyzer unit tests.
2015-02-05 14:31:20 -05:00