Commit Graph

239 Commits

Author SHA1 Message Date
Hans-Joachim aa64b8eab8 get rid of SaaS definitions 2017-03-21 22:56:43 +01:00
Lucas Bickel c29285ae48 Vagrant Debian support (and experimental CentOS)
This changes the Vagrant setup to support multiple installations as multiple
boxes. In addition to Ubuntu Vagrant can now be used to install on Debian
as well as on CentOS.

I took the chance to clean up the .deb install a bit and backported analyzer
and celery to SysV proper so it runs there. Some of the distro specfics were
moved to the install script from the python setup scripts to acheive this.

For the CentOS support I added a rather involved OS prepare script. In the
long term this will be added to the preparing-the-server docs we already have.

I had to switch the default port to http-alt (8080). On CentOS 9080 is registered
for ocsp and getting it to work for apache without hacking SELinux is hard. I
think 8080 is the RFC way to go anyhow. If anyone want to override this it
should be rather easy using the --web-port arg and by hacking Vagrantfile.

The PyOpenSSL code has been refactored for all the distros that the Vagrantfile
now supports.

As far as my checks go, I tried this code with all the distros, uploaded a track
and downloaded a unicode and a ssl podcast and was able to listen to them
in each case.

In the experimental CentOS case, the UI is not up to spec since services
need to get scheduled through systemctl and the status overview (ie. on the /?config page)
do not work properly. They need to be as follows:

```
sudo systemctl start airtime-playout
sudo systemctl start airtime-liquidsoap
sudo systemctl start airtime_analyzer.service
sudo systemctl start airtime-celery.service
```
2017-03-12 13:42:22 +01:00
Lucas Bickel 188c127057 Merge branch 'master' into feature/analyzer-on-travis
# Conflicts:
#	.travis.yml
2017-03-04 16:35:11 +01:00
Lucas Bickel 9b3d0c90da mostly run analyzer tests on travis
* [x] regonfigured the build matrix with more php jobs and a separate python job (we can add more python jobs later)
* [x] run tests on travis' trusty beta container (it's closer to what we need anyway)
* [x] install packages needed for analyzer tests in build env
* [x] added docs on how to run nosetests locally
* [x] don't run initctl in analyzer setup so setup can also be used on travis (and add it to the install script directly)
* [x] ignore replaygain checks on travis (it has proven quite impossible to get the needed python-gi module to work in the provided virtualenv)

I tried a lot of solutions to get the replaygain checks to run. I needed to decide that this has gone far enough, maybe someone who is more of a pythonista than me can take a crack at it and get it solved. Even without running those tests on CI/CD there are still plenty others.

This PR only has parts of what are needed for getting python tests running on travis as per #15. I only took a quick shot at anything not analyzer and figured I would not be able to "fix" them without digging a bit deeper (ie. also getting rid of std_err_override).
2017-03-03 20:38:27 +01:00
Lucas Bickel e28ad471f9 Rewrite config from /etc/airtime-saas to plain /etc/airtime
This is the results of sed -i -e 's|/etc/airtime-saas/|/etc/airtime/|' `grep -irl 'airtime-saas' airtime_mvc/ python_apps/` :P

It might need more testing, the airtime-saas part never really made sense, zf1 has environments for that, ie you would create a saas env based on production for instance.

I beleive legacy upstream was using this to share configuration between customers (ie. analyser runs only once and writes to a shared S3 bucket). I assume they mount the airtime-saas folder onto individual customers instances with a global config. Like I said, I don't feel that this makes sense since all it does is make hacking at the configs in airtime-saas a bit easier. A serious SaaS operation should be using something like puppet or ansible to achieve this.
2017-03-03 15:57:41 +01:00
Albert Santoni 965a78c54a Merge branch 'saas-dev' into saas-dev-publishing
Conflicts:
	airtime_mvc/application/controllers/ApiController.php
2015-11-18 12:38:18 -05:00
Albert Santoni c85944785b Comments 2015-10-30 18:09:56 -04:00
Albert Santoni b49bb2e262 Loosen up Mutagen vs. Silan cue out length threshold a bit 2015-10-30 17:54:24 -04:00
Albert Santoni 7be548f30e Fixed the airtime_analyzer unit tests 2015-10-30 17:15:33 -04:00
Albert Santoni fc51e63214 Fix typo in airtime_analyzer setup.py 2015-10-30 17:13:12 -04:00
Albert Santoni 76f202106b Defensive coding against Silan bugs and bump to Mutagen 1.31 2015-10-30 17:12:13 -04:00
Duncan Sommerville a78a0d0cc8 * Fix track number bug in analyzer
* Move library search bar out of header
2015-09-16 17:50:21 -04:00
Albert Santoni 67c5b73db4 airtime_analyzer unit test backwards compatibility with older versions of requests 2015-07-08 10:32:35 -04:00
Duncan Sommerville 4c797cf100 Update installer to work with saas branch 2015-06-23 19:02:55 -04:00
Albert Santoni c7aa6b3e5e airtime_analyzer: use requests >= 2.7.0 2015-05-20 10:36:55 -04:00
Albert Santoni 3d8cd8e927 Added daemon dependency back in for airtime_analyzer 2015-05-19 13:45:08 -04:00
Albert Santoni e1c50ecc7b Squash urllib3 warnings to see if it helps with SAAS-714 2015-05-19 13:09:43 -04:00
Albert Santoni add211d13c Cleanup to help with SAAS-714 2015-05-19 12:02:19 -04:00
Albert Santoni b3a19ae49e Bugfix for embedded player sending too many requests on load, quieter logging for boto 2015-04-29 11:46:44 -04:00
Albert Santoni 28fc1cf2fc Fixed some airtime_analyzer unit tests 2015-04-10 15:08:44 -04:00
Albert Santoni e3390b99ef Disable analyzer log file rotation (already handed by Upstart), clean up 2015-04-10 15:03:58 -04:00
Albert Santoni d5012c25cb Another small bugfix for error handling in the analyzer 2015-04-06 17:33:08 -04:00
Albert Santoni 492a7f329a Minor airtime_analyzer error handling improvements and documentation 2015-04-06 17:22:13 -04:00
Albert Santoni 7fd58ac51b Merge branch 'saas-s3proxy' into saas 2015-04-01 13:49:47 -04:00
Albert Santoni ae02fe7816 Log traceback in analyzer status_reporter.py exception handler 2015-03-30 13:21:26 -04:00
Albert Santoni 271dc266fa S3 proxy cache support + 1 minor analyzer bugfix 2015-03-26 12:08:52 -04:00
Albert Santoni d1b28fd564 Fixed a bug in FTP upload hook, other minor improvements
* Ensure we have write permissions on any newly uploaded files by FTP
* Don't silence warnings in moveFileToStor()
* Reduced the FTP upload max retry count
* Log E_NOTICE and E_WARNING levels to Sentry
2015-03-24 14:56:03 -04:00
Albert Santoni e182e73626 SAAS-602: airtime_analyzer deadlocks in S3 hostname lookup
* Stopped using multiprocess.subprocess because it's dangerous and
  unreliable due to Python bug 6721: http://bugs.python.org/issue6721
2015-03-11 18:33:08 -04:00
Albert Santoni 6cecb7c239 Simplified the FTP upload hook 2015-03-06 11:06:17 -05:00
Albert Santoni ab6d83f49d Merge branch 'saas' into saas-media-refactor
Conflicts:
	airtime_mvc/application/Bootstrap.php
	airtime_mvc/application/modules/rest/controllers/MediaController.php
2015-02-26 13:10:13 -05:00
Albert Santoni d82f461f35 Fix missing filesize and md5 for WAVE imports 2015-02-26 13:06:29 -05:00
Albert Santoni edd07c18ae Fixed invalid detection of S3 backend in analyzer 2015-02-26 12:02:02 -05:00
Albert Santoni 3a1141d4c8 Merge branch 'saas' into saas-media-refactor
Conflicts:
	airtime_mvc/application/cloud_storage/ProxyStorageBackend.php
	airtime_mvc/application/controllers/ApiController.php
2015-02-25 13:02:11 -05:00
Albert Santoni 37f9f273c7 Fixed the airtime_analyzer unit tests 2015-02-24 17:03:38 -05:00
Albert Santoni 702de0f5f5 SAAS-602: Fix getaddrinfo deadlock (again)
Conflicts:
	python_apps/airtime_analyzer/airtime_analyzer/cloud_storage_uploader.py
2015-02-24 13:14:21 -05:00
Albert Santoni 8aaf300491 Attempt a workaround for unicode hostname resolution deadlock in the
analyzer
2015-02-23 18:22:16 -05:00
Albert Santoni 51aa846a4c Merge branch 'cc-5709-airtime-analyzer-refactor' into saas-media-refactor
Conflicts:
	airtime_mvc/application/models/RabbitMq.php
	python_apps/airtime_analyzer/airtime_analyzer/analyzer_pipeline.py
	python_apps/airtime_analyzer/airtime_analyzer/message_listener.py
2015-02-20 18:25:38 -05:00
Albert Santoni c868136d26 Make airtime_analyzer respect the storage_backend setting passed to it
by Airtime
2015-02-20 18:21:49 -05:00
drigato 58f9ffd0eb Merge branch 'cc-5709-airtime-analyzer-store-file-size-and-hash-in-db' into saas-store-file-size-and-hash-in-db 2015-02-19 15:40:27 -05:00
drigato b6acfb2dce SAAS-596: Store file size and hash in database
Renamed metadata md5 value to match cc_files column
2015-02-19 15:39:55 -05:00
drigato 6ef085f460 Merge branch 'saas' into saas-store-file-size-and-hash-in-db 2015-02-19 11:43:34 -05:00
drigato 0f921cb481 Merge branch 'cc-5709-airtime-analyzer-store-file-size-and-hash-in-db' into saas-store-file-size-and-hash-in-db 2015-02-19 10:51:53 -05:00
drigato cc9e6efbca SAAS-596: Store file size and hash in database
Removed try/except while getting file size and hash
2015-02-19 10:38:30 -05:00
Albert Santoni 70f5bd3748 Fixed the unit tests 2015-02-19 10:17:42 -05:00
Albert Santoni d0bd4bfe81 Merge branch 'saas' into saas-store-file-size-and-hash-in-db
Conflicts:
	airtime_mvc/application/modules/rest/controllers/MediaController.php
2015-02-18 19:29:45 -05:00
drigato 5bd3371ec2 SAAS-596: Store file size and hash in database
Removed getFileSize from CloudFile.php since it can call the parent
function.
Moved setting filesize metadata from cloud_storage_uploader.
2015-02-17 16:51:25 -05:00
drigato c1086842d3 Merge branch 'cc-5709-airtime-analyzer-store-file-size-and-hash-in-db' into saas-store-file-size-and-hash-in-db
Conflicts:
	airtime_mvc/application/Bootstrap.php

airtime_mvc/application/modules/rest/controllers/MediaController.php
2015-02-17 16:19:58 -05:00
drigato 66a6a8f985 SAAS-596: Store file size and hash in database
Added file size and hash to the metadata dict in the analyzer
2015-02-17 16:02:49 -05:00
Albert Santoni d8f5e41fca Set a connection timeout for Boto to prevent cloud storage deadlock 2015-02-17 14:58:19 -05:00
drigato cee0ff4881 SAAS-560: Deploy separate cloud storage config files for each development environment
Default to production config files if dev env specific files are not
found.
Fix analyzer unit tests.
2015-02-05 14:31:20 -05:00