High Available Django Setup

Roland van Laar

Entepreneur Education and IT

twitter: @rvanlaar github: rvanlaar

Application Demo

Goals

Challenges

Overview

BDR

CSYNC2

WebSockets

That one time we were down

Demo: EduWorld

EduWorld

Customer's Goals

Not expensive

Always up

Reliable

Technical Goals

i.e. My Goals

Buildable

Functional

Maintainable

Challenges

Challenges

Media Files

Websockets

Database

Sessions

Overview

Network Topology

Django

Mostly CRUD

Treebeard, DjangoRestFrameWork

New Django Postgres features

Front End

Single Page JQuery app

Choose a random api host for ajax requests

Changes api host when api is not responding

Bi-directional Replication (BDR)

Postgres async Multi Master

Awesome

docs

BDR: gotchas

Replication user

Sequences: USING BDR, UUID

Backups

DDL: limitations

Needs a lock on all nodes

Downtime

Limitations on DDL

DDL and Migrations

operations = [
  migrations.RunSQL(
    """
    ALTER TABLE "backend_user" ADD COLUMN "voorkeuren" text;
    ALTER TABLE "backend_user" ALTER COLUMN "voorkeuren" SET DEFAULT '""';
    UPDATE "backend_user" SET "voorkeuren" = DEFAULT;
    ALTER TABLE "backend_user" ALTER COLUMN "voorkeuren" SET NOT NULL;
    ALTER TABLE "backend_user" ALTER COLUMN "voorkeuren" DROP DEFAULT;
    """,
    state_operations=[
        migrations.AddField(
            model_name='user',
            name='voorkeuren',
            field=jsonfield.fields.JSONField(default=''),
            preserve_default=False,)])]

CSYNC2

group media {
 host host1.example.com host2.example.com;
 key /etc/csync2/csync2-media.key;
 include /var/www/django/media;
 auto younger;
}

Cronjob on both hosts: 

/usr/sbin/csync2 -xv

Django Code

import socket
hostname = socket.gethostname()
media_urls = {
    'host1': 'https://host1.example.com/media/',
    'host2': 'https://host2.example.com/media/'
    }

if hostname in media_urls.keys():
    MEDIA_URL = media_urls[hostname]
else:
    ERROR HANDLING

Websockets

Tornado

Reconnecting websockets

small id for initial connect

uuid afterwards

That one time we were down