What is the current issue

When you’re running a self-hosted app, not really designed to run on Kubernetes, most devs will choose a database system like sqlite because it’s just a file. A

Some reasons you would choose Sqlite

  1. Lightweight
  2. Fast
  3. Reliable
  4. Portable

I personally disagree on points 3, it sometimes just breaks.

When you are running an application in Kubernetes, it’s usually stateless. Kubernetes has historically been very, very good at running stateless apps, and quite bad at stateful apps. That naturally has changed as more development has been put in to getting kubernetes to a point that it can run Stateful apps well.

What this means for us is, storage is a royal pain. You can run with local volumes, but I run Talos Linux and that is very much against the point and ethos of Talos.

You land up putting SQLite on an NFS mount, or an iSCSI backed PV. Both these solutions are clunky in my opinion, as they still mean you’re at the mercy of the network connection to ensure there’s no breakage between writes.

The solution

litestream is a lightweight replication daemon for SQLite. It continuously streams the write-ahead log (WAL) to an external replica target such as S3, GCS, or a local filesystem

I’m assuming that if you’ve been running SQLite in your cluster already, you’ve got some sort of external NFS related system setup. Wether that be open media vault, true nas or Synology etc.

The design

  • The Pod will mount an empty dir for the SQLite database
  • Litestream will replicate the database to an NFS volume also mounted

Example

We will be using the example application of surmai which is a trip planning application I have recently discovered.

First, we need to create an NFS share on our Nas. I will not be going in to this as each NAS is different.

Let’s assume that our share is called /volume1/sqlite-replication/surami

apiVersion: v1
kind: PersistentVolume
metadata:
  name: trip-planner-litestream
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: <dns or ip of your nas>
    path: /volume1/sqlite-replication/trip-planner
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: trip-planner-litestream
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeName: trip-planner-litestream

This now creates a PVC called trip-planner-litestream where we can replicate our LTX directories to (Litestream Transaction Log)

We need to then tell Litestream where the sqlite database is, and where to replicate it to.

Remember, we will be mounting the pvc trip-planner-litestream to /sqlite-replication on the Pod.

Create a config map like the below:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: litestream
data:
  litestream.yaml: |-
    addr: ":9090"
    sync-interval: 1s
    dbs:
      - path: /data/data.db
        replica:
          type: file
          path: /sqlite-replication/data.db

Let me explain the lines

  • sync-interval: 1s : Sync the sqlite database once a second
  • - path: /data/data.db : Path and name of the sqlite database
  • path: /sqlite-replication/data.db : Path and name of the replicated database

Now we’ve got the config setup, we can configure the application!

I will first pull out the specific sections that are related to Litestream, and then give you the full file

First we’ve got the Init container.

This checks if the database exists in the 10gb temp, and if not pulls it from the NFS mount

apiVersion: apps/v1
kind: Deployment
metadata:
  name: surmai
spec:
  replicas: 1
  selector:
    matchLabels:
      app: surmai
  template:
    spec:
      restartPolicy: Always
      initContainers:
        - name: init-litestream
          image: litestream/litestream:0.5.9
          args: [ 'restore', '-if-db-not-exists', '-if-replica-exists', '/data/data.db' ]
          volumeMounts:
            - mountPath: /etc/litestream.yaml
              name: litestream
              subPath: litestream.yaml
            - mountPath: /sqlite-replication
              name: sqlite-replication
            - mountPath: /data
              name: sqlite
          env:
            - name: LITESTREAM_CONFIG
              value: "/etc/litestream.yaml"
      volumes:
        - name: sqlite-replication
          persistentVolumeClaim:
            claimName: trip-planner-litestream
        - name: litestream
          configMap:
            name: litestream
        - name: sqlite
          emptyDir:
            sizeLimit: 10Gi

Then, there’s the continual replication section

apiVersion: apps/v1
kind: Deployment
metadata:
  name: surmai
spec:
  replicas: 1
  selector:
    matchLabels:
      app: surmai
  template:
    spec:
      restartPolicy: Always
      containers:
        - name: litestream
          image: litestream/litestream:0.5.9
          args: ['replicate']
          volumeMounts:
            # Mounts the litestream config map to a file
            - mountPath: /etc/litestream.yaml
              name: litestream
              subPath: litestream.yaml
              # Mounts the NFS volume
            - mountPath: /sqlite-replication
              name: sqlite-replication
            - mountPath: /data
              name: sqlite
          env:
            - name: LITESTREAM_CONFIG
              value: "/etc/litestream.yaml"
      volumes:
        - name: sqlite-replication
          persistentVolumeClaim:
            claimName: trip-planner-litestream
        - name: litestream
          configMap:
            name: litestream
        - name: sqlite
          emptyDir:
            sizeLimit: 10Gi

Full Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: surmai
  labels:
    app: surmai
spec:
  replicas: 1
  selector:
    matchLabels:
      app: surmai
  template:
    metadata:
      name: surmai
      labels:
        app: surmai
    spec:
      restartPolicy: Always
      initContainers:
        - name: init-litestream
          image: litestream/litestream:0.5.9
          args: ['restore', '-if-db-not-exists', '-if-replica-exists', '/data/data.db']
          volumeMounts:
            - mountPath: /etc/litestream.yaml
              name: litestream
              subPath: litestream.yaml
            - mountPath: /sqlite-replication
              name: sqlite-replication
            - mountPath: /data
              name: sqlite
          env:
            - name: LITESTREAM_CONFIG
              value: "/etc/litestream.yaml"
      containers:
        - name: surmai
          image: ghcr.io/rohitkumbhar/surmai:v0.4.10
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - mountPath: /data
              name: sqlite
            - mountPath: /data/storage
              name: data
          envFrom:
            - configMapRef:
                name: surmai
          ports:
            - containerPort: 8080
              protocol: TCP
        - name: litestream
          image: litestream/litestream:0.5.9
          args: ['replicate']
          volumeMounts:
            - mountPath: /etc/litestream.yaml
              name: litestream
              subPath: litestream.yaml
            - mountPath: /sqlite-replication
              name: sqlite-replication
            - mountPath: /data
              name: sqlite
          env:
            - name: LITESTREAM_CONFIG
              value: "/etc/litestream.yaml"
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: trip-planner
        - name: sqlite-replication
          persistentVolumeClaim:
            claimName: trip-planner-litestream
        - name: litestream
          configMap:
            name: litestream
        - name: sqlite
          emptyDir:
            sizeLimit: 10Gi