Learning paths / Deploy on ComputeSphere / Data: databases and storage

Stateless services and where state lives

Reading · 5 min · Module 5, lesson 1 of 546 min left in this module

Module 5 · Data: databases and storageLesson 1 of 5

Goal: Decide where each piece of your app's data belongs, and keep it out of the spherelet's own filesystem.

Key idea

Anything your app writes to its own filesystem disappears when the spherelet is replaced. Keep data somewhere built to keep it: a database, object storage, or a SphereFiles volume.

The spherelet's filesystem is scratch space

Each spherelet starts with a fresh copy of the filesystem from your image. Your app can write to it, but those writes belong to that one spherelet and vanish when it's replaced.

That happens more often than you'd expect: on every restart and every redeploy that applies a change, after a crash, and when you scale. A new spherelet never sees files an old one wrote.

So a service that saves uploads to /app/uploads works in testing, then loses every file on the next deploy. With two spherelets it's worse: an upload lands on one, and the next request may reach the other.

Stateless is the goal

A stateless service keeps nothing it needs on its own disk. Any spherelet can answer any request, so you can replace, restart or add spherelets without losing anything. Most web services and workers can be written this way.

Temporary files are still fine: a file you write, use and delete in one request, or a cache you can rebuild. The rule is only that losing the file must never lose data.

Where state belongs

Kind of dataWhere it belongsExample
Records you query: users, ordersA databaseSphereDB (managed PostgreSQL), or a database from a provider
Files your services read and write at a pathA volume (SphereFiles)Uploads, media, generated reports
Files fetched by URL, or very manyObject storageA bucket from your cloud or storage provider
Values that are cheap to loseMemory or a cacheRendered pages, rate-limit counters

Object storage is a service that stores whole files by name over HTTP; "S3-compatible" means it speaks the same API as Amazon S3, so most libraries work with it.

For records, ComputeSphere runs SphereDB, managed PostgreSQL in your environment, or you can bring a database from a provider and connect with secrets (lesson 5.5.3). For object storage, bring a bucket from a provider. For files your services keep, it provides SphereFiles volumes: a volume stays in your environment when the spherelets using it are replaced, and several services can mount it at once.

A volume or a database?

A volume looks like a folder, which makes it tempting for everything. It's a good home for files: every spherelet of every service that mounts it sees the same ones, so a service with a volume can still scale out. Copies of the files for recovery are up to you.

It's a poor home for records. A volume is network storage built for files, not for a database's own files, and two spherelets writing the same file at once can clash. A database handles connections from many spherelets at once and gives you queries and transactions (several changes that succeed or fail together). If you query it, put it in a database.

A quick test for any service

Imagine it running on three spherelets, then restarted. Would any request fail, or any data be missing? If yes, find the file or in-memory value that caused it and move it to one of the places in the table.

Variables and secrets are configuration, not data. Keep them in the service's settings rather than baked into the image, so the same image runs in every environment.

Check yourself

Your web service runs on 3 spherelets and needs to keep users' profile photos. Where should the photos go?

In the docs