Replacing Redis with SQLite for Local Caching

For the past two years, our team ran Redis as the caching layer for every service we deployed - including a small internal tool that processed fewer than 200 requests per minute. After the third time we got paged at 2 AM because the Redis instance on our staging box ran out of memory, we decided it was time to reconsider whether we actually needed a separate caching server for this workload.

The Problem

Redis is exceptional software. For high-throughput, distributed systems it remains the obvious choice. But for our situation - a single-process Python service running on one machine with modest traffic - Redis introduced operational overhead that outweighed its benefits:

• A separate process to monitor, restart, and upgrade
• Network round-trips for every cache read and write, even on localhost
• Memory consumption that grew unpredictably under our access patterns
• Configuration drift between development, staging, and production
• An extra dependency in our Docker Compose stack and CI pipeline

We were paying the complexity tax of a distributed system for a problem that wasn't distributed.

Why SQLite?

SQLite is already linked into the Python standard library. There is no server, no socket, no protocol negotiation. It stores data in a single file on disk, and with WAL mode enabled, concurrent reads are non-blocking. For our use case, it checked every box:

• Zero deployment overhead - no extra container, no sidecar, no port mapping
• ACID transactions - cache writes either commit fully or not at all
• Built-in expiry via SQL - a simple WHERE expires_at > now clause handles TTL
• Predictable memory usage - the OS page cache manages hot data automatically
• Trivial backups - copy a single file, or use VACUUM INTO for a consistent snapshot
• Identical behavior everywhere - same library in dev, CI, staging, and production

Implementation

The cache interface is deliberately minimal. We store serialized values with an expiration timestamp and let SQLite handle the rest.

import sqlite3
import json
import time
from pathlib import Path
from contextlib import contextmanager

class SQLiteCache:
    def __init__(self, db_path: str = "cache.db"):
        self.db_path = Path(db_path)
        self._init_db()

    def _init_db(self):
        with self._connect() as conn:
            conn.execute("PRAGMA journal_mode=WAL")
            conn.execute("PRAGMA synchronous=NORMAL")
            conn.execute("""
                CREATE TABLE IF NOT EXISTS cache (
                    key TEXT PRIMARY KEY,
                    value TEXT NOT NULL,
                    expires_at REAL NOT NULL
                )
            """)
            conn.execute("""
                CREATE INDEX IF NOT EXISTS idx_expires
                ON cache(expires_at)
            """)

    @contextmanager
    def _connect(self):
        conn = sqlite3.connect(self.db_path, timeout=5.0)
        try:
            yield conn
            conn.commit()
        finally:
            conn.close()

    def get(self, key: str):
        with self._connect() as conn:
            row = conn.execute(
                "SELECT value FROM cache WHERE key = ? AND expires_at > ?",
                (key, time.time()),
            ).fetchone()
            return json.loads(row[0]) if row else None

    def set(self, key: str, value, ttl: int = 3600):
        with self._connect() as conn:
            conn.execute(
                "INSERT OR REPLACE INTO cache (key, value, expires_at) VALUES (?, ?, ?)",
                (key, json.dumps(value), time.time() + ttl),
            )

    def delete(self, key: str):
        with self._connect() as conn:
            conn.execute("DELETE FROM cache WHERE key = ?", (key,))

    def clear_expired(self):
        with self._connect() as conn:
            conn.execute("DELETE FROM cache WHERE expires_at <= ?", (time.time(),))

The clear_expired method runs on a background timer every five minutes. In practice, expired rows are also filtered out on read, so stale data is never returned even if cleanup hasn't run yet.

Benchmarks

We ran 50,000 iterations of get/set operations with 1 KB payloads on the same hardware. Redis was accessed over a Unix socket to give it the best possible local performance.

Read performance favored SQLite since there was no IPC overhead. Write performance was slightly slower due to filesystem sync, but well within our latency budget. Memory usage was dramatically lower - SQLite keeps its working set in the OS page cache, which the kernel manages adaptively.

Migration Steps

1. Define the SQLiteCache class with the same interface as the existing Redis wrapper
2. Run both caches in parallel for one week, comparing hit rates and latencies
3. Add a feature flag to route reads through SQLite while still writing to both stores
4. Monitor error rates and p99 latency during the parallel phase
5. Disable Redis writes once SQLite metrics are confirmed stable
6. Remove the Redis container from the deployment manifests
7. Schedule a periodic clear_expired job and a weekly VACUUM
8. Update runbooks and on-call documentation

The entire migration took three days of implementation and one week of parallel validation.

Results

After running in production for six weeks, the SQLite cache has handled over 12 million requests with zero incidents. Our staging environment hasn't triggered a single memory alert. Deployment is one container instead of two. The on-call team stopped getting paged about cache infrastructure entirely.

The lesson here isn't that SQLite is better than Redis - it's that the right tool depends on the actual problem. We spent two years maintaining infrastructure for a distributed caching layer that was never distributed. Sometimes the simplest solution is the one you already have.