Production notes from a builder who ships, operates, and occasionally breaks his own AI systems.
Latest
Continuous batching schedules LLM inference requests at each token generation step instead of locking the batch. This lets one GPU serve dozens of concurrent users by evicting finished sequences and admitting queued ones mid-flight, trading per-user latency for throughput.
Data modeling is designing which facts live where. Learn the three layers (conceptual, logical, physical), normalization vs. denormalization, and why your schema is the contract every pipeline depends on.
Your API falls over at 200 concurrent users while Postgres sits at 4% CPU. That paradox is what connection pooling exists to fix. Here's what opening a database connection actually costs, the max_connections=100 wall, how a pool turns connections into a borrowed-and-returned resource, why it's a queue and not a multiplier, how to size it, and why 'pool exhausted' is almost always a slow query — with code, and the same idea applied to LLM serving.
Mixture of Experts (MoE) lets a 400B-parameter model compute like a 40B one by routing each token to only 2 of 8 expert sub-networks. Learn how sparse routing trades memory for compute efficiency.
Learn why your LLM output drifts in production: zero-shot asks without examples, few-shot pastes worked examples so the model copies your exact format. Same weights, different prompt.
You write one query to list 100 products, and the database quietly runs 101. That is the N+1 problem — and the fix is not an index. Here's why a query's real cost is the round-trip, why an index makes ONE query fast but can't change how many you issue, how a single JOIN collapses 101 queries to 1, and why the same shape shows up in REST calls, GraphQL resolvers, and 500 sequential LLM awaits.
The same query, 4.2 seconds then 3 milliseconds — and the only thing that changed was one line of SQL. Most explanations stop at 'it's like the index in a book.' This one goes a level below: what a table actually is on disk, why a full table scan is the database's only option without an index, how a B-tree gets you there in three hops, and the cost nobody mentions — every write has to update every index.
HNSW turns nearest-neighbor search from brute-force (2M comparisons, 1.2s) into a hierarchical graph walk (1.8k comparisons, 2ms). Learn how greedy navigation trades exact answers for speed.
500 LLM calls in a for-loop takes 17 minutes — and your CPU does nothing for all of it. await is not concurrency. Here's how asyncio.gather puts every request in flight at once, why it's named for the results and not the speed, and why asyncio.Semaphore is the one line that stops you from DDoS-ing yourself into a wall of 429s.
Production-shaped code you can clone, read, and run — the same patterns I write about.
