Writing

Notes on engineering, AI, and building for the people technology leaves behind.

  • 16 October 2026
    6 min read

    The regulation is a reducer

    Dutyclock turns the drivers' hours rules into a pure function over a log of taps, with golden tests built from the regulator's own worked examples. What it shows is advisory, and it says so.

    Rules EnginesRegulationAndroid
  • 15 October 2026
    6 min read

    Without a TO clause, every policy is public

    A policy without a TO clause applies to PUBLIC, which includes anon. That made KabutarBaazi's listings and profiles readable with a curl and the key inside the APK.

    PostgreSQLRow-Level SecuritySupabase
  • 14 October 2026
    6 min read

    The tuner subtracts its own drone

    A tuner that plays a drone through the speaker can hear itself louder than the player. Beatwheel cancels its own drone with a recursive least squares filter and shows beats near unison.

    AndroidSignal ProcessingAudio
  • 13 October 2026
    6 min read

    Two cards tell you nothing

    Keyfob splits a recovery phrase into guardian cards with Shamir sharing over the AES field. Below the threshold every phrase stays equally likely, and a test proves it from outside the app.

    CryptographySecret SharingAndroid
  • 12 October 2026
    6 min read

    An unsolvable puzzle cannot ship

    Every puzzle in Axiom is a theorem checked by a 758-line kernel on the phone; before a release, tools/par.js runs the kernel's own solver over all 44 theorems and fails if one cannot be finished.

    Natural DeductionProof CheckingSearch
  • 12 October 2026
    6 min read

    The agent never sees the DOM

    Kamapathy hands an agent a bounded text view of a page, 4,000 characters and 40 controls by default, and a fixed set of routes. That view is a product decision with a budget.

    AI AgentsBrowser AutomationElectron
  • 26 September 2026
    5 min read

    The free tier is a lease

    In June 2026 Oracle halved its Always Free Ampere allowance by editing a docs page, and over-limit instances were shut down. A free tier is a lease; place only what runs at half.

    Oracle CloudFree TierLLM Serving
  • 25 September 2026
    6 min read

    Capabilities in code, roles in rows

    Permissions change with the code, so they live in code; roles change with a customer's org chart, so they live in rows. The deploy line is the test for which side each belongs on.

    RBACPostgreSQLMulti-tenant SaaS
  • 24 September 2026
    5 min read

    A pipeline you cannot replay is a spreadsheet

    The real product of a CRM is the history of state changes. If it cannot show the pipeline exactly as it stood at nine last Monday, every historical report is a reconstruction.

    Data ModellingCRMDatabases
  • 23 September 2026
    6 min read

    Base fee takes six blocks to double

    EIP-1559 caps the base fee change at 12.5 percent a block, so it needs about six blocks to double and twenty to rise tenfold. In that lag the tip is the whole auction.

    GasEthereumFee Markets
  • 22 September 2026
    5 min read

    Write the page for two readers

    The 2026 evidence on llms.txt is in, and it says the file is barely read. Every crawler that matters reads the HTML and the structured data, so the work belongs in the page itself.

    SEOStatic SitesAI Crawlers
  • 21 September 2026
    5 min read

    A ticket is a promise with a clock

    Most SLA breaches are accounting failures: a clock paused by the wrong party, never resumed, or two running on one ticket. Give each state one clock and one actor who can stop it.

    TicketingService DeskState Machines
  • 20 September 2026
    5 min read

    The price of leaving is not your problem

    The EU Data Act bans cloud switching charges from January 2027, and the big providers already waive exit egress. For a small SaaS the exit bill never mattered; serving does.

    Cloud CostEgressEU Data Act
  • 19 September 2026
    6 min read

    Which language your AI fails in

    Building AI in India means shipping a product that works in English and fails quietly in the languages most users think in. Find and publish the failure languages before users do.

    AI PolicyIndiaEvaluation
  • 18 September 2026
    5 min read

    The package everyone imports

    In a Turborepo monorepo, CI time is set by the few packages that change often and are imported everywhere: a task's cache key includes every dependency's hash. Find and split them.

    TurborepoMonoreposCI
  • 18 September 2026
    5 min read

    You pay for the context every turn

    In a retrieval chat the bill is input tokens: every turn resends the prompt, passages and history while the model writes a few hundred back. One ratio shows where the cost is.

    CostPrompt CachingRAG
  • 17 September 2026
    5 min read

    The test that reads another tenant's rows

    Tenant isolation in a shared Postgres schema is only as trustworthy as the test that tries to break it. A generated probe turns a convention into a property CI enforces.

    PostgreSQLMulti-tenant SaaSTesting
  • 17 September 2026
    6 min read

    Every external call is a handover

    Reentrancy is a symptom. The event underneath is control leaving your contract while state is half-written. Count those moments per function and audit the count.

    SoliditySecurityAudits
  • 16 September 2026
    6 min read

    The fifth custom field is a table

    Custom fields postpone a data model decision, and vendors sell the ceiling as a feature. Fields that are filled together and empty together are a table waiting to be named.

    Data ModellingCRMPostgreSQL
  • 15 September 2026
    5 min read

    Two dials on the rate limiter

    A per-tenant limit stops tenants hurting each other; a per-principal limit stops one bug hurting you. They come from different numbers, and one dial with one number gives neither.

    Rate LimitingAPI DesignMulti-tenant SaaS
  • 14 September 2026
    6 min read

    Postgres counts processes, not requests

    Connection exhaustion under serverless is a topology problem. Postgres runs one backend per connection, every instance carries a pool, and the platform sets the instance count.

    PostgreSQLServerlessNext.js
  • 13 September 2026
    5 min read

    The text never leaves the phone

    deAIfy strips the tells of machine writing on the device and calls a model only with the user's own key. Making that the default, not the fallback, changes three things at once.

    PrivacyReact NativeOn-Device
  • 13 September 2026
    6 min read

    The backflow ratio finds your fake stages

    A pipeline stage should record a fact proven about a deal. When deals keep moving backwards out of a stage, it was recording a mood, and the proof is in a table every CRM keeps.

    CRMSales PipelinesData Modelling
  • 12 September 2026
    5 min read

    Idempotency keys pointed the other way

    A payment API's idempotency key and a gate's anti-replay check are the same primitive with opposite second answers. Seen as one mechanism, the design collapses to two questions.

    API DesignIdempotencySecurity
  • 11 September 2026
    5 min read

    Your upload key is disposable

    Under Play App Signing the two keys are different classes: the upload key proves you are you and can be reset; the signing key is what devices trust. In 2026 the split sharpened.

    AndroidRelease EngineeringPost-Quantum
  • 11 September 2026
    5 min read

    The part of your schema no grammar checks

    A JSON Schema is compiled to a grammar for constrained decoding, and several keywords cannot survive the compilation. Know which, and validate that residue after decoding.

    Structured OutputJSON SchemaValidation
  • 10 September 2026
    5 min read

    The spread that survives the fees

    Most cross-venue spreads a detector flags are gross spreads. After two taker fees, two slippage terms, transfer cost and decay, the number that matters is what survives.

    ArbitrageMarket MicrostructureShayanomaly
  • 9 September 2026
    6 min read

    Glamsterdam removes the relay, not the builder

    ePBS deletes the relay as a trusted party and makes the builder's payment unconditional. The builder still sees every bundle, and gains an in-protocol way to drop the block.

    MEVEthereumGlamsterdam
  • 9 September 2026
    5 min read

    The engine cannot see the screen

    A rules engine that compiles without the game framework is the cheapest anti-cheat, test suite and replay system you can build. Turup Chaal's is plain C#; the rest followed.

    UnityArchitectureTesting
  • 8 September 2026
    6 min read

    The install script is the attack

    Every big package compromise since 2018 has used one mechanism: code that runs at install time with your credentials. Count those scripts in every lockfile and drive them to zero.

    Supply ChainNpmSecurity
  • 8 September 2026
    5 min read

    Five order books, one frame budget

    A terminal on five venues receives more order-book updates than it can paint; rendering per message freezes the tab when a trader needs it. The unit of work is the frame.

    WebSocketsRealtimeFrontend Performance
  • 7 September 2026
    5 min read

    Stateless tokens still need a kill switch

    Every token system has a revocation lag, and a JWT sets it to the token lifetime. Write the allowed lag down per endpoint class and the stateless debate turns into arithmetic.

    AuthenticationJWTSecurity
  • 7 September 2026
    5 min read

    The gap between signing and verifying

    Signature checks fail far more often from re-serialisation than from cryptography: the sender signed one byte string and the receiver re-encoded it into another before checking.

    SecuritySigningWebhooks
  • 6 September 2026
    6 min read

    Treat the model like a game client

    An agent's tool calls are intents from a client you do not control. Games solved that years ago: validate every intent on the host, and send each client only what it may see.

    AgentsTool UseSecurity
  • 6 September 2026
    5 min read

    The key that is never assembled

    Shamir rebuilds the private key in one process every time you sign, so it protects the key at rest and abandons it when it matters. Threshold signatures never assemble it.

    CryptographyMPCWallets
  • 5 September 2026
    2 min read

    The hand you were never dealt

    How Turup Chaal stops card-game cheating by construction. The host validates every move and never sends a client a card it should not know about.

    MultiplayerUnityAnti-cheat
  • 5 September 2026
    6 min read

    An app with no server has no outages

    Habits keeps every record in a Room database on the phone: no account, no sync, no backend. That removes whole categories of work and two things users want. Pricing both sides.

    AndroidOffline-FirstProduct Strategy
  • 4 September 2026
    5 min read

    A widget is a photograph of your app

    A home-screen widget is not a running view; it is a picture the launcher shows until you send a new one. Design for the age of the picture, not for the code that drew it.

    AndroidJetpack GlanceUX
  • 3 September 2026
    5 min read

    The keys only I hold

    For an engineer-founder the dangerous list is not the tasks only they can do but the credentials only they hold. Delegate by irreversibility, not by how often a key is used.

    FoundingSecurityAndroid
  • 2 September 2026
    5 min read

    Every paper has one load-bearing number

    A paper's claim rests on one number that, a fifth worse, would have sunk it. Find that number first, ask four questions about how it was measured, and read the rest as context.

    ResearchReading PapersEvaluation
  • 2 September 2026
    5 min read

    When a custom CRM is the wrong answer

    I build CRMs for a living. Most teams should not hire me. A custom CRM is justified by one missing verb, never by the price on the vendor's page.

    CRMBuild Vs BuyContract Work
  • 1 September 2026
    5 min read

    An escrow bot that cannot see faces

    Anonymity in a mediated trade is not a property of the transport. It is a property of one table, the map from a relayed message to its sender, and every feature is judged by it.

    TelegramPrivacySystem Design
  • 31 August 2026
    5 min read

    Play reads your manifest before your users do

    Every permission line in an Android manifest is a promise Google Play reviews before a user sees the app, with less context than the user has. The account carries the risk.

    AndroidGoogle PlayRelease Engineering
  • 30 August 2026
    5 min read

    One percent of users cannot tell you much

    A staged rollout is a statistical test, and at one percent the sample is too small to see the crash-rate rise Play penalises. Write each halt rule from what the stage can detect.

    AndroidStaged RolloutsRelease Engineering
  • 29 August 2026
    5 min read

    An AMM is a limit order you cannot cancel

    A constant-product pool is a continuum of resting limit orders, and the liquidity provider's defining disadvantage is that none of them can be pulled when the price moves.

    AMMDeFiMarket Making
  • 28 August 2026
    6 min read

    Eight apps and one release spine

    A portfolio of small apps is only cheaper than one big app if shipping each costs almost nothing. Play charges that cost per app, per year; a shared release system pays it once.

    AndroidRelease EngineeringProduct Strategy
  • 27 August 2026
    5 min read

    The first thing a quantised model forgets

    Quantised models recover over 99 percent of accuracy on average, and the average hides where the loss lands. A canary set from the classes that fail first should choose the format.

    QuantisationInferenceEvaluation
  • 21 August 2026
    6 min read

    A citation is a claim about a passage

    A citation marker asserts that this passage supports this sentence, and it fails on its own schedule. Make the model commit to spans, check them outside the model, drop the rest.

    RAGGroundingCitations
  • 18 August 2026
    6 min read

    On Solana the caller writes your argument list

    An EVM contract reads its own storage; a Solana program is handed every account it touches by the caller, so the account list is untrusted input. Count what you accept unchecked.

    SolanaAnchorSecurity
  • 15 August 2026
    5 min read

    Chain of thought is the memory transformers lack

    A fixed-depth transformer has no working memory that grows with the input, so it cannot recognise regular languages at any length. Chain of thought writes that memory as tokens.

    TransformersChain Of ThoughtFormal Languages
  • 12 August 2026
    5 min read

    A liquidation is a race funded by the loser

    On Aave the liquidation bonus comes out of the borrower's collateral, so the race only runs when the bonus beats gas. Below that line positions rot into bad debt.

    AaveDeFiLiquidations
  • 11 August 2026
    5 min read

    What a student workshop actually rewards

    The ACL 2026 Student Research Workshop took 130 of 402 submissions. Its public process says what it selects for: one thing proved, one measured, one related to the field.

    ResearchACLWriting
  • 5 August 2026
    5 min read

    Your backtest saw the next tick

    Most leakage in time-ordered data is not a leaked column but a leaked clock. Features joined by when an event happened, not when your system could have known it, leak the future.

    EvaluationData LeakageTime Series
  • 22 July 2026
    5 min read

    The kernel you share with a stranger

    A container is a process in a costume; it shares the host kernel with everything else on the box, so its isolation is the kernel's bug count. Author and lifetime pick the boundary.

    FirecrackerSandboxingAI Agents
  • 18 July 2026
    5 min read

    The last passage that mattered

    Retrieval depth is set by feel and never touched, yet every extra passage is prefill and latency on every turn. Measure the rank of the last passage an answer used, then set k.

    RAGRetrievalLatency
  • 14 July 2026
    5 min read

    What happens when you stop giving

    Engineers give the way they ship: pilots, dashboards, scale. Children's needs recur, and the damage is done by a gift that stops. The first question is about the month after.

    PhilanthropyFoundingChildren
  • 11 July 2026
    5 min read

    Kubernetes is a payroll decision

    For a team of a few engineers, the cost of Kubernetes is not compute. It is a second product with its own release train, and the question is who gets paged for it.

    KubernetesSmall TeamsOperations
  • 8 July 2026
    6 min read

    Your migration tool does not know about locks

    Prisma and Drizzle run the SQL you give them and neither sets a lock timeout. The danger of a migration is the lock it takes and the queue behind it, not the size of the change.

    PostgreSQLMigrationsPrisma
  • 6 July 2026
    5 min read

    When the host leaves the table

    Host-authoritative multiplayer has one weakness: the match lives in one player's process. Per-player secrecy and host migration pull in opposite directions. Choose on purpose.

    MultiplayerPhotonNetcode
  • 5 July 2026
    5 min read

    A book that looks alive can be dead

    A local order book fed by WebSocket deltas keeps updating while being wrong. One dropped message desynchronises every level it touched, and a live stream proves nothing.

    WebSocketsOrder BooksCCXT
  • 4 July 2026
    5 min read

    What survives a pg_dump

    Disk encryption defends against a stolen drive; databases are breached by a dump run with the app's own credentials. Sort each column by whether it stays unreadable in that dump.

    PostgreSQLEncryptionThreat Modelling
  • 19 June 2026
    5 min read

    Two models your eval cannot tell apart

    A benchmark's item count sets its resolution. On HumanEval a six-point gap between two models is a tie, and most leaderboard gaps are smaller than that.

    EvaluationStatisticsBenchmarks
  • 12 June 2026
    6 min read

    Estimates are a count of open questions

    Estimation error comes from the questions nobody asked, not from optimism about visible work. Price the questions, not the features; no fixed price while the client's stay open.

    Contract WorkEstimationConsulting
  • 7 June 2026
    5 min read

    What the letters after my name owe

    Postnominals like FRSA and FRGS are read as credentials, and they are not. Both societies began by owning a problem, and a fellowship is what the fellow owes that problem.

    FellowshipsHistoryPhilanthropy
  • 6 June 2026
    5 min read

    Four parsers, one grammar, no winner

    CYK, Earley, LR(1) and recursive descent are usually ranked by asymptotic cost, which is the wrong axis. The right axes are how often the grammar changes and how ambiguous it is.

    ParsingStructured OutputAlgorithms
  • 31 May 2026
    5 min read

    Drift is a speed problem

    Drift is treated as a discipline failure: someone touched the console. It is a race. People use the console when code is slower than the emergency, so the fix is in the pipeline.

    TerraformInfrastructure As CodeCloud
  • 15 May 2026
    5 min read

    The agreement ceiling is not a ceiling

    Folk wisdom says a model cannot beat inter-annotator agreement. It can, because agreement compares two noisy people while the model is scored against a majority better than either.

    LabellingData QualityStatistics
  • 13 May 2026
    5 min read

    Price the outage, not the hour

    For infrastructure and security work the hour is the wrong unit. The client is buying a lower probability of an expensive failure, which is what an insurer sells, with arithmetic.

    PricingConsultingReliability
  • 7 May 2026
    5 min read

    What hardware-backed actually promises

    A key in the Android Keystore cannot be copied off the device, and that is the whole promise. An attacker with root can still use it. Most vault designs quietly assume more.

    AndroidKey ManagementHardware Security
  • 6 May 2026
    6 min read

    Every AES-GCM key has a collision horizon

    With a random 96-bit nonce, AES-GCM's safety is a countdown, not a property. Compute the countdown for every key, and design so that it can never be reached.

    CryptographyAES-GCMAndroid
  • 1 May 2026
    5 min read

    German needs a third more room

    Translating an app into nine languages is a layout problem before it is a language problem. Short labels grow by a third or more and plurals go from two forms to six.

    LocalisationAndroidUX
  • 24 April 2026
    5 min read

    The cube nobody actually pays

    CYK is quoted as cubic in sentence length. For a treebank-scale grammar the grammar term dominates until sentences are longer than people write, so shrink the grammar first.

    ParsingAlgorithmsNLP
  • 8 April 2026
    5 min read

    The client pays for the research

    Contract work is not a distraction from the product if every contract is chosen for what it leaves behind. A rule for picking which paying work a small product company should take.

    FoundingConsultingProduct Strategy
  • 3 April 2026
    5 min read

    What breaks when you leave Salesforce

    Migration cost is about references, not rows. Contacts export cleanly. What breaks is the graph, and you can measure how much will break in an afternoon, before you quote.

    CRM MigrationData ModellingSalesforce
  • 31 March 2026
    5 min read

    Your ticket does not need a blockchain

    A ledger earns its cost only when several parties who distrust each other must write to it. A venue's ticketing has one writer, and one writer means a signature and a row.

    ArchitectureBlockchainTicketing
  • 30 March 2026
    5 min read

    A QR ticket is a bearer token that walks

    A signature on a ticket proves it was issued, not that it is unused. Replay protection is a state problem, and it lives in one atomic write at the moment of admission.

    SecurityQR TicketingSystem Design
  • 26 March 2026
    5 min read

    The lockfile your pipeline never had

    Package managers got lockfiles a decade ago; CI workflows did not, which is how one moved tag reached 23,000 repositories. Would your build survive the marketplace vanishing?

    GitHub ActionsSupply ChainCI/CD
  • 24 March 2026
    6 min read

    Fine-tuning taught it to sound sure

    Fine-tuning makes a model more accurate and less calibrated at once: cross-entropy keeps rewarding larger logits after the answers stop changing. Refit the temperature last.

    CalibrationFine-TuningProbability
  • 23 March 2026
    5 min read

    What Chomsky normal form costs you

    Converting a grammar to Chomsky normal form is taught as a formality. It can multiply the rule count many times over, and the unit-rule step, not binarisation, does the damage.

    Formal LanguagesParsingGrammars
  • 22 March 2026
    5 min read

    The parse that quietly became zero

    A Viterbi parse multiplies one probability per rule. In float32 the best parse of a typical sentence underflows to zero and the argmax is a tie. Log space is not an optimisation.

    ParsingNumerical MethodsNLP
  • 21 March 2026
    5 min read

    Counting people out is the hard part

    Gateless entry gives a perfect entry count and no exit count, so live occupancy is an estimate that drifts up all day. Show the drift, and let it decide when to count exits.

    TicketingAnalyticsQueueing
  • 20 March 2026
    5 min read

    The pumping lemma as a model test

    The pumping lemma is a doubling argument: a finite-memory recogniser that accepts a long string must accept it with the middle repeated. It tests a model for grammar versus memory.

    Formal LanguagesGeneralisationTransformers
  • 19 March 2026
    5 min read

    Compound PCFGs leak context on purpose

    A compound PCFG keeps the tree context-free so the inside algorithm still works, and lets context in through a per-sentence latent that rewrites rule probabilities.

    Grammar InductionPCFGNLP
  • 17 March 2026
    5 min read

    Secrets have a half-life

    Leaked credentials keep working for years. The number that matters is not how many leak but how fast a leaked one stops working, and a secrets programme should be built on that.

    Secrets ManagementCI/CDSecurity
  • 8 March 2026
    5 min read

    Every wrong answer has an exchange rate

    Accuracy rewards guessing. The fix is a penalty for wrong answers, and the penalty is not a research constant. It is a business number, and it sets the abstention threshold.

    HallucinationEvaluationHelpdesk
  • 1 March 2026
    6 min read

    One shared hour with Canada is enough

    From New Delhi, Kuala Lumpur overlaps my working day by six and a half hours and Dubai by seven and a half. Toronto and Vancouver overlap it by zero; the shared hour is made.

    Remote WorkContract WorkTime Zones
  • 27 February 2026
    5 min read

    The queue that joins your transaction

    The reason to run background jobs from a Postgres table is not that it is cheaper than Redis. It is that the enqueue can share a transaction with the row that caused it.

    PostgreSQLBackground JobsRedis
  • 8 February 2026
    6 min read

    The phone in the house is not the child's

    About 90 percent of rural Indian teenagers have a smartphone at home; a third own one. Learning apps run on borrowed, shared devices, and that changes what offline-first means.

    Digital AccessEducationAndroid
  • 28 January 2026
    6 min read

    What a grammar can never check

    Constrained decoding guarantees an output's shape, never its reference. Equality of two strings is the copy language, which no grammar recognises; a checker after decoding must.

    Structured OutputFormal LanguagesConstrained Decoding
  • 17 January 2026
    5 min read

    The tokeniser does not know your grammar

    Constrained decoding is sold as a parser masking logits, but the model emits tokens that ignore the parser's boundaries. The tokens that straddle a boundary set the cost.

    Structured OutputTokenisationLLM Inference
  • 13 January 2026
    5 min read

    The user reads at five tokens a second

    For a streamed answer, decode speed only needs to beat the reader. Adults read about 238 words a minute, so speed above a floor is invisible; spend the capacity elsewhere.

    InferenceLatencyLLM Serving
  • 18 December 2025
    5 min read

    A LoRA adapter is a poor place for facts

    LoRA updates are low-rank and learn less new knowledge than full fine-tuning. Adapters hold format well and facts badly, and facts change faster than anyone retrains.

    Fine-TuningLoRARAG
  • 13 December 2025
    5 min read

    The flash loan is free until it reverts

    Aave's premium is the smallest cost a flash-loan arbitrageur pays. The real cost is gas burned on attempts that revert, and that cost depends on how the transaction was sent.

    Flash LoansAaveMEV
  • 17 November 2025
    5 min read

    Autoscaling buys you the next spike

    Reactive autoscaling cannot serve the spike that triggers it. The new capacity arrives after a chain of delays that add up to minutes. What it buys is the spike after this one.

    AWSAutoscalingCapacity
  • 10 November 2025
    6 min read

    The third adult in every classroom app

    AI tutors solve the content bottleneck, but the best Indian evidence for software tutoring came from staffed centres. The unpriced input is the adult who makes the child sit down.

    EducationAI TutorsPhilanthropy
  • 4 November 2025
    5 min read

    The pumping lemma is pigeonhole on trees

    Students learn the pumping lemma as a five-variable spell. It is one observation: a tall parse tree repeats a nonterminal, and a repeated nonterminal is a subtree you can copy.

    Formal LanguagesProofsNLP
  • 3 September 2025
    6 min read

    There are two gaps, not one

    Every write-then-publish has two places to crash. The outbox pattern closes only the first, so a team that stops there has traded lost events for duplicated ones.

    Event-DrivenPostgreSQLReliability
  • 1 September 2025
    5 min read

    Your similarity threshold has no zero

    A cosine threshold of 0.8 is not 80 percent of anything. Every embedding model puts unrelated text at its own baseline, and a threshold only means something measured above it.

    EmbeddingsVector SearchRAG
  • 22 August 2025
    5 min read

    Timeouts run in two directions

    Through a proxy chain, idle timeouts must get longer as you go inward and deadlines must get shorter. Most timeout bugs are one of those staircases built the wrong way round.

    Reverse ProxiesNode.jsReliability
  • 28 July 2025
    6 min read

    The number that grows before the loss spikes

    Loss spikes in transformer runs are preceded by growth of the largest attention logit. QK-norm, z-loss and QK-clip bound that quantity, so log it rather than waiting for the loss.

    Training DynamicsTransformersNumerics
  • 9 June 2025
    5 min read

    Restore time is the only backup metric

    Backup success rate is a vanity number. The number that decides whether a company survives is how long a restore takes from nothing, and in every public postmortem it ran long.

    Disaster RecoveryBackupsPostmortems
  • 3 May 2025
    5 min read

    Your test set has been seen before

    Near-duplicates across the split boundary are the commonest reason a held-out score is not a generalisation score. Dedupe before you split, and report the overlap next to accuracy.

    Data PipelinesEvaluationDeduplication
  • 24 April 2025
    5 min read

    Publish the denominator

    Impact reports count outputs and drop two numbers: how many were eligible, and what would have happened anyway. A metric with no baseline would not ship, so do not fund one.

    PhilanthropyMeasurementEvidence
  • 20 April 2025
    5 min read

    The judge shares the defendant's blind spots

    An LLM judge is trained on nearly the same data as the model it grades, so their errors are correlated. Judge scores inflate on the hard items; sample the calibration set there.

    EvaluationLLM-as-Judge
  • 7 April 2025
    5 min read

    Nutrition has no app

    Technologists fund education because it looks like an information problem and skip nutrition because it does not. The evidence on school meals says that instinct is backwards.

    PhilanthropyNutritionEducation
  • 15 February 2025
    5 min read

    Four bits are enough, three are not

    Quantisation error is a group's range divided by its levels, and a few outlier weights set the range for everyone. Sixteen levels leave the bulk enough resolution; eight do not.

    QuantisationLinear AlgebraInference
  • 9 January 2025
    5 min read

    Stale is a feature

    Most origins that fall over could have kept serving for hours if the CDN had been told a slightly old answer was acceptable. Write that tolerance down per route, in one header.

    CDNCachingCloudflare
  • 20 August 2024
    6 min read

    A content update is code

    CrowdStrike staged its sensor code but shipped the Channel File 291 update, filed as data, with no canary. Anything that changes what a fleet executes is code and ships like code.

    CrowdStrikeStaged RolloutsPostmortems
  • 14 May 2024
    6 min read

    The tarball is not the repository

    The xz backdoor's trigger was never in git, only in the signed release tarballs that distributions compiled. Build from the tag, and treat test files as code.

    Supply ChainOpen SourceRelease Engineering

Get new posts by email

Occasional essays on engineering, AI, and building for the people technology leaves behind.

One email per new post. Unsubscribe any time.

Subscribe with RSS