Reference notes ·

Receipt Upload Queue: Reference Notes

The complete design of a durable, idempotent receipt upload path across iOS, Node, and MySQL. Every invariant, every crash window, and the code that holds each one. Companion to “The Expense That Showed Up Twice.”

SwiftiOSNodeMySQLCore Datasystem design
Sections
  1. The project
  2. The product constraints that mattered
  3. Durable, non-blocking uploads
  4. Idempotency: at-least-once delivery, once-only effect
  5. The user report: duplicate expenses
  6. How I approached the investigation
  7. The local-cache fix
  8. Updating the UI without another list request
  9. Queue state and AsyncStream
  10. The final flow
  11. Crash windows: recover instead of simulating a transaction across systems
  12. Orphan JPEG cleanup
  13. Where the AI over-engineered the project
  14. Complexity we kept because it protects a real invariant
  15. The human role in AI-assisted development
  16. The resulting mental model
  17. Interview notes

This is the long version. The short one, with the bug and the argument, is The Expense That Showed Up Twice. I keep this page for myself: it is the whole system laid out so I can walk it in an interview without reconstructing it from memory.

The project

I was adding native expense entry to an internal SwiftUI app for a field sales team. The existing expense data came from Expensify, but the goal was to let people create expenses directly in our app without changing the downstream workflow.

The visible feature sounded modest:

That last group of requirements turned a form-and-camera feature into a small distributed system. There is durable state on the phone, a background transfer managed partly by iOS, a Node backend, MySQL, a receipt image on disk, a Core Data cache, and a SwiftUI list observing that cache. Any boundary between those pieces can fail at an inconvenient moment.

The engineering challenge was real. The equally important lesson was that an AI collaborator will make the architecture more complicated than the product needs unless someone keeps forcing the discussion back to evidence, invariants, and recovery behavior.

The product constraints that mattered

The app is an internal business tool, not a social network or photo library. A receipt image needs to remain legible for review and auditing, but nobody needs a media pipeline optimized for millions of public views.

The existing backend already had an expensify table and a 25-character transactionid column. The clean product decision was to keep using that object and table. An unassigned and an assigned expense are not different kinds of data. Assignment is state on the same expense.

The storage design was deliberately plain:

That was enough. A separate native-expense table, original-and-thumbnail URLs, CDN design, and an image migration layer were all plausible future ideas, but they were not requirements for this version.

Durable, non-blocking uploads

The user should not stare at a spinner while a receipt uploads. Saving should feel immediate, and the user should be able to add another receipt or use the rest of the app.

That means the upload cannot exist only as an in-memory Task. If the process is suspended or killed, an in-memory queue disappears. The app needs durable intent: before it tells the user the receipt was saved, it must have persisted enough information to retry the operation.

For the expected volume, a handful of pending receipts rather than thousands, the persistent queue is intentionally small:

The order of operations when enqueuing is what makes the acknowledgement honest.

Generate UUID idempotency key Write UUID.jpg atomic Append job in memory Write pending.json atomic replace Schedule upload background URLSession durable from here: user sees “Saved” manifest write fails: remove job, delete JPEG, throw
Figure 1. Enqueue order. Nothing is acknowledged until the manifest is on disk, and a failed manifest write undoes everything before it.

If the manifest write fails, the store rolls back the array mutation and removes the new JPEG. The app never acknowledges a save for which it failed to persist the upload intent.

NativeExpenseUploadStore.swift

actor NativeExpenseUploadStore {
    static let shared = NativeExpenseUploadStore()

    private var items: [NativeExpensePendingUpload] = []
    private var hasLoaded = false

    func enqueue(
        id: UUID,
        imageData: Data,
        upload: NativeExpenseUpload,
        activeUsername: String
    ) throws -> NativeExpensePendingUpload {
        try loadIfNeeded()

        // A retry of the same client UUID returns the job that already exists.
        if let existingItem = items.first(where: { $0.id == id }) {
            return existingItem
        }

        let item = NativeExpensePendingUpload(
            id: id,
            activeUsername: activeUsername,
            imageFilename: "\(id.uuidString.lowercased()).jpg",
            upload: upload,
            createdAt: .now,
            status: .queued,
            retryCount: 0
        )

        try FileManager.default.createDirectory(at: Self.directory, withIntermediateDirectories: true)
        try imageData.write(to: item.imageURL, options: .atomic)
        items.append(item)
        do {
            try saveManifest()
        } catch {
            // Never acknowledge a job whose intent is not on disk.
            items.removeLast()
            try? FileManager.default.removeItem(at: item.imageURL)
            throw error
        }
        return item
    }

    func remove(id: UUID) throws {
        try loadIfNeeded()
        guard let index = items.firstIndex(where: { $0.id == id }) else { return }
        let item = items.remove(at: index)
        do {
            try saveManifest()
        } catch {
            items.insert(item, at: index)
            throw error
        }
        // Manifest first, then the image. A crash here leaves an orphan
        // file, not a job whose image is missing.
        try? FileManager.default.removeItem(at: item.imageURL)
    }

    private func saveManifest() throws {
        let manifestURL = Self.directory.appendingPathComponent("pending.json")
        let data = try JSONEncoder().encode(items)
        try data.write(to: manifestURL, options: .atomic)
    }
}

Why JSON instead of SQLite?

SQLite would be a good answer for a large or relational job system. It supports transactional updates, indexed queries, many jobs, dependencies, priorities, and potentially multiple processes. Upload queues at much larger companies use SQLite for exactly those reasons.

It would also have been unnecessary machinery here. Rewriting a tiny JSON metadata array is cheap. The image bytes are separate files, so changing a status does not rewrite the image. An actor serializes access inside the process, and atomic replacement prevents readers from seeing a half-written manifest.

The trade-off is explicit: this design is simple because the queue is small. If the workload changed materially, SQLite would become the better choice.

Idempotency: at-least-once delivery, once-only effect

A durable queue necessarily retries. A timeout does not tell the client whether the server failed before creating the expense or succeeded and lost the response. Retrying with a new identity could create a second expense.

The client therefore generates one UUID when the job is created and persists it with the job. Every retry carries the same UUID. That UUID is the idempotency key.

The backend must make the idempotency decision atomically. Checking first in application code and inserting second is not sufficient: two Node processes behind a load balancer could both observe that no row exists. The MySQL unique constraint is the concurrency authority. One insert wins. A competing insert with the same transaction ID hits the unique key, and the handler returns the expense that already exists.

The existing transactionid column only allows 25 characters, so the server deterministically maps the UUID to the schema:

client UUID  →  SHA-256  →  first 22 hex characters  →  "cp-" prefix

That produces exactly 25 characters. Twenty-two hex characters provide 88 bits of identity space, far beyond what this product needs. The database uniqueness constraint still decides the vanishingly unlikely collision case.

Controllers/nativeExpenseRouter.js

function transactionIDFor(uploadID) {
  const digest = crypto
    .createHash("sha256")
    .update(uploadID)
    .digest("hex")
    .slice(0, 22);
  return `cp-${digest}`;
}

router.post(
  "/",
  express.raw({ type: "image/jpeg", limit: maximumReceiptBytes }),
  async function (req, res, next) {
    try {
      const metadata = requestMetadata(req);
      const username = req.user.name;
      const uploadID = clientUploadID(req);
      const transactionid = transactionIDFor(uploadID);

      // Fast path: this UUID already produced an expense. The client lost
      // the response or is retrying; hand it the same object again.
      const existingExpense = await Expensify.findCanonicalOwned(
        transactionid,
        username,
        req.dbPool
      );
      if (existingExpense) {
        return res.status(200).json(canonicalExpenseResponse(existingExpense));
      }

      await ReceiptStorage.write(transactionid, req.body);

      let createdExpense;
      try {
        createdExpense = await helpers.withTransaction(req.dbPool, async (connection) => {
          await Expensify.createUploadedExpense(
            { transactionid, username, ...metadata, url: ReceiptStorage.receiptURL(transactionid) },
            connection
          );
          return Expensify.findCanonicalOwned(transactionid, username, connection);
        });
      } catch (error) {
        // Two processes raced past the fast path. The unique key on
        // transactionid is the arbiter; the loser returns the winner's row.
        if (error.code === "ER_DUP_ENTRY") {
          const duplicate = await Expensify.findCanonicalOwned(transactionid, username, req.dbPool);
          if (duplicate) {
            return res.status(200).json(canonicalExpenseResponse(duplicate));
          }
        }
        throw error;
      }

      res.status(201).json(canonicalExpenseResponse(createdExpense));
    } catch (error) {
      next(error);
    }
  }
);

In a payment system, the idempotency key and the resource ID are often separate. The server stores a mapping from (caller, idempotency key) to the payment or order it created. Here the deterministic transaction ID lets one column serve both roles.

A simplification I had to insist on

The first server mapping hashed the username together with the UUID. That was defense in depth, but it did not protect a realistic requirement. A properly generated UUID already has negligible collision risk, the operation belongs to an authenticated user, and the database has a unique constraint. Including the username made the mapping harder to explain and created extra code without changing a meaningful outcome.

The simplified rule is easier to reason about: the same client UUID always maps to the same server transaction ID. The iOS side carries the same derivation so the client can predict the ID before the server ever sees the upload.

NativeExpenseUpload.swift

public enum NativeExpenseIdentity {
    /// Must remain byte-for-byte compatible with the backend's
    /// `transactionIDFor(uploadID)` implementation.
    public static func transactionID(clientUploadID: UUID) -> String {
        var hasher = SHA256()
        hasher.update(data: Data(clientUploadID.uuidString.lowercased().utf8))
        let digest = hasher.finalize().map { String(format: "%02x", $0) }.joined()
        return "cp-\(digest.prefix(22))"
    }
}

The user report: duplicate expenses

The feature shipped to beta testers in version 2.63. On the first day, two users reported that expenses appeared twice, and one user saw the problem multiple times. A screenshot showed visually identical pairs: the same merchant, comment, amount, and date.

I could not reproduce it on my own phone.

The tempting responses were all guesses:

The AI leaned quickly toward speculative guards: disable rapid taps, deduplicate on merchant, amount, and date. I pushed back. A double-tap guard is harmless as UI polish, but it cannot substitute for idempotency. A content hash is actively dangerous. Two legitimate lunches can have the same merchant, amount, and date, and we should not silently discard a valid business expense because it resembles another one.

The first question had to be more basic:

Are there two server expenses, or is one server expense represented twice in the local cache?

That distinction changes the entire investigation.

How I approached the investigation

Separate the identities

The system contains several things that can look like “the expense.” Naming them is most of the work.

PHONE SERVER PHONE Upload UUID pending.json POST Transaction ID cp- + 22 hex INSERT MySQL row UNIQUE key refresh Core Data object …or two of them observe SwiftUI row what you see different IDs → the server made two same ID on both → the cache doubled it
Figure 2. The five things that can look like "the expense." One question, which stage holds two of them, decides the whole investigation.

I added the transaction ID unobtrusively at the bottom of the expense detail view. If a user reports duplicates, I can ask for both IDs:

That tiny debugging affordance is worth more than pages of speculation.

Inspect the backend evidence

The server path already had an idempotent transaction ID and a MySQL uniqueness constraint. The evidence did not support a simple rapid double-tap explanation for all of the reported rows. At least part of the problem could be explained by one server object becoming multiple local managed objects.

This did not prove the complete history of every user report. It identified a concrete defect that could create exactly the visible symptom without requiring a duplicate server upload.

Trace the local refresh path

Core Data was being used as a server-backed cache. SwiftUI’s @FetchRequest observes that cache. It does not observe the server.

The list refresh is long-standing code that parses server data on a private child context, and for years it had a single caller. The new upload queue added a second one: a full list refresh after every completed upload. Two overlapping refreshes could each fetch before the other’s insert became visible, conclude that an expense did not exist, and insert separate managed objects with the same transaction ID. When both changes reached the parent context, SwiftUI displayed duplicate rows.

BEFORE · overlapping refreshes on private contexts A fetch → miss insert B fetch → miss insert parent context two objects, one transactionid AFTER · one in-flight refresh, main context, unique constraint A fetch → miss insert B waits for A fetch → hit update view context UNIQUE (transactionid) merges any overlap serialization removes the race; the constraint holds the invariant if some other path overlaps anyway
Figure 3. The duplicate-row race, and the two-layer fix. Serialized refreshes prevent the conflict. The uniqueness constraint guarantees the invariant regardless.

Running the work on private contexts kept parsing off the main thread, but it also expanded the concurrency surface. The optimization was not free.

The local-cache fix

The next model version made Expensify.transactionid a Core Data uniqueness constraint. That gives the cache the same invariant the backend already had: no two expense objects may share a transaction ID.

The complete fix included more than adding the model constraint:

DataManager.swift

public init(inMemory: Bool = false) {
    // ...load the model and point the store at the app group container...

    if let storeURL {
        rebuildPersistentStoreForTransactionIDConstraintIfNeeded(at: storeURL, model: model)
    }

    container.loadPersistentStores { _, error in
        if let error { fatalError("Error: \(error.localizedDescription)") }
    }
    container.viewContext.mergePolicy = NSMergePolicy.mergeByPropertyObjectTrump
}

/// Rebuilds the server-backed cache once before enabling the uniqueness
/// constraint. Credentials and pending receipt uploads live elsewhere and
/// survive this. Only the app performs it; the widget shares the store
/// but must never destroy it.
private nonisolated func rebuildPersistentStoreForTransactionIDConstraintIfNeeded(
    at storeURL: URL,
    model: NSManagedObjectModel
) {
    guard Bundle.main.bundleURL.pathExtension != "appex" else { return }

    let defaults = UserDefaults(suiteName: Persistance.appGroupName) ?? .standard
    guard !defaults.bool(forKey: Self.rebuildKey) else { return }

    do {
        if FileManager.default.fileExists(atPath: storeURL.path) {
            let coordinator = NSPersistentStoreCoordinator(managedObjectModel: model)
            try coordinator.destroyPersistentStore(at: storeURL, type: .sqlite, options: nil)
        }
        defaults.set(true, forKey: Self.rebuildKey)
    } catch {
        // Leave the marker unset so the next launch retries.
    }
}

The one-time rebuild is appropriate because this Core Data store is a disposable cache of server data. Pending uploads and credentials live elsewhere. Deleting this cache does not lose the source of truth. The app fetches it again.

This distinction matters because “nuke Core Data” would be reckless if Core Data contained unsynchronized user-created data. Here it was a controlled migration strategy for a replaceable cache.

The Core Data constraint and merge policy are the final safety net, not the only concurrency control. Serializing refreshes prevents needless conflicts and makes behavior easier to reason about. The constraint preserves the invariant if an unexpected code path overlaps anyway.

Updating the UI without another list request

The original completion flow uploaded the expense, then issued a full GET for all expenses, wrote that response to Core Data, and waited for the list to notice the change. This created more network work, more refresh overlap, and a longer period in which the pending row could appear stuck.

The improved response contract returns the canonical expense object from the upload endpoint:

The iOS app upserts that one object directly into Core Data. The @FetchRequest observes the save and renders it automatically. The queue then removes the completed job and publishes its new snapshot.

NativeExpenseUploadQueue.swift

private func storeCompletedUpload(id: UUID, response: NativeExpenseUploadResponse) async {
    do {
        try await ExpensifyNetworking.storeUploadedExpense(response)
        guard await hasStoredExpense(transactionID: response.transactionid) else {
            throw NativeExpenseUploadQueueError.localExpenseWasNotStored
        }
        try await store.remove(id: id)
        await publishSummary()
    } catch {
        recordQueueError(error, stage: "store_uploaded_expense")
        // The job is still in the manifest. Retrying reuses the same UUID,
        // so the server answers 200 with the same expense.
        if let item = try? await store.item(id: id) {
            await scheduleRetry(for: item, error: error.localizedDescription)
        }
    }
}

Older app versions remain compatible because Swift’s Decodable ignores unknown JSON keys by default. They decode the transaction ID they expect, ignore the additional expense fields, and continue using their full refresh behavior. That makes a server-first deployment possible.

Queue state and AsyncStream

The UI also needs to know about jobs that are not yet server expenses. A pending upload should look like a row in the expense list, not a vague banner that can sit at “1 receipt uploading” forever.

The queue publishes complete summaries through an AsyncStream. Each subscriber receives the current snapshot immediately and then the newest snapshot whenever queue state changes. Buffering only the newest value is right because the UI cares about current truth, not a replay of every intermediate event.

NativeExpenseUploadQueue.swift

actor NativeExpenseUploadQueue {
    private var updateContinuations: [UUID: AsyncStream<NativeExpenseUploadSummary>.Continuation] = [:]

    func updates() async -> AsyncStream<NativeExpenseUploadSummary> {
        let subscriberID = UUID()
        let (stream, continuation) = AsyncStream.makeStream(
            of: NativeExpenseUploadSummary.self,
            bufferingPolicy: .bufferingNewest(1)
        )
        updateContinuations[subscriberID] = continuation
        continuation.onTermination = { [weak self] _ in
            Task { await self?.removeUpdateContinuation(id: subscriberID) }
        }

        await reconcileSessionState()
        continuation.yield(await currentSummary())
        return stream
    }

    private func publishSummary() async {
        guard !updateContinuations.isEmpty else { return }
        let summary = await currentSummary()
        for continuation in updateContinuations.values {
            continuation.yield(summary)
        }
    }
}

UnassignedExpensesView.swift

.task {
    for await summary in await uploadQueue.updates() {
        pendingUploadSummary = summary
    }
}

This is the modern concurrency mechanism for delivering queue state, but it is important not to over-credit it. AsyncStream did not fix duplicate Core Data rows. It fixed state propagation and made the UI model easier to follow.

Pending rows are visible in the same list as uploaded expenses. While a row is syncing, it is not editable. Tapping it explains that the upload must finish. A job that exhausts automatic retries becomes “needs attention” and offers retry or discard rather than pretending to upload forever. Pull to refresh reconciles both queue state and server-backed expenses.

The final flow

PHONE SERVER Save UUID + JPEG pending.json Background POST same UUID every retry Upsert expense Core Data, unique Remove job publish snapshot pending row visible now @FetchRequest renders it UUID, metadata, JPEG Derive transaction ID cp- + sha256[0..22] Write receipt temp file, rename INSERT row UNIQUE decides 201 new expense 200 existing expense canonical expense JSON timeout or lost response: retry
Figure 4. The complete path. The retry loop and the unique key are what turn at-least-once delivery into an exactly-once expense.

The most useful way to understand it is by assigning one invariant to each layer.

InvariantAuthoritative mechanism
A saved upload survives process deathJPEG plus durable queue manifest
Only one component mutates the in-process queue at a timeSwift actor
iOS can continue the transfer while the app is suspendedBackground URLSession
Retrying a request does not create another server expenseStable client UUID plus MySQL unique transaction ID
One server expense appears once in the local cacheCore Data uniqueness constraint plus serialized reconciliation
SwiftUI reflects current pending workSnapshot-based AsyncStream
SwiftUI reflects saved expenses@FetchRequest observing Core Data
Abandoned local images do not accumulate indefinitelyStartup orphan cleanup

Crash windows: recover instead of simulating a transaction across systems

The manifest, background session, backend database, filesystem, and Core Data store cannot participate in one atomic transaction. The realistic goal is not “no crash can ever occur between two lines.” The goal is that every interrupted state converges safely.

Interruption pointRecovery behavior
Before the manifest is savedEnqueue fails. Array mutation and JPEG are rolled back.
After enqueue, before upload beginsThe durable job is scheduled on the next launch.
After the task is created, before resume()Startup finds the suspended task in the background session and resumes it.
While the request is in flightThe background session may continue. Otherwise the same UUID is retried.
Server commits, response is lostRetry reaches the unique transaction ID and returns the existing expense.
Response arrives before Core Data saveIf the app dies, the pending job remains and safely retries.
Core Data saves before queue removalIf the app dies, retry and upsert use the same transaction ID and the cache constraint merges it.
Manifest removal succeeds before JPEG deletionThe image is orphaned temporarily. Startup cleanup removes it.
Core Data cache is missing or rebuiltA normal list refresh reconstructs it from the server.

The third row is worth a closer look, because it was a real gap. The app creates a background upload task, persists the job as uploading, and then calls resume(). A process death in that narrow interval leaves a suspended task that the session still reports as present. The first version of the startup check counted suspended tasks as live, so the job would sit at “uploading” forever with no retry path. The right response was a small reconciliation rule, not a second persistence subsystem.

NativeExpenseUploadQueue.swift

private func currentSessionUploadIDs() async -> Set<UUID> {
    await withCheckedContinuation { continuation in
        session.getAllTasks { tasks in
            let uploadIDs = tasks.compactMap { task -> UUID? in
                switch task.state {
                case .running:
                    break
                case .suspended:
                    // The process died after creating the task and persisting
                    // the job as uploading, but before calling resume(). Finish
                    // that interrupted intent rather than reporting a stalled upload.
                    task.resume()
                case .canceling, .completed:
                    return nil
                @unknown default:
                    return nil
                }
                return task.taskDescription.flatMap(UUID.init(uuidString:))
            }
            continuation.resume(returning: Set(uploadIDs))
        }
    }
}

Orphan JPEG cleanup

Queue deletion deliberately saves the manifest without the job before deleting its JPEG. If the process dies between those operations, the result is an orphan file, not a manifest entry whose required image is missing. That is the safer failure mode.

On startup, after successfully decoding the manifest, the store builds a set of referenced image filenames and deletes unreferenced files matching the queue’s exact lowercase UUID.jpg naming convention. It does not touch arbitrary JPEGs. If the manifest cannot be decoded, cleanup does not run, because deleting images without trustworthy references would be unsafe.

NativeExpenseUploadStore.swift

private func loadIfNeeded() throws {
    guard !hasLoaded else { return }
    try FileManager.default.createDirectory(at: Self.directory, withIntermediateDirectories: true)

    let manifestURL = Self.directory.appendingPathComponent("pending.json")
    if FileManager.default.fileExists(atPath: manifestURL.path) {
        items = try JSONDecoder().decode([NativeExpensePendingUpload].self, from: Data(contentsOf: manifestURL))
    }

    // Only after the manifest decoded: the references are trustworthy.
    cleanupOrphanedImages()
    hasLoaded = true
}

private func cleanupOrphanedImages() {
    let referencedFilenames = Set(items.map(\.imageFilename))
    let fileManager = FileManager.default
    guard let files = try? fileManager.contentsOfDirectory(
        at: Self.directory,
        includingPropertiesForKeys: nil,
        options: [.skipsHiddenFiles]
    ) else { return }

    for file in files where isManagedReceiptImage(file)
        && !referencedFilenames.contains(file.lastPathComponent) {
        try? fileManager.removeItem(at: file)
    }
}

/// Only files the queue itself would have written: lowercase `UUID.jpg`.
private func isManagedReceiptImage(_ file: URL) -> Bool {
    guard file.pathExtension == "jpg" else { return false }
    let identifier = file.deletingPathExtension().lastPathComponent
    guard let uuid = UUID(uuidString: identifier) else { return false }
    return file.lastPathComponent == "\(uuid.uuidString.lowercased()).jpg"
}

This is a small mark-and-sweep garbage collector. It is fast when there is nothing to do and avoids requiring perfect cleanup timing in every code path.

Where the AI over-engineered the project

The AI was useful for generating code, enumerating failure modes, and discussing concurrency. It also repeatedly treated every imaginable future concern as a present requirement.

It invented a second expense model

An early design introduced a native_expenses table even though the product already had an expensify expense object. That would have created two schemas, mapping logic, migration questions, and ambiguity about which object the app was displaying.

The correction was simple: use the existing table and existing object. Assignment is a state transition, not a different entity.

It designed a photo platform for audit receipts

The early proposal included an original image, a thumbnail, two URLs, possible CDN behavior, and migration machinery. Those are sensible capabilities for a photo-heavy product, but this feature needs one legible receipt image that users rarely reopen.

The simpler design stores one URL. The server controls what is behind it, so storage can still migrate later without forcing the client to know the physical directory.

It claimed implementation before the destination was known

At one point the AI described image uploading as implemented while the actual destination URL and storage directory were still unresolved. The code may have had an upload request, but the end-to-end feature was not implemented until the server contract and durable storage target were concrete and verified.

This became a useful rule: do not accept “implemented” for an integration without asking where the bytes go, how the record points to them, how they are retrieved, and what happens after deployment.

It added the username to the transaction-ID derivation

The client UUID already provided the idempotency identity. Hashing the username with it was extra defense for a collision scenario too improbable to drive this design. It also obscured the central invariant that one client job maps deterministically to one server transaction.

Removing the username reduced code and made retries easier to explain.

It tried to persist the entire server response in the queue

The AI proposed storing the returned expense object in pending.json to cover a crash after the server response but before the Core Data save. That would add another durable state, schema evolution for the response snapshot, and reconciliation branches.

The existing mechanisms already recover:

Persisting the full response solved a theoretical micro-window twice.

It reached for speculative duplicate prevention

The first duplicate theories produced ideas like a double-tap timer and merchant/amount/date hashes before establishing whether the duplicates existed on the server. These could hide symptoms without fixing the system and could reject legitimate equal-valued expenses.

The better move was to surface the transaction ID and inspect each storage layer.

It optimized camera and image processing before locating the blur

When a tester reported blurry photos, proposed capture and backend image changes arrived before we had isolated whether the blur came from capture, compression, storage, transport, or the viewer. Those changes were reverted while zoom support, an independently useful viewer feature, remained.

The lesson is the same: instrumentation and reproduction first, optimization second.

Complexity we kept because it protects a real invariant

“Avoid over-engineering” does not mean deleting all resilience. Several pieces are essential:

The design became simpler without giving up these guarantees.

The human role in AI-assisted development

This project changed how I think about working with AI on production code.

AI is very good at answering “What could go wrong?” It is less naturally disciplined about the next questions:

  1. Did that failure actually happen?
  2. Which invariant would it violate?
  3. Which existing layer already handles it?
  4. What is the smallest additional mechanism that closes the gap?
  5. How will we prove the mechanism works?

I repeatedly had to slow the process down and insist on one concept at a time. That was especially valuable when discussing idempotency, persistent queues, actor reentrancy, Core Data contexts, and server concurrency. The conversation became productive once we stopped treating a pile of code as an answer and started assigning ownership of each invariant.

Some practical habits emerged:

The resulting mental model

The queue provides at-least-once delivery. The stable UUID and unique server transaction ID turn that into an exactly-once business effect. Core Data is a disposable projection of server state, and its uniqueness constraint prevents one transaction from becoming two visible rows. AsyncStream reports the current queue snapshot, while @FetchRequest reports the current cached expense snapshot.

No component alone guarantees the whole experience. The system works because each layer has a narrow responsibility and every non-atomic boundary has a recovery path.

The most important simplification was not a particular API or framework. It was replacing “What code can the AI add for every theoretical failure?” with a better question:

What is the smallest system whose invariants make the user’s data safe and whose state we can actually explain?

That question produced less code, a stronger mental model, and a system that should be easier to operate when the next real bug report arrives.

Interview notes

The same project, compressed into the questions a system design interview would actually ask.

  1. Why must the idempotency key exist before the job is durable? Because the retry has to carry the same identity as the first attempt. If the key were generated at send time, a crash between attempts would produce a new key and a second expense.
  2. How does at-least-once delivery become an exactly-once effect? The client retries freely. The server maps the key to a deterministic resource ID and lets a unique index decide the winner. Retries read the winner’s row back.
  3. Why a database unique index rather than a lock in the Node process? Two processes behind a load balancer share no memory. The database is the only thing both of them talk to.
  4. When is a JSON manifest enough, and when is SQLite justified? JSON when the job count is tiny, the writes are whole-file, and one process owns it. SQLite when you need partial updates, indexes, many jobs, or multiple writers.
  5. What does the actor actually guarantee? Serialized access to in-memory state within one process. Not durability, not cross-process safety, and not atomicity across an await. State checks after every suspension point still matter.
  6. Why can’t the server response, the Core Data save, and the manifest deletion be one transaction? Three different stores with no shared coordinator. Design each boundary to converge on retry instead.
  7. How do you handle a crash at each boundary? Enumerate the boundaries, and for each one, name the state left behind and the code path that repairs it on the next launch.
  8. What did AsyncStream fix, and what didn’t it fix? State propagation to the UI. It had nothing to do with database uniqueness, and claiming otherwise is a red flag.
  9. Duplicate business object or duplicated cache projection? Compare the IDs. Different IDs mean the server has two. Same ID means the cache or the view layer doubled one.
  10. How do you work with an AI on this without losing judgment? Separate evidence from theory, assign an owner to each invariant, and refuse any mechanism that doesn’t close a gap you can name.