Reference notes ·
Receipt Upload Queue: Reference Notes
The complete design of a durable, idempotent receipt upload path across iOS, Node, and MySQL. Every invariant, every crash window, and the code that holds each one. Companion to “The Expense That Showed Up Twice.”
Sections
- The project
- The product constraints that mattered
- Durable, non-blocking uploads
- Idempotency: at-least-once delivery, once-only effect
- The user report: duplicate expenses
- How I approached the investigation
- The local-cache fix
- Updating the UI without another list request
- Queue state and AsyncStream
- The final flow
- Crash windows: recover instead of simulating a transaction across systems
- Orphan JPEG cleanup
- Where the AI over-engineered the project
- Complexity we kept because it protects a real invariant
- The human role in AI-assisted development
- The resulting mental model
- Interview notes
This is the long version. The short one, with the bug and the argument, is The Expense That Showed Up Twice. I keep this page for myself: it is the whole system laid out so I can walk it in an interview without reconstructing it from memory.
The project
I was adding native expense entry to an internal SwiftUI app for a field sales team. The existing expense data came from Expensify, but the goal was to let people create expenses directly in our app without changing the downstream workflow.
The visible feature sounded modest:
- Start an expense manually or scan a receipt.
- Let the user review and edit the merchant, date, amount, category, and comment.
- Attach a receipt photo from the camera or photo library.
- Save the expense as unassigned so it can later be associated with a meeting.
- Upload in the background so the user can immediately continue working.
- Show enough status that the user knows the save worked.
- Retry when connectivity is poor or the app is interrupted.
- Never create duplicate expenses.
That last group of requirements turned a form-and-camera feature into a small distributed system. There is durable state on the phone, a background transfer managed partly by iOS, a Node backend, MySQL, a receipt image on disk, a Core Data cache, and a SwiftUI list observing that cache. Any boundary between those pieces can fail at an inconvenient moment.
The engineering challenge was real. The equally important lesson was that an AI collaborator will make the architecture more complicated than the product needs unless someone keeps forcing the discussion back to evidence, invariants, and recovery behavior.
The product constraints that mattered
The app is an internal business tool, not a social network or photo library. A receipt image needs to remain legible for review and auditing, but nobody needs a media pipeline optimized for millions of public views.
The existing backend already had an expensify table and a 25-character transactionid column. The clean product decision was to keep using that object and table. An unassigned and an assigned expense are not different kinds of data. Assignment is state on the same expense.
The storage design was deliberately plain:
- Store one appropriately sized, legible JPEG per receipt.
- Put the file in a server-managed web directory.
- Store one URL on the existing expense record.
- Let the backend own the physical storage location so the client contract stays stable if storage changes later.
That was enough. A separate native-expense table, original-and-thumbnail URLs, CDN design, and an image migration layer were all plausible future ideas, but they were not requirements for this version.
Durable, non-blocking uploads
The user should not stare at a spinner while a receipt uploads. Saving should feel immediate, and the user should be able to add another receipt or use the rest of the app.
That means the upload cannot exist only as an in-memory Task. If the process is suspended or killed, an in-memory queue disappears. The app needs durable intent: before it tells the user the receipt was saved, it must have persisted enough information to retry the operation.
For the expected volume, a handful of pending receipts rather than thousands, the persistent queue is intentionally small:
- Each receipt image is a separate
UUID.jpgfile. pending.jsonis a JSON array of job metadata.- A Swift actor owns reads and mutations of that manifest.
- Each manifest write uses atomic file replacement.
- A background
URLSessionperforms the upload and can continue while the app is suspended.
The order of operations when enqueuing is what makes the acknowledgement honest.
If the manifest write fails, the store rolls back the array mutation and removes the new JPEG. The app never acknowledges a save for which it failed to persist the upload intent.
NativeExpenseUploadStore.swift
actor NativeExpenseUploadStore {
static let shared = NativeExpenseUploadStore()
private var items: [NativeExpensePendingUpload] = []
private var hasLoaded = false
func enqueue(
id: UUID,
imageData: Data,
upload: NativeExpenseUpload,
activeUsername: String
) throws -> NativeExpensePendingUpload {
try loadIfNeeded()
// A retry of the same client UUID returns the job that already exists.
if let existingItem = items.first(where: { $0.id == id }) {
return existingItem
}
let item = NativeExpensePendingUpload(
id: id,
activeUsername: activeUsername,
imageFilename: "\(id.uuidString.lowercased()).jpg",
upload: upload,
createdAt: .now,
status: .queued,
retryCount: 0
)
try FileManager.default.createDirectory(at: Self.directory, withIntermediateDirectories: true)
try imageData.write(to: item.imageURL, options: .atomic)
items.append(item)
do {
try saveManifest()
} catch {
// Never acknowledge a job whose intent is not on disk.
items.removeLast()
try? FileManager.default.removeItem(at: item.imageURL)
throw error
}
return item
}
func remove(id: UUID) throws {
try loadIfNeeded()
guard let index = items.firstIndex(where: { $0.id == id }) else { return }
let item = items.remove(at: index)
do {
try saveManifest()
} catch {
items.insert(item, at: index)
throw error
}
// Manifest first, then the image. A crash here leaves an orphan
// file, not a job whose image is missing.
try? FileManager.default.removeItem(at: item.imageURL)
}
private func saveManifest() throws {
let manifestURL = Self.directory.appendingPathComponent("pending.json")
let data = try JSONEncoder().encode(items)
try data.write(to: manifestURL, options: .atomic)
}
}
Why JSON instead of SQLite?
SQLite would be a good answer for a large or relational job system. It supports transactional updates, indexed queries, many jobs, dependencies, priorities, and potentially multiple processes. Upload queues at much larger companies use SQLite for exactly those reasons.
It would also have been unnecessary machinery here. Rewriting a tiny JSON metadata array is cheap. The image bytes are separate files, so changing a status does not rewrite the image. An actor serializes access inside the process, and atomic replacement prevents readers from seeing a half-written manifest.
The trade-off is explicit: this design is simple because the queue is small. If the workload changed materially, SQLite would become the better choice.
Idempotency: at-least-once delivery, once-only effect
A durable queue necessarily retries. A timeout does not tell the client whether the server failed before creating the expense or succeeded and lost the response. Retrying with a new identity could create a second expense.
The client therefore generates one UUID when the job is created and persists it with the job. Every retry carries the same UUID. That UUID is the idempotency key.
The backend must make the idempotency decision atomically. Checking first in application code and inserting second is not sufficient: two Node processes behind a load balancer could both observe that no row exists. The MySQL unique constraint is the concurrency authority. One insert wins. A competing insert with the same transaction ID hits the unique key, and the handler returns the expense that already exists.
The existing transactionid column only allows 25 characters, so the server deterministically maps the UUID to the schema:
client UUID → SHA-256 → first 22 hex characters → "cp-" prefix
That produces exactly 25 characters. Twenty-two hex characters provide 88 bits of identity space, far beyond what this product needs. The database uniqueness constraint still decides the vanishingly unlikely collision case.
Controllers/nativeExpenseRouter.js
function transactionIDFor(uploadID) {
const digest = crypto
.createHash("sha256")
.update(uploadID)
.digest("hex")
.slice(0, 22);
return `cp-${digest}`;
}
router.post(
"/",
express.raw({ type: "image/jpeg", limit: maximumReceiptBytes }),
async function (req, res, next) {
try {
const metadata = requestMetadata(req);
const username = req.user.name;
const uploadID = clientUploadID(req);
const transactionid = transactionIDFor(uploadID);
// Fast path: this UUID already produced an expense. The client lost
// the response or is retrying; hand it the same object again.
const existingExpense = await Expensify.findCanonicalOwned(
transactionid,
username,
req.dbPool
);
if (existingExpense) {
return res.status(200).json(canonicalExpenseResponse(existingExpense));
}
await ReceiptStorage.write(transactionid, req.body);
let createdExpense;
try {
createdExpense = await helpers.withTransaction(req.dbPool, async (connection) => {
await Expensify.createUploadedExpense(
{ transactionid, username, ...metadata, url: ReceiptStorage.receiptURL(transactionid) },
connection
);
return Expensify.findCanonicalOwned(transactionid, username, connection);
});
} catch (error) {
// Two processes raced past the fast path. The unique key on
// transactionid is the arbiter; the loser returns the winner's row.
if (error.code === "ER_DUP_ENTRY") {
const duplicate = await Expensify.findCanonicalOwned(transactionid, username, req.dbPool);
if (duplicate) {
return res.status(200).json(canonicalExpenseResponse(duplicate));
}
}
throw error;
}
res.status(201).json(canonicalExpenseResponse(createdExpense));
} catch (error) {
next(error);
}
}
);
In a payment system, the idempotency key and the resource ID are often separate. The server stores a mapping from (caller, idempotency key) to the payment or order it created. Here the deterministic transaction ID lets one column serve both roles.
A simplification I had to insist on
The first server mapping hashed the username together with the UUID. That was defense in depth, but it did not protect a realistic requirement. A properly generated UUID already has negligible collision risk, the operation belongs to an authenticated user, and the database has a unique constraint. Including the username made the mapping harder to explain and created extra code without changing a meaningful outcome.
The simplified rule is easier to reason about: the same client UUID always maps to the same server transaction ID. The iOS side carries the same derivation so the client can predict the ID before the server ever sees the upload.
NativeExpenseUpload.swift
public enum NativeExpenseIdentity {
/// Must remain byte-for-byte compatible with the backend's
/// `transactionIDFor(uploadID)` implementation.
public static func transactionID(clientUploadID: UUID) -> String {
var hasher = SHA256()
hasher.update(data: Data(clientUploadID.uuidString.lowercased().utf8))
let digest = hasher.finalize().map { String(format: "%02x", $0) }.joined()
return "cp-\(digest.prefix(22))"
}
}
The user report: duplicate expenses
The feature shipped to beta testers in version 2.63. On the first day, two users reported that expenses appeared twice, and one user saw the problem multiple times. A screenshot showed visually identical pairs: the same merchant, comment, amount, and date.
I could not reproduce it on my own phone.
The tempting responses were all guesses:
- Maybe users double-tapped Save.
- Maybe they did not notice the background-upload message and entered the receipt again.
- Maybe a request timed out and the queue uploaded it twice.
- Maybe the UI rendered the same object twice.
- Maybe users had already deleted one of two server records before we inspected the data.
The AI leaned quickly toward speculative guards: disable rapid taps, deduplicate on merchant, amount, and date. I pushed back. A double-tap guard is harmless as UI polish, but it cannot substitute for idempotency. A content hash is actively dangerous. Two legitimate lunches can have the same merchant, amount, and date, and we should not silently discard a valid business expense because it resembles another one.
The first question had to be more basic:
Are there two server expenses, or is one server expense represented twice in the local cache?
That distinction changes the entire investigation.
How I approached the investigation
Separate the identities
The system contains several things that can look like “the expense.” Naming them is most of the work.
I added the transaction ID unobtrusively at the bottom of the expense detail view. If a user reports duplicates, I can ask for both IDs:
- Different transaction IDs mean two business objects were created.
- The same transaction ID means the local cache or presentation layer duplicated one business object.
That tiny debugging affordance is worth more than pages of speculation.
Inspect the backend evidence
The server path already had an idempotent transaction ID and a MySQL uniqueness constraint. The evidence did not support a simple rapid double-tap explanation for all of the reported rows. At least part of the problem could be explained by one server object becoming multiple local managed objects.
This did not prove the complete history of every user report. It identified a concrete defect that could create exactly the visible symptom without requiring a duplicate server upload.
Trace the local refresh path
Core Data was being used as a server-backed cache. SwiftUI’s @FetchRequest observes that cache. It does not observe the server.
The list refresh is long-standing code that parses server data on a private child context, and for years it had a single caller. The new upload queue added a second one: a full list refresh after every completed upload. Two overlapping refreshes could each fetch before the other’s insert became visible, conclude that an expense did not exist, and insert separate managed objects with the same transaction ID. When both changes reached the parent context, SwiftUI displayed duplicate rows.
Running the work on private contexts kept parsing off the main thread, but it also expanded the concurrency surface. The optimization was not free.
The local-cache fix
The next model version made Expensify.transactionid a Core Data uniqueness constraint. That gives the cache the same invariant the backend already had: no two expense objects may share a transaction ID.
The complete fix included more than adding the model constraint:
- Serialize expense refreshes through one in-flight
@MainActortask. - Reconcile expenses synchronously on the main view context.
- Use a merge policy appropriate for server-backed objects.
- Deduplicate legacy cached rows by transaction ID while reconciling.
- Rebuild the existing Core Data store once before activating the new constraint.
DataManager.swift
public init(inMemory: Bool = false) {
// ...load the model and point the store at the app group container...
if let storeURL {
rebuildPersistentStoreForTransactionIDConstraintIfNeeded(at: storeURL, model: model)
}
container.loadPersistentStores { _, error in
if let error { fatalError("Error: \(error.localizedDescription)") }
}
container.viewContext.mergePolicy = NSMergePolicy.mergeByPropertyObjectTrump
}
/// Rebuilds the server-backed cache once before enabling the uniqueness
/// constraint. Credentials and pending receipt uploads live elsewhere and
/// survive this. Only the app performs it; the widget shares the store
/// but must never destroy it.
private nonisolated func rebuildPersistentStoreForTransactionIDConstraintIfNeeded(
at storeURL: URL,
model: NSManagedObjectModel
) {
guard Bundle.main.bundleURL.pathExtension != "appex" else { return }
let defaults = UserDefaults(suiteName: Persistance.appGroupName) ?? .standard
guard !defaults.bool(forKey: Self.rebuildKey) else { return }
do {
if FileManager.default.fileExists(atPath: storeURL.path) {
let coordinator = NSPersistentStoreCoordinator(managedObjectModel: model)
try coordinator.destroyPersistentStore(at: storeURL, type: .sqlite, options: nil)
}
defaults.set(true, forKey: Self.rebuildKey)
} catch {
// Leave the marker unset so the next launch retries.
}
}
The one-time rebuild is appropriate because this Core Data store is a disposable cache of server data. Pending uploads and credentials live elsewhere. Deleting this cache does not lose the source of truth. The app fetches it again.
This distinction matters because “nuke Core Data” would be reckless if Core Data contained unsynchronized user-created data. Here it was a controlled migration strategy for a replaceable cache.
The Core Data constraint and merge policy are the final safety net, not the only concurrency control. Serializing refreshes prevents needless conflicts and makes behavior easier to reason about. The constraint preserves the invariant if an unexpected code path overlaps anyway.
Updating the UI without another list request
The original completion flow uploaded the expense, then issued a full GET for all expenses, wrote that response to Core Data, and waited for the list to notice the change. This created more network work, more refresh overlap, and a longer period in which the pending row could appear stuck.
The improved response contract returns the canonical expense object from the upload endpoint:
201 Createdreturns the newly created expense.200 OKon an idempotent retry returns the existing expense.
The iOS app upserts that one object directly into Core Data. The @FetchRequest observes the save and renders it automatically. The queue then removes the completed job and publishes its new snapshot.
NativeExpenseUploadQueue.swift
private func storeCompletedUpload(id: UUID, response: NativeExpenseUploadResponse) async {
do {
try await ExpensifyNetworking.storeUploadedExpense(response)
guard await hasStoredExpense(transactionID: response.transactionid) else {
throw NativeExpenseUploadQueueError.localExpenseWasNotStored
}
try await store.remove(id: id)
await publishSummary()
} catch {
recordQueueError(error, stage: "store_uploaded_expense")
// The job is still in the manifest. Retrying reuses the same UUID,
// so the server answers 200 with the same expense.
if let item = try? await store.item(id: id) {
await scheduleRetry(for: item, error: error.localizedDescription)
}
}
}
Older app versions remain compatible because Swift’s Decodable ignores unknown JSON keys by default. They decode the transaction ID they expect, ignore the additional expense fields, and continue using their full refresh behavior. That makes a server-first deployment possible.
Queue state and AsyncStream
The UI also needs to know about jobs that are not yet server expenses. A pending upload should look like a row in the expense list, not a vague banner that can sit at “1 receipt uploading” forever.
The queue publishes complete summaries through an AsyncStream. Each subscriber receives the current snapshot immediately and then the newest snapshot whenever queue state changes. Buffering only the newest value is right because the UI cares about current truth, not a replay of every intermediate event.
NativeExpenseUploadQueue.swift
actor NativeExpenseUploadQueue {
private var updateContinuations: [UUID: AsyncStream<NativeExpenseUploadSummary>.Continuation] = [:]
func updates() async -> AsyncStream<NativeExpenseUploadSummary> {
let subscriberID = UUID()
let (stream, continuation) = AsyncStream.makeStream(
of: NativeExpenseUploadSummary.self,
bufferingPolicy: .bufferingNewest(1)
)
updateContinuations[subscriberID] = continuation
continuation.onTermination = { [weak self] _ in
Task { await self?.removeUpdateContinuation(id: subscriberID) }
}
await reconcileSessionState()
continuation.yield(await currentSummary())
return stream
}
private func publishSummary() async {
guard !updateContinuations.isEmpty else { return }
let summary = await currentSummary()
for continuation in updateContinuations.values {
continuation.yield(summary)
}
}
}
UnassignedExpensesView.swift
.task {
for await summary in await uploadQueue.updates() {
pendingUploadSummary = summary
}
}
This is the modern concurrency mechanism for delivering queue state, but it is important not to over-credit it. AsyncStream did not fix duplicate Core Data rows. It fixed state propagation and made the UI model easier to follow.
Pending rows are visible in the same list as uploaded expenses. While a row is syncing, it is not editable. Tapping it explains that the upload must finish. A job that exhausts automatic retries becomes “needs attention” and offers retry or discard rather than pretending to upload forever. Pull to refresh reconciles both queue state and server-backed expenses.
The final flow
The most useful way to understand it is by assigning one invariant to each layer.
| Invariant | Authoritative mechanism |
|---|---|
| A saved upload survives process death | JPEG plus durable queue manifest |
| Only one component mutates the in-process queue at a time | Swift actor |
| iOS can continue the transfer while the app is suspended | Background URLSession |
| Retrying a request does not create another server expense | Stable client UUID plus MySQL unique transaction ID |
| One server expense appears once in the local cache | Core Data uniqueness constraint plus serialized reconciliation |
| SwiftUI reflects current pending work | Snapshot-based AsyncStream |
| SwiftUI reflects saved expenses | @FetchRequest observing Core Data |
| Abandoned local images do not accumulate indefinitely | Startup orphan cleanup |
Crash windows: recover instead of simulating a transaction across systems
The manifest, background session, backend database, filesystem, and Core Data store cannot participate in one atomic transaction. The realistic goal is not “no crash can ever occur between two lines.” The goal is that every interrupted state converges safely.
| Interruption point | Recovery behavior |
|---|---|
| Before the manifest is saved | Enqueue fails. Array mutation and JPEG are rolled back. |
| After enqueue, before upload begins | The durable job is scheduled on the next launch. |
After the task is created, before resume() | Startup finds the suspended task in the background session and resumes it. |
| While the request is in flight | The background session may continue. Otherwise the same UUID is retried. |
| Server commits, response is lost | Retry reaches the unique transaction ID and returns the existing expense. |
| Response arrives before Core Data save | If the app dies, the pending job remains and safely retries. |
| Core Data saves before queue removal | If the app dies, retry and upsert use the same transaction ID and the cache constraint merges it. |
| Manifest removal succeeds before JPEG deletion | The image is orphaned temporarily. Startup cleanup removes it. |
| Core Data cache is missing or rebuilt | A normal list refresh reconstructs it from the server. |
The third row is worth a closer look, because it was a real gap. The app creates a background upload task, persists the job as uploading, and then calls resume(). A process death in that narrow interval leaves a suspended task that the session still reports as present. The first version of the startup check counted suspended tasks as live, so the job would sit at “uploading” forever with no retry path. The right response was a small reconciliation rule, not a second persistence subsystem.
NativeExpenseUploadQueue.swift
private func currentSessionUploadIDs() async -> Set<UUID> {
await withCheckedContinuation { continuation in
session.getAllTasks { tasks in
let uploadIDs = tasks.compactMap { task -> UUID? in
switch task.state {
case .running:
break
case .suspended:
// The process died after creating the task and persisting
// the job as uploading, but before calling resume(). Finish
// that interrupted intent rather than reporting a stalled upload.
task.resume()
case .canceling, .completed:
return nil
@unknown default:
return nil
}
return task.taskDescription.flatMap(UUID.init(uuidString:))
}
continuation.resume(returning: Set(uploadIDs))
}
}
}
Orphan JPEG cleanup
Queue deletion deliberately saves the manifest without the job before deleting its JPEG. If the process dies between those operations, the result is an orphan file, not a manifest entry whose required image is missing. That is the safer failure mode.
On startup, after successfully decoding the manifest, the store builds a set of referenced image filenames and deletes unreferenced files matching the queue’s exact lowercase UUID.jpg naming convention. It does not touch arbitrary JPEGs. If the manifest cannot be decoded, cleanup does not run, because deleting images without trustworthy references would be unsafe.
NativeExpenseUploadStore.swift
private func loadIfNeeded() throws {
guard !hasLoaded else { return }
try FileManager.default.createDirectory(at: Self.directory, withIntermediateDirectories: true)
let manifestURL = Self.directory.appendingPathComponent("pending.json")
if FileManager.default.fileExists(atPath: manifestURL.path) {
items = try JSONDecoder().decode([NativeExpensePendingUpload].self, from: Data(contentsOf: manifestURL))
}
// Only after the manifest decoded: the references are trustworthy.
cleanupOrphanedImages()
hasLoaded = true
}
private func cleanupOrphanedImages() {
let referencedFilenames = Set(items.map(\.imageFilename))
let fileManager = FileManager.default
guard let files = try? fileManager.contentsOfDirectory(
at: Self.directory,
includingPropertiesForKeys: nil,
options: [.skipsHiddenFiles]
) else { return }
for file in files where isManagedReceiptImage(file)
&& !referencedFilenames.contains(file.lastPathComponent) {
try? fileManager.removeItem(at: file)
}
}
/// Only files the queue itself would have written: lowercase `UUID.jpg`.
private func isManagedReceiptImage(_ file: URL) -> Bool {
guard file.pathExtension == "jpg" else { return false }
let identifier = file.deletingPathExtension().lastPathComponent
guard let uuid = UUID(uuidString: identifier) else { return false }
return file.lastPathComponent == "\(uuid.uuidString.lowercased()).jpg"
}
This is a small mark-and-sweep garbage collector. It is fast when there is nothing to do and avoids requiring perfect cleanup timing in every code path.
Where the AI over-engineered the project
The AI was useful for generating code, enumerating failure modes, and discussing concurrency. It also repeatedly treated every imaginable future concern as a present requirement.
It invented a second expense model
An early design introduced a native_expenses table even though the product already had an expensify expense object. That would have created two schemas, mapping logic, migration questions, and ambiguity about which object the app was displaying.
The correction was simple: use the existing table and existing object. Assignment is a state transition, not a different entity.
It designed a photo platform for audit receipts
The early proposal included an original image, a thumbnail, two URLs, possible CDN behavior, and migration machinery. Those are sensible capabilities for a photo-heavy product, but this feature needs one legible receipt image that users rarely reopen.
The simpler design stores one URL. The server controls what is behind it, so storage can still migrate later without forcing the client to know the physical directory.
It claimed implementation before the destination was known
At one point the AI described image uploading as implemented while the actual destination URL and storage directory were still unresolved. The code may have had an upload request, but the end-to-end feature was not implemented until the server contract and durable storage target were concrete and verified.
This became a useful rule: do not accept “implemented” for an integration without asking where the bytes go, how the record points to them, how they are retrieved, and what happens after deployment.
It added the username to the transaction-ID derivation
The client UUID already provided the idempotency identity. Hashing the username with it was extra defense for a collision scenario too improbable to drive this design. It also obscured the central invariant that one client job maps deterministically to one server transaction.
Removing the username reduced code and made retries easier to explain.
It tried to persist the entire server response in the queue
The AI proposed storing the returned expense object in pending.json to cover a crash after the server response but before the Core Data save. That would add another durable state, schema evolution for the response snapshot, and reconciliation branches.
The existing mechanisms already recover:
- If Core Data was not saved, the pending job retries with the same UUID and the server returns the existing expense.
- If Core Data was saved, the unique constraint makes the retry and upsert harmless.
- If the queue and cache disagree later, a normal refresh restores the server truth.
Persisting the full response solved a theoretical micro-window twice.
It reached for speculative duplicate prevention
The first duplicate theories produced ideas like a double-tap timer and merchant/amount/date hashes before establishing whether the duplicates existed on the server. These could hide symptoms without fixing the system and could reject legitimate equal-valued expenses.
The better move was to surface the transaction ID and inspect each storage layer.
It optimized camera and image processing before locating the blur
When a tester reported blurry photos, proposed capture and backend image changes arrived before we had isolated whether the blur came from capture, compression, storage, transport, or the viewer. Those changes were reverted while zoom support, an independently useful viewer feature, remained.
The lesson is the same: instrumentation and reproduction first, optimization second.
Complexity we kept because it protects a real invariant
“Avoid over-engineering” does not mean deleting all resilience. Several pieces are essential:
- A persistent queue, because an in-memory task loses user data when the process dies.
- A stable client-generated UUID, because ambiguous responses require safe retries.
- A database unique constraint, because two server processes cannot coordinate through memory.
- A Core Data unique constraint, because a server-backed cache should not represent one transaction twice.
- Background
URLSession, because users should not wait on the upload screen. - Retry with backoff and a terminal needs-attention state, because networks fail and infinite invisible retries are bad UX.
- Atomic manifest writes and rollback, because acknowledging an unpersisted job would lose data.
- Orphan cleanup, because filesystem and manifest changes cannot be atomic together.
- Observable queue snapshots, because users need clear, current feedback.
The design became simpler without giving up these guarantees.
The human role in AI-assisted development
This project changed how I think about working with AI on production code.
AI is very good at answering “What could go wrong?” It is less naturally disciplined about the next questions:
- Did that failure actually happen?
- Which invariant would it violate?
- Which existing layer already handles it?
- What is the smallest additional mechanism that closes the gap?
- How will we prove the mechanism works?
I repeatedly had to slow the process down and insist on one concept at a time. That was especially valuable when discussing idempotency, persistent queues, actor reentrancy, Core Data contexts, and server concurrency. The conversation became productive once we stopped treating a pile of code as an answer and started assigning ownership of each invariant.
Some practical habits emerged:
- Ask the AI to state the evidence separately from its theory.
- Do not merge a speculative fix merely because it is cheap to generate.
- Make identifiers visible enough to debug real user reports.
- Trace one object across client job, request, database row, cache object, and UI row.
- Prefer recovery and convergence over trying to make unrelated storage systems behave like one transaction.
- Use the database for cross-process uniqueness and actors for in-process serialization.
- Keep server-backed caches disposable when possible.
- Deploy backward-compatible server response changes before clients that depend on them.
- Revert optimizations when the measured problem is elsewhere or still unknown.
- Keep asking whether complexity protects a product requirement or only a hypothetical edge.
The resulting mental model
The queue provides at-least-once delivery. The stable UUID and unique server transaction ID turn that into an exactly-once business effect. Core Data is a disposable projection of server state, and its uniqueness constraint prevents one transaction from becoming two visible rows. AsyncStream reports the current queue snapshot, while @FetchRequest reports the current cached expense snapshot.
No component alone guarantees the whole experience. The system works because each layer has a narrow responsibility and every non-atomic boundary has a recovery path.
The most important simplification was not a particular API or framework. It was replacing “What code can the AI add for every theoretical failure?” with a better question:
What is the smallest system whose invariants make the user’s data safe and whose state we can actually explain?
That question produced less code, a stronger mental model, and a system that should be easier to operate when the next real bug report arrives.
Interview notes
The same project, compressed into the questions a system design interview would actually ask.
- Why must the idempotency key exist before the job is durable? Because the retry has to carry the same identity as the first attempt. If the key were generated at send time, a crash between attempts would produce a new key and a second expense.
- How does at-least-once delivery become an exactly-once effect? The client retries freely. The server maps the key to a deterministic resource ID and lets a unique index decide the winner. Retries read the winner’s row back.
- Why a database unique index rather than a lock in the Node process? Two processes behind a load balancer share no memory. The database is the only thing both of them talk to.
- When is a JSON manifest enough, and when is SQLite justified? JSON when the job count is tiny, the writes are whole-file, and one process owns it. SQLite when you need partial updates, indexes, many jobs, or multiple writers.
- What does the actor actually guarantee? Serialized access to in-memory state within one process. Not durability, not cross-process safety, and not atomicity across an
await. State checks after every suspension point still matter. - Why can’t the server response, the Core Data save, and the manifest deletion be one transaction? Three different stores with no shared coordinator. Design each boundary to converge on retry instead.
- How do you handle a crash at each boundary? Enumerate the boundaries, and for each one, name the state left behind and the code path that repairs it on the next launch.
- What did
AsyncStreamfix, and what didn’t it fix? State propagation to the UI. It had nothing to do with database uniqueness, and claiming otherwise is a red flag. - Duplicate business object or duplicated cache projection? Compare the IDs. Different IDs mean the server has two. Same ID means the cache or the view layer doubled one.
- How do you work with an AI on this without losing judgment? Separate evidence from theory, assign an owner to each invariant, and refuse any mechanism that doesn’t close a gap you can name.