The Case for Physical Security

When building systems for high-stakes environments like financial fraud enforcement, software-based security (encryption, private endpoints, SOC2 compliance) is insufficient because it relies on configuration, which is prone to human error. To ensure data is defensible in court, the California Department of Financial Protection and Innovation (DFPI) moved to an air-gapped, offline architecture.

They replaced software firewalls with a physical data diode: a fiber optic cable cut in half where the internal side has only a receiver. This creates a physical impossibility for data to leak outward, moving trust from a policy-based construct to a physical property of the system.

AI as a Data Pipeline, Not a Magic Box

Initial attempts to feed raw, messy data directly into an LLM failed within two hours. The team realized that most AI problems are actually data engineering problems wearing an AI mask. They implemented a robust pipeline to ensure reproducibility and auditability:

  • Ingestion: Kafka buffers traffic spikes and maintains sequential event ordering, allowing the team to "time travel" and replay the exact state of the system at the moment a decision was made.
  • Processing: Spark clusters clean and normalize disparate data formats (audio, screenshots, bank statements) before they ever reach the model.
  • Storage: Apache Iceberg provides an immutable, time-traveled data store, ensuring that evidence presented in court is reproducible years later.

Optimizing Compute with Semantic Routing

Using a frontier model for every task is inefficient and costly. The team treated their GPU resources like a hospital: they stopped making the "neurosurgeon" (the largest model) take everyone's blood pressure. By implementing a semantic router, they analyze incoming requests and forward 80% of them to the smallest, fastest, and cheapest model capable of handling the task. This architectural shift tripled throughput on existing hardware and reduced processing costs by 70%.

Protecting Sensitive Data

To handle PII (Personally Identifiable Information) securely, the team uses a cryptographic vault with hardware security modules. The encryption keys are physically bolted to the server rack. If an attacker gains access to the data, they must physically break into the office and the server rack to make sense of the information, reinforcing the principle that when stakes are high, hardware-level security is more reliable than software-level abstractions.