The DRAM Constraint

Edge AI development is currently limited by DRAM costs rather than raw compute power. As hardware costs rise and manufacturers reduce memory in consumer devices, the primary engineering challenge is fitting models into limited memory footprints. The Google AI Edge team addresses this by prioritizing quantization and model size reduction to ensure applications run on devices like Raspberry Pis and entry-level IoT hardware.

Small vs. Tiny Model Strategies

• Small Models (1-4B parameters): These models are suitable for high-end phones and laptops with 4GB+ of RAM. They are highly capable at reasoning and can be deployed using zero-shot prompting or LoRA adapters. Through 2.9-bit quantization and memory-efficient runtimes, these models can achieve usable decode speeds (e.g., ~8 tokens/s on a Raspberry Pi).
• Tiny Models (<500M parameters): When small models are too large for older hardware or background system tasks, tiny models are required. Unlike small models, which rely on prompting, tiny models often require fine-tuning to achieve high reliability for specific tasks.

The Fine-Tuning Playbook

To make tiny models production-ready, the recommended approach is to use synthetically generated data. For specific tasks like function calling or text cleanup, generating between 10,000 and 10 million synthetic samples allows developers to achieve high reliability (e.g., >86% accuracy in function calling). This approach enables complex features like offline voice-to-function calling, where an ASR model feeds into a fine-tuned tiny Gemma model, bypassing the need for cloud subscriptions and ensuring user privacy.