LiteRT: On-Device AI Framework

Overview

LiteRT (Lite Runtime) is Google's framework for deploying ML and generative AI on edge devices. It's the successor to TensorFlow Lite with advanced GPU/NPU acceleration delivering up to 100x faster inference than CPU.

Platform Support

Quick Start

Android (Kotlin)

// Add dependency: implementation 'com.google.ai.edge.litert:litert:2.1.0'

val model = CompiledModel.create(
    context.assets,
    "model.tflite",
    CompiledModel.Options(Accelerator.GPU)  // or NPU, CPU
)

val inputBuffers = model.createInputBuffers()
val outputBuffers = model.createOutputBuffers()

inputBuffers[0].writeFloat(inputData)
model.run(inputBuffers, outputBuffers)
val result = outputBuffers[0].readFloat()

C++

#include "litert/cc/litert_compiled_model.h"
#include "litert/cc/litert_environment.h"

LITERT_ASSIGN_OR_RETURN(auto env, Environment::Create({}));
LITERT_ASSIGN_OR_RETURN(auto compiled_model,
    CompiledModel::Create(env, "model.tflite", kLiteRtHwAcceleratorGpu));

LITERT_ASSIGN_OR_RETURN(auto inputs, compiled_model.CreateInputBuffers());
LITERT_ASSIGN_OR_RETURN(auto outputs, compiled_model.CreateOutputBuffers());
compiled_model.Run(inputs, outputs);

Python

from ai_edge_litert.interpreter import Interpreter

interpreter = Interpreter(model_path='model.tflite')
interpreter.allocate_tensors()
interpreter.set_tensor(input_index, input_data)
interpreter.invoke()
output = interpreter.get_tensor(output_index)

APIs

CompiledModel API (Recommended)

• Modern API for hardware acceleration
• Supports GPU, NPU, CPU
• Zero-copy buffer interop
• Async execution

Interpreter API (Legacy)

• TensorFlow Lite compatible
• CPU-only in v2.x
• Use for backward compatibility

Task Decision Tree

Running inference on device?

• Use CompiledModel API with appropriate accelerator
• See gpu-acceleration.md or npu-acceleration.md

Deploying LLMs (Gemma, Phi, Qwen)?

• Use LiteRT-LM framework
• See litert-lm.md

Converting models to .tflite?

• PyTorch: Use litert-torch package
• TensorFlow: Use tf.lite.TFLiteConverter
• JAX: Use jax2tf bridge
• See model-conversion.md

Migrating from TensorFlow Lite?

• Package name changes only
• See migration.md

Performance Tips

1. Choose the right accelerator: NPU > GPU > CPU for most models
2. Use zero-copy buffers: Pass camera/GPU buffers directly
3. Enable async execution: Overlap CPU/GPU work
4. Cache NPU compilation: Use CompilerCacheDir environment option
5. Quantize models: INT8 reduces size 4x, improves speed

Dependencies

Android (Gradle)

implementation 'com.google.ai.edge.litert:litert:2.1.0'

Python

pip install ai-edge-litert           # Runtime
pip install litert-torch             # PyTorch conversion
pip install ai-edge-quantizer        # Quantization

Resources

• Official docs: https://ai.google.dev/edge/litert
• GitHub: https://github.com/google-ai-edge/LiteRT
• LiteRT-LM: https://github.com/google-ai-edge/LiteRT-LM

Reference Files

• gpu-acceleration.md - GPU setup, zero-copy, async execution
• npu-acceleration.md - NPU setup, AOT/JIT compilation, vendor support
• litert-lm.md - LLM deployment with LiteRT-LM
• model-conversion.md - Converting PyTorch/TF/JAX to .tflite
• migration.md - Migration from TensorFlow Lite