Google unveils LiteRT Tensor API for customizing on-device AI models

Google has launched the LiteRT Tensor API, a lightweight C++ interface designed to help engineers construct, customize, and optimize machine learning models specifically for on-device execution. The platform combines the flexibility of native model authoring with cross-platform portability, allowing a single computation graph to run efficiently across different accelerators without requiring code rewrites.

Solving hardware coupling

The traditional method for deploying AI on mobile devices involves manually writing computations in C++ to achieve maximum control over every operation. However, this approach creates significant friction because implementations become tightly coupled to specific backends like Metal, OpenCL, or Vulkan. Consequently, developers must rewrite their code whenever a new hardware target emerges.

Google highlights the engineering burden of managing zero-copy buffer interoperability and synchronization across CPU, GPU, and NPU units while integrating pre-processing, inference, and post-processing steps. The LiteRT Tensor API addresses these issues by enabling operation-by-operation graph construction in C++. This provides greater control over model architecture and accelerator execution while ensuring the same graphs can run on multiple hardware targets through LiteRT without modification.

Key capabilities and AI agent support

Beyond full control over model graphs, the API offers several critical features aimed at streamlining development and improving performance:

• Direct weight ingestion: Developers can load weights directly from checkpoints, including Hugging Face .safetensors files.
• Zero-copy buffer interoperability: This feature avoids copying accelerator buffers across pre-processing, inference, and post-processing stages to reduce latency.
• AI coding agent compatibility: The API supports AI agents that can construct, modify, validate, and benchmark model graphs by generating C++ code automatically.

Google demonstrated the practical application of these capabilities using a coding agent to create a real-world app for multi-object segmentation and tracking. This application served as the basis for benchmarking performance across various platforms:

We ran the same C++ code for the SAM 2 model on the GPUs of three devices: a macOS laptop (Metal), an iOS phone (Metal), and an Android phone (OpenCL). In our tests on the MacBook Pro, it ran about 1.5x faster than Apple MLX (fp16, image encoder plus mask decoder).

Optimization through structured primitives

Google emphasized that defining model graphs in plain C++ enables coding agents to perform meaningful optimizations. Specifically, the agent "moved SAM 2's per-frame decisions, such as whether the object is still visible, from CPU code into the graph" and rewrote operations that triggered CPU fallback on WebGPU using GPU-compatible arithmetics.

Laurence Moroney, director of AI at Arm, commented on the announcement via LinkedIn regarding the strategic role of this technology:

Going from under 10 to over 30 frames per second at 512x512 on WebGPU, with a coding agent doing the iteration, shows how much structured primitives help agents do real optimization work.

Moroney advised developers that they should always start with model conversion and turn to the C++ API only "when you need graph-level control or want pre and post-processing to stay on the accelerator." He also noted the benefits of defining model graphs in plain C++.

Future roadmap

Google is currently working on several planned improvements to expand the capabilities of the LiteRT Tensor API. These upcoming features include direct ML Drift backend support, additional NPU-friendly operations, automatic full-graph quantization, and expanded dynamic-shape handling for different image sizes, batch sizes, and sequence lengths.