thenewstack.io/google-tpu-sync-inference Google claims its new AI software cuts inference latency by over 90% — but there’s a catch
Google released TPU Sync, an open-source C++ framework that moves data between accelerator memory, host DRAM, storage and the network to reduce AI inference latency. Google reports that in some tests time to first token dropped by more than 90%, though the APIs are still changing and support is limited to specific Cloud TPU versions and JAX/PyTorch requirements.
