arxiv.org/abs/2608.08097 OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
OasisKV is an LLM inference system built on vLLM that moves the KV cache out of HBM into other memory tiers, prefetching only the most relevant entries predicted via speculative decoding. With 2,048 KV tokens in HBM it stays within 0.61 accuracy points of full attention and reaches 1.86x vLLM throughput on AIME24.
— via arXiv Distributed Computing, Can Xiao, Sukmin Cho, Junbong We, Zhixiong Niu, Jianyi Cheng, Yiren Zhao, Youngjin Kwon, Yongqiang Xiong, Rui Ma, Junyi Liu
