Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

Factual evidence

What the source reports

Researchers propose Cascade, an LLM serving framework that uses SLO-aware latency budgets to optimize inference cost and throughput.