FA2: Fast, Accurate Autoscaling for Serving Deep Learning Inference with SLA Guarantees

Kamran Razavi, Manisha Luthra, Boris Koldehofe, Max Mühlhäuser, Lin Wang

OnderzoeksoutputAcademicpeer review

1 Citaat (Scopus)
39 Downloads (Pure)


Deep learning (DL) inference has become an essential building block in modern intelligent applications. Due to the high computational intensity of DL, it is critical to scale DL inference serving systems in response to fluctuating workloads to achieve resource efficiency. Meanwhile, intelligent applications
often require strict service level agreements (SLAs), which need to be guaranteed when the system is scaled. The problem is complex and has been tackled only in simple scenarios so far. This paper describes FA2, a fast and accurate autoscaler
concept for DL inference serving systems. In contrast to related works, FA2 adopts a general, contrived two-phase approach. Specifically, it starts by capturing the autoscaling challenges in a comprehensive graph-based model. Then, FA2 applies targeted graph transformation and makes autoscaling decisions with an efficient algorithm based on dynamic programming. We implemented FA2 and built and evaluated a prototype. Compared with
state-of-the-art autoscaling solutions, our experiments showed FA2 to achieve significant resource reduction (19% under CPUs and 25% under GPUs, on average) in combination with low SLA violations (less than 1.5%). FA2 performed close to the theoretical optimum, matching exactly the optimal decisions (with the least required resources) in 96.8% of all the cases in our evaluation.
Originele taal-2English
TitelProceedings of the 28th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS 2022)
Aantal pagina's14
ISBN van geprinte versie978-1-6654-9998-9
StatusPublished - 29-jun.-2022
Evenement2022 IEEE 28th Real-Time and Embedded Technology and Applications Symposium (RTAS) - Milano, Italy
Duur: 4-mei-20226-mei-2022


Conference2022 IEEE 28th Real-Time and Embedded Technology and Applications Symposium (RTAS)

Citeer dit