Aceso: Latency-Aware Geo-Distribution of Microservices for Cost-Effective Carbon Reduction
Georgia Christofidi, Francisco Álvarez-Terribas, Ioannis Roumpos, Nicolas Kourtellis, Jesus Omaña Iglesias, Thaleia Dimitra Doudali
AI 速览：Aceso 是一种面向区域性受限基础设施的自适应微服务放置方法，在满足延迟约束的前提下，同时考虑碳强度与成本，动态调整微服务的地理分布。
Microservices are a dominant architecture in cloud computing, offering scalability and modularity, but also posing complex deployment challenges. As data centers contribute significantly to global carbon emissions, carbon-aware scheduling is a promising mitigation strategy. However, most existing solutions target batch, HPC, or serverless workloads and assume global-scale infrastructure. Such an assumption does not hold for many regional small to medium-sized enterprises (SMEs) with microservice applications, due to geographic, cost, and data sovereignty constraints. In this paper, we present Aceso, an Adaptive Carbon- and Efficiency-aware placement for microservices that considers carbon, cost, and latency constraints. Aceso dynamically places microservices across geographically constrained regions using a scalable optimization with insight-based search space pruning techniques. Evaluation on a real-world deployment shows that Aceso adapts to real-time changes in workload and carbon intensity and reduces carbon emissions by up to 37.4% and operational cost by up to 3.36%, on average across days, compared to a static deployment within a single country, while maintaining the application's latency targets. In this way, Aceso enables carbon- and cost-aware microservice deployment for latency-sensitive applications in regionally limited infrastructures.
Multimodal Large Language Models (MLLMs) power platforms like ChatGPT, Gemini, and Copilot, enabling rich interactions across text, images, and videos. Unlike text-only workloads, multimodal requests are inherently heterogeneous, differing by orders of magnitude in latency and memory footprint. Existing LLM serving systems, designed for homogeneous text workloads, struggle under this heterogeneity: large image and video requests monopolize GPU resources, causing severe head-of-line blocking and degrading the responsiveness of lightweight, latency-critical text queries. Recent multimodal serving systems mitigate this interference through disaggregation, isolating encoding and inference stages across GPUs. While effective at large scale, such designs introduce severe GPU underutilization in more practical, moderate-scale deployments. In response, we build RPS-Serve a modality-aware scheduling framework that leverages the extreme heterogeneity in multimodal workloads and introduces a simple abstraction: videos behave like rocks, images like pebbles, and text like sand. Then, it lets sand flow quickly through pebbles and rocks, maximizing GPU utilization while ensuring fast responsiveness for lightweight requests and controlled execution for heavier ones. To achieve this, RPS-Serve estimates the GPU admission cost of each request, classifies requests into rocks, pebbles, and sand using lightweight clustering, and applies priority-based scheduling with aging to balance performance across modalities. Evaluation across state-of-the-art MLLMs shows that RPS-Serve increases GPU utilization by 65% and reduces TTFT by up to 9.3x, on average across models, compared to disaggregation-based multi-modal systems.
Panagiotis Georgios Pennas, Konstantinos Papaioannou, Marco Guarnieri, Thaleia Dimitra Doudali
Large Language Models (LLMs) rely on optimizations like Automatic Prefix Caching (APC) to accelerate inference. APC works by reusing previously computed states for the beginning part of a request (prefix), when another request starts with the same text. While APC improves throughput, it introduces timing side channels: cache hits are faster than misses, creating observable latency differences. In multi-tenant systems, attackers can exploit these differences to infer sensitive information, e.g., by incrementally reconstructing another user's request by observing hit/miss patterns. Current defenses take a sledgehammer approach: they disable APC and cache sharing, isolating users, and sacrificing efficiency for regular users. This paper presents CacheSolidarity, a system that secures multi-tenant LLM serving systems against APC side channels without sacrificing performance and efficiency. CacheSolidarity monitors cache reuse across users, flags suspicious sharing, and selectively isolates prefixes, restricting their reuse only when necessary. Evaluation shows that CacheSolidarity enables up to 70% higher cache reuse and 30% lower inference latency compared to existing user-isolation defenses. CacheSolidarity's lightweight design demonstrates how LLM serving can be secured without sacrificing performance or incurring unbearable overheads.