Google Cloud Networking at Next ’24

Google Cloud is rolling out a series of networking enhancements, including:

• Planet-scale networking for AI/ML workloads 
• Any cloud to any service connectivity
• Securing the workload, data, and users 
• Gemini-powered network operations 

One highlight is Cloud Load Balancing for inference workloads. Custom metrics provide queue depth as a metric for load balancing AI workloads to deliver faster user response time to prompts while optimizing TPU and GPU utilization. Cloud Load Balancing for streaming inference uses metrics based on number of streams, bytes-in, and bytes-out, versus requests per second and CPU utilization to optimize performance. In addition, Cloud Load Balancing with traffic management for AI models monitors the health of individual model service endpoints and routes requests to healthy endpoints, initiates cross-region failover when an outage is detected, and splits traffic across different models and model versions, helping you to manage rollouts.

Another new feature is Private Service Connect transitivity over Network Connectivity Center, which enables services in a spoke VPC to be transitively accessible from other spoke VPCs. 

“AppLovin operates one of the most successful platforms for app developers to grow their business, reaching over 1.4 billion daily active users (DAUs) worldwide. We are leveraging Google Cloud to advance our next gen AI platform with state-of-the-art hardware to power our training and inference workloads. Google Cloud’s global front-end solution with Load Balancer, Cloud Armor, and CDN not only protects our users but helps businesses reach, monetize, and grow their audiences.” - Omer Hasan, VP of Operations, AppLovin

https://cloud.google.com/blog/products/networking/whats-new-for-networking-at-next24