Guest post by
Abhiroop Soni
Staff Engineer – DevOps at ShareChat
Abhiroop brings nearly 10 years of experience in designing, building, and delivering scalable infrastructure solutions. He specializes in DevOps-driven practices, observability, and architecting distributed systems that scale reliably, efficiently, and cost-effectively. Outside of work, Abhiroop enjoys playing badminton to unwind and recharge.
Companies that consume large amounts of cloud compute eventually face challenges with efficient resource utilization.
ShareChat is one of India’s largest Google Cloud users, so you can only imagine the scale of the challenge we faced. As heavy Kubernetes users, we experimented with the native GKE autoscaler, but the solution left a lot of headroom for underutilized resources.
Facing a steady demand flow from our applications, we purchased Google Cloud’s Committed Use Discounts (CUD) to score savings on compute capacity. However, we still dealt with a gap in resource utilization, for some clusters reaching even 50% of unused capacity we have already paid for.
Then, there was the manual effort. If I wanted to use both commitments for two different VM families in one cluster, I would have to create separate node pools, which would be time-consuming and inefficient.
This is why we started looking for optimization options. We solved a big part of the utilization problem with Cast AI, a Kubernetes automation solution that helps teams like ours reduce overprovisioning to a minimum through a mix of automation mechanisms like autoscaling, bin-packing, and workload rightsizing. This helped us eliminate the need to manually configure node pools while bin-packing all the pods into as few nodes as possible to boost utilization. Thanks to this, we ended up saving a lot of time and money.
But this was just the first part of our journey. We wanted to optimize resource usage across all of our Committed Use Discounts, which led us to develop a solution with the Cast AI team.
Before I dive into that solution, let’s quickly recap why commitments are so challenging to manage for teams looking to improve their resource utilization.
Problem: Inefficient use of CUD resources
At ShareChat, we had commitments for a mix of different VM families: N1, N2, N2D, C2, and others. We used to run clusters entirely on one kind of family, so we had to consistently tune the number of instances.
For example, if I committed to 5000 cores from the C2 family of VMs and one of my clusters runs on 7000 cores on average, I’d configure a static percentage using Cast AI’s Mutating Admission Webhook to say 70% of the entire cluster usage (4900 cores). To clarify, the Mutating Admission Webhook gives users the flexibility to run all or a portion of workloads on Spot VMs without having to modify manifest files.
So, using the Webhook, 70% of the cluster’s compute would run on our on-demand C2 commitments, and the rest would run on Spot VMs.
Reaching that sweet spot required extensive manual work and tweaking. To make our lives easier, we used to whitelist those instance families on specific clusters. But that quickly became inefficient because we couldn’t drive synergies between two clusters if they used different VM families at the same time.
Then there was the question of fluctuating traffic. Between 11 pm and 5 am, our traffic drops significantly. But we’d still be running 70% on-demand via commitments and 30% on Spot VMs, even if our traffic was close to zero.
Now, there are ways to hack it. I could just set a custom Webhook percentage across all clusters, but that wouldn’t scale well. How long would I have to do it all manually when our company constantly grows along with compute requirements?
My conversations with the Cast AI team inspired a brand-new feature: a specialized rebalancer for optimizing resource utilization across CUDs and other types of commitments.
A specialized rebalancer for CUDs: key use cases
Easier configuration and utilization boost
If I have commitments for C2 and N2D VM families, I can allocate them to multiple clusters and prioritize them based on each business use case.
One cluster may run entirely on on-demand resources, while the other cluster may run entirely on Spot VMs. But simultaneously, my committed use discounts are used at 100%. On a best-effort basis, they’re very close to 100%, and everything else runs on Spot. So, there’s no need for me to configure fixed percentages anymore.
All I need to do is set a given number of CPUs to be used on my commitments, and everything else will run on a mix of different Spot VMs. This makes a big difference, both cost-wise and implementation-wise.
Running multiple commitments on a single cluster
Moreover, I can mix commitments. In the previous setup, I couldn’t run a mix of C2s and C2Ds, but now I can just create two commitments and map them both to one or multiple clusters.. This removes a great deal of manual work around managing two different instances via static instance pools from GKE.
Now we can assign n number of commitments to one or more clusters, and it just works. I don’t have to do anything manually, like deciding on the right percentages (except for the one set at the org level).
Including on-prem deployments
Cast AI is a Kubernetes-focused solution that optimizes resource utilization in Kubernetes clusters. However, we also use a lot of non-Kubernetes resources, which are part of the same commitments.
We now calculate roughly the percentage of the total spend that is on Kubernetes and the percentage that is not related to Kubernetes, and then reduce that percentage.
For example, if I have 5000 committed cores and 2000 of them are not running on Kubernetes, essentially 40% of them aren’t on Kubernetes. I can feed my configuration to the rebalancer. For example, I only want to use 60% of my commitments so that the rest (40%) is left for my regular compute usage. And this 60% is all covered across my Kubernetes resources.
Prioritized utilization of CUDs
The next optimization step addressed the challenge of the mandatory 24-hour usage of committed resources. In CUD terms, you must commit cores every month, and there is a daily cap on the number of cores you can consume in 24 hours at a reduced rate.
As I mentioned, we see a significant usage drop and get very low traffic at night. We could allocate that entirely to our job clusters if we onboarded all of the spare on-demand capacity from our CUDs. Anything remaining would run on Spot. However, this caused several issues.
This is why we introduced prioritized utilization. For example, I want most of my CUDs to be prioritized for these clusters first because they are important, user-facing clusters. The other clusters don’t matter that much in terms of momentary interruptions, etc.
With prioritization in place, when we carry out rebalancing, whichever clusters have higher priority will get those on-demand resources back into their system. For this to work, we have to set up the system so that the other lower-priority clusters have to release these CUDs.
We set up a rebalancing schedule to reclaim CUDs and release CUDs. One schedule gets those clusters to reclaim CUDs later in the morning, and the other releases the CUD cards at night so that the other job clusters can use them more efficiently.
Results
As a result of this optimization, our on-demand usage increased at night, and Spot VM usage decreased proportionally.
Now, I don’t have to do anything manually, and we’re close to 99% commitment utilization. I used to do capacity planning twice a week for CUD management – now I do that once every three months.
The graphs below illustrate the resource utilization before optimization – it often failed to reach the line marking committed capacity:
After implementing Cast AI’s solution, our capacity utilization increased to around 98%:
Wrap up
A lot of companies like ours, especially those just starting out and being cost-conscious, really care about maximizing every dollar spent on cloud services. This made a significant difference for us because – as it turns out – purchasing CUDs is just the first step to lowering your cloud bill. The next step is optimizing your resource usage, no matter what discount level you’re running on.
