About the Foundation
The Mockingbird Foundation was created in 1996 as an all-volunteer non-profit organization dedicated to raising funds for children’s music education. Created around the fanbase of the band Phish, the Foundation has since provided over 715 awards totaling over $2.5 million across all 50 states.
Challenge
The Foundation’s core digital asset is Phish.net, a comprehensive resource about the band Phish. Although the Phish.net architecture had evolved into a microservices structure, it still operated on a single machine. Due to a lack of isolation, a resource-intensive process or failure in one part of the system could crash the entire platform, creating significant risks during peak traffic periods like live shows. Moreover, such sudden traffic spikes dramatically increased the Foundation’s cloud expenses.
Solution
To solve these issues, the team transitioned the entire system to a real microservices architecture and deployed it to Google Cloud, using Kubernetes for scalable orchestration and Cast AI for cost and resource optimization. This solution provided the needed flexibility and cost efficiency, ensuring that the site could handle spikes in traffic while keeping costs low.
Results
- ~50% cost savings that directly translate into fund increases for music grants
- Optimal performance during traffic spikes
- Future-proof infrastructure for a core non-profit service
The graphs below show the compute instance cost and the current expenses, illustrating the nearly 50% cost reduction achieved with Cast.
Compute costs before integrating Cast
Significant cost reduction achieved with Cast
Phish.net is the vehicle by which most people learn about us. It’s an extremely accurate and respected site within the community, and we consider it the gold standard for information about the band.
But maintaining Phish.net comes with costs. Every dollar we spend on infrastructure is a dollar that doesn’t go toward grants.
That’s why a solution like Cast AI is so valuable in controlling our spending. The more efficiently we manage our budget, the more we can put toward our primary mission: supporting music education.
Adam Scheinberg, Former President of The Mockingbird Foundation
Supporting music education around the United States
How did the Mockingbird Foundation get started?
The Mockingbird Foundation was initially formed because the band Phish has a strong charitable nature, and its fans share that connection to social consciousness. They’re concerned about the environment, about acceptance, treating one another well, and, of course, about music. Music is a huge draw for this community. So, we had two key goals from day one when the Foundation was formed.
The first goal was to gather all the knowledge that exists around the band and the community—essentially compiling the best, most accurate information out there. And to protect that information legally, we defined our mission as collecting, sharing, and protecting the intellectual property surrounding the band Phish.
Protecting intellectual property isn’t necessarily the most exciting cause, and it’s not something people tend to get passionate about. So, we had a second, more generally impactful goal, which was to use any money generated to support music education for children.
Why? First, it’s relevant. Second, everyone loves the idea of music and arts education, yet it tends to be chronically under-funded. So while there are plenty of important causes to support, this one resonates with our community. There’s really no downside to more music and arts education.
Adam Scheinberg, Former President of The Mockingbird Foundation
How has the support for the Foundation evolved over time?
Over the years, the fans, who were once just kids, grew up. They became doctors, lawyers, and other professionals, and their contributions grew with them. What started as a dollar for a printout at a show in 1996 turned into tens of thousands of dollars that people entrusted to us for scholarships.
Most of our grants are what we call “micro-donations.” We get many $20 or $50 donations and very few that exceed $5,000. So, most of our donations are small, but over time, they add up. We’ve given out well over $2 million, and we’re extremely proud of that.
We currently offer several types of grants. Our annual grants range between $5,000 and $10,000 and are given to schools and music programs. We also offer emergency grants. These are often unsolicited—when we learn about a school destroyed by a fire, for example, we know they’ll focus first on classrooms, and the music program might be an afterthought. So, we show up and say, “We’re going to reinstate your music program.”
Could you give some examples of when you’ve provided emergency support?
We’ve stepped in after events like the Joplin, Missouri tornadoes, hurricanes in the Northeast, and even blizzards that destroyed schools. Whenever there’s an emergency, we try to help where we can. We also offer something called a “tour grant.” Every time Phish goes on tour, in every city where the band stops, we find a local school and surprise them with a $1,000 grant.
The Foundation is entirely volunteer-run. While we pay for some professional services, like cloud hosting or tax accounting, we have no paid staff. It’s all volunteer-driven and funded by small-dollar donations from fans. Much of that money has been cycled back into music education for children, which we’re very proud of.
Modernizing the cloud infrastructure
Phish.net is your primary property, can you explain its role in the Foundation’s work?
Phish.net is the vehicle by which most people learn about us. It’s an extremely accurate and respected site within the community, and we consider it the gold standard for information about the band.
But maintaining Phish.net comes with costs. Every dollar we spend on infrastructure is a dollar that doesn’t go toward grants. That’s why a solution like Cast AI is so valuable in controlling our spending. The more efficiently we manage our budget, the more we can put toward our primary mission: supporting music education.
Adam Scheinberg, Former President of The Mockingbird Foundation
What challenges have you encountered regarding the cloud infrastructure?
What we had over the years was essentially a microservices architecture, but running on a single machine. While we did have a separate database instance and a different web server instance, we had several microservices—each responsible for different portions of the site.
For example, there’s an API that powers information on other apps and services requests from Phish.net. During a show, it fetches detailed data like, “What song was just played?” or “How many times has it been played historically?” or “What’s the gap between when it was just played and how many shows ago it was last played?” All of this detailed information is being programmatically fetched.
There’s also a backend for administrators to update the site in real-time, a forum where fans can discuss the shows, the main site that hosts statistics, and a bunch of other sub-assets. All of these relied on the same database and shared cache. We also had cron jobs running asynchronously in the background to update certain data on a schedule.
Because everything was running on this single machine, if one part of the system took up too many resources, it could crash the whole thing. For example, you might have issues with the database layer, the caching layer, or even something on the main site. Even if the problem was unrelated to the forum, the entire system could go down because they were all sharing the same computer.
What steps did you take to address this?
The first thing we needed to do was take what was essentially a microservice architecture in theory and actually deploy it as microservices. We got everything into Docker containers and used Docker Compose so we could run and develop it locally. This way, each service was isolated and could be managed independently. We also set up a good Git workflow and followed best practices for development.
What did you do once everything was broken up into microservices? How did you decide where to host them?
Once we had everything separated into microservices, the next question was, “Where do we deploy this?” We needed to decide on a cloud provider and deployment strategy. Ultimately, we settled on Google Cloud and Kubernetes, and we layered Caston top of that to manage costs and resources efficiently.
Scaling Kubernetes automatically with Cast
Why did you choose Kubernetes and Cast?
I’m not sure I would have chosen Kubernetes as a solution for this if I hadn’t been aware of Cast’s existence. Personally, I feel like Cast is kind of table stakes for using Kubernetes. It’s crazy to think about pinning every workload to a specific amount of CPU and memory, predetermining node types, and dealing with the average waste that results during idle times.
For example, on regular days, we’re scaling to maybe single-digit or low double-digit thousands of users. But on show days, there’s this massive influx of traffic.
Without Cast, we would have had to manually redeploy everything, rescale CPU, RAM, and node sizes based on expected traffic for shows. Instead, Cast handles not only the vertical and horizontal scaling of individual pods, but also the spinning up of new types of nodes, as many as we need, as fast as we need them. It ensures we can serve this traffic reliably, given the bursty nature of the project. It was absolutely critical.
For many companies, you might use Cast just for the savings, but for us, it was a combination of savings and a necessary performance feature. The site had to be hugely elastic, especially when dealing with a mix of deprecated technology and some real brilliance.
Ben Heller, Tech Team Volunteer at The Mockingbird Foundation
Did you have previous experience with Cast?
When I work at my day job, where I also use Cast, a slight cost overhead is built into the business model. But with the Mockingbird Foundation, I see every dollar we spend on cloud services as a dollar that isn’t going to someone trying to educate themselves, learn music, and make it part of their life. It feels like a tragedy if money is going to a cloud service instead of a music student.
What was the integration process like?
I’ll actually speak to the first time I integrated Cast, which wasn’t for this project but for another one—was interesting. I spoke to someone in our organization, our VP of engineering, and said, “Hey, I’d like to look at Cast AI.” The response was that we needed to have a meeting with our DevOps lead, find time to schedule it, and it turned into a large conversation.
The assumption was that this would be a multi-week or even multi-month migration, where we would have to replace all our schedulers and autoscalers. But, lo and behold, we opened up the console, and there’s a one-line script. We ran the script, and Cast was fully active on the cluster. We hit one toggle, and we were off to the races.
Ben Heller, Tech Team Volunteer at The Mockingbird Foundation
For Mockingbird and phish.net, we mainly used the console with a couple of Helm annotations to help configure specific services, but the onboarding process was as close to nothing as you can imagine for something that has such a huge impact. Honestly, that’s magic, especially in the Kubernetes world, where everything tends to be a little more complex than it needs to be.
Results: Massive cost savings and optimal performance
What level of cost savings have you reached with Cast?
It’s a bit tough to get an exact number since by the time we launched for a show day, we already had Cast in place. Typically, we see our daily bill almost double on show days.
For context, when we initially deployed the site, the dev environment alone cost about $700 a month, and this was a non-profit where every dollar counts. Now, dev and production together, even with millions of visits, are about $650 total.
Ben Heller, Tech Team Volunteer at The Mockingbird Foundation
We keep the dev environment for convenience, but it nearly doubles the cost since we need to provision a new cluster, networking, etc. We’ve been smart about this, though, and if needed, we could run just the production environment at around $300 a month, which is remarkable. On show days, we see nearly triple the compute resources, and we use scheduled rebalancing with Cast to automatically scale down when usage drops.
We care about every dollar, down to things like reducing log levels. Even saving $300 a year makes a big difference for us. With cloud services, complexity typically means high cost, but we’re aiming to handle complex traffic at low cost. Cast has allowed us to do just that.
Ben Heller, Tech Team Volunteer at The Mockingbird Foundation
Which Cast features were the biggest game-changers for you?
We dove into Cast’s standard feature set and expanded to use the workload autoscaler. As soon as the vertical autoscaler became available, we replaced our native horizontal pod autoscalers with Cast’s autoscaling.
When we initially provisioned simple EC2 instances in the dev environment, we were spending much more. Cast’s ability to schedule on-demand nodes and Spot nodes, while giving us diversity in instance types, has been key. We run as many spot instances as possible, and we haven’t had any production outages related to Cast.
We’re using a single region, single zone setup to minimize costs, with a CDN serving global traffic. This level of optimization was new to me, and I’ve taken these learnings back to commercial environments, wondering why we weren’t adopting this earlier.
How did this new setup handle peak loads during major events?
Once we had the new system in place, we stress-tested it by launching during the biggest show of the year—New Year’s Eve. This was the ultimate test to see where the bottlenecks were. Immediately, we got a sense of what areas needed improvement, and it allowed us to fine-tune the infrastructure.
It’s remarkable how little we’ve had to touch the site during daily operations, even during peak times.
Ben Heller, Tech Team Volunteer at The Mockingbird Foundation
Once we right-sized the resources, Cast has largely handled scaling up and down automatically. We don’t need to intervene before a concert – it just works. Even with unexpected jumps in traffic, the site has been remarkably stable.
Adam Scheinberg, Former President of The Mockingbird Foundation
A partnership for the future
Did you feel supported by the Cast team throughout the process of integrating and operating the solution?
Cast is a wonderful piece of elegance, and coming from your team, we’ve had excellent access to support. You’re always available via Slack. If we have questions, someone is looking into it within hours or even minutes.
We had one issue where we were struggling with scheduling certain instance types, but that was specifically due to Google Cloud’s integration. Your support team was all over it, trying different things and understanding where the disconnect was. We got through it, and now we can provision every possible instance type. In the interim, you provided a workaround by helping us set up our node templates to ignore the problematic instance type.
Having access to great support that’s conversational and friendly is especially appreciated in a world where, for most cloud services, good luck getting to a representative unless you’re a huge customer. Someone like Mockingbird would never qualify for a dedicated account rep at Google.
So, it’s been great to have that level of support from Cast, particularly when you’re dealing with something that is gating your production service and governing your relationship with your customers and consumers. It’s very much appreciated.
Ben Heller, Tech Team Volunteer at The Mockingbird Foundatio
