This post explores Kubernetes external services, methods for using them in your environments, and configuration best practices.
What is a Kubernetes external service?
The short answer is:
A Kubernetes service with type=ExternalName.
Although it seems like a slight difference, it turns the Kubernetes service concept on its head.
Services normally expose a set of cluster pods/endpoints to consumers. External services do the opposite – they expose what is (usually, but not mandatory) an endpoint outside the cluster to consumers inside the cluster.
In practice, creating a service of type ExternalName causes Kubernetes to populate its DNS with a CNAME record for the service target. Resolving the service name will give the client another DNS name, which needs to be resolved again before finding the final server IPs.
The process is shown below:
External services rely exclusively on DNS, so there are no label selectors, ports, or endpoints to track. This is another difference from other Kubernetes service types.
Use cases for Kubernetes external services
There are various situations in which using an external service may be useful. They usually revolve around abstracting the details of the external service from the applications:
- Hybrid deployments – in a hybrid deployment, certain services function in an on-premises server environment while others run in a Kubernetes cluster.
- External dependencies – because of particular needs or dependencies, some services, such as databases or storage, might need to be located outside a cluster.
- Migration or restructuring – using an external service will help you prevent needless downtime and efficiently reroute traffic during a migration or restructuring phase of your Kubernetes cluster.
- Configuration centralization – when many workloads use an external service, it can be hard to properly propagate network configuration to all (for example, due to URL changes). ExternalName services provide a central spot to update, and all workloads will automatically pick up the change via normal DNS resolution.
- Multi-cloud setup – an organization could have workloads spanning multiple cloud providers. However, some dependencies might only be deployed in one of them. External services provide a way to refer to those dependencies in the same way, regardless of their location.
ExternalName: practical example
To see ExternalName services in action, imagine a disaster recovery situation.
In this scenario, the database lives outside the Kubernetes cluster (for example, it is a PostgreSQL instance deployed in AWS RDS).
When the database goes down, workloads must be migrated to another instance in another region to continue operations. However, the database is used by many applications owned by different teams, and there is a tight deadline for reaction time.
We need a way to propagate the change quickly and as transparently as possible to clients. Enter ExternalName services!
First, the database endpoint will be abstracted away via a service:
apiVersion: v1
kind: Service
metadata:
name: webscaler-db
namespace: default
spec:
type: ExternalName
externalName: primary-db.example.comThe parts that differ from a regular service are:
- type is ExternalName
- externalName is populated
- no selector or ports are defined
To simulate our client workload, we will deploy a pod and connect to it. Here’s the pod definition:
apiVersion: v1
kind: Pod
metadata:
name: webscaler-client
namespace: default
spec:
containers:
- name: ubuntu-container
image: ubuntu:latest
command: ["/bin/bash", "-c", "while true; do sleep 3600; done"]Let’s see the service in action:
root@local:/# kubectl exec -it webscaler-client -- /bin/bash
## Installs curl and some dns helpers
root@webscaler-client:/# apt-get update && apt-get install -y dnsutils curl
root@webscaler-client:/# nslookup webscaler-db
## Ignore the error here as we use made up DNS name on the backend
;; Got recursion not available from 10.96.0.10
Server: 10.96.0.10
Address: 10.96.0.10#53
webscaler-db.default.svc.cluster.local canonical name = primary-db.example.com.
** server can't find primary-db.example.com: NXDOMAIN
As expected, the DNS server resolves webscaler-db.default.svc.cluster.local to primary-db.example.com.
Now let’s exit, change the service, and try again:
root@webscaler-client:/# exit
## Patch the service to initiate disaster recovery and retry the experiment
root@local:/# kubectl patch service webscaler-db --type='json' -p='[{"op": "replace", "path": "/spec/externalName", "value": "secondary-db.example.com"}]'
root@local:/# kubectl exec -it webscaler-client -- /bin/bash
root@webscaler-client:/# nslookup webscaler-db
;; Got recursion not available from 10.96.0.10
Server: 10.96.0.10
Address: 10.96.0.10#53
webscaler-db.default.svc.cluster.local canonical name = secondary-db.example.com.
** server can't find secondary-db.example.com: NXDOMAINAs you can see, the workload received the updated DNS name (secondary-db.example.com) without needing a restart. So, we successfully propagated the change to all workloads using the service in seconds. Go, external services!
To show a working example (and a pitfall of external services), we will now deploy another service that uses httpbin.org as its backend:
apiVersion: v1
kind: Service
metadata:
name: webscaler-manual
namespace: default
spec:
type: ExternalName
externalName: httpbin.orgReturning to our test pod, we will do a lookup and then try to call the service over http and https. Let’s see the results:
root@webscaler-client:/# nslookup webscaler-manual
;; Got recursion not available from 10.96.0.10
Server: 10.96.0.10
Address: 10.96.0.10#53
webscaler-manual.default.svc.cluster.local canonical name = httpbin.org.
Name: httpbin.org
Address: 34.235.14.55
Name: httpbin.org
Address: 75.101.249.194
Name: httpbin.org
Address: 44.217.225.103
Name: httpbin.org
Address: 52.45.59.35
Name: httpbin.org
Address: 3.224.242.112
Name: httpbin.org
Address: 3.230.74.9
root@webscaler-client:/# curl -X GET http://webscaler-manual/anything?test=value
{
"args": {
"test": "value"
},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Host": "webscaler-manual",
"User-Agent": "curl/8.5.0"
},
"json": null,
"method": "GET",
"origin": "62.73.121.222",
"url": "http://webscaler-manual/anything?test=value"
}
root@webscaler-client:/# curl -X GET https://webscaler-manual/anything?test=value
curl: (60) SSL: no alternative certificate subject name matches target host name 'webscaler-manual'
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@webscaler-client:/# curl -X GET https://webscaler-manual/anything?test=value -k
{
"args": {
"test": "value"
},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Host": "webscaler-manual",
"User-Agent": "curl/8.5.0"
},
"json": null,
"method": "GET",
"origin": "62.73.121.222",
"url": "https://webscaler-manual/anything?test=value"
}
Let’s break down each step.
First, the nslookup returns proper IPs since we resolved a real DNS name.
Second, doing a curl works as expected and hits the server, returning the values we passed. However, notice that the https request fails with an SSL error: curl: (60) SSL: no alternative certificate subject name matches target host name 'webscaler-manual'. This is because our client expects a hostname of webscaler-manual, but the server can only present a certificate for httpbin.org. Therefore, the SSL/TLS connection cannot be established.
To verify this, we issue the same request with the -k flag (which tells curl to ignore certificate errors) – and the request now passes. Therefore, special care must be taken when using protocols dependent on hostnames.
Kubernetes external services “gotchas”
1. Load balancing
A service of type ExternalName does not do any load balancing out of the box, unlike regular Kubernetes services. Even if the DNS name behind the service resolves to multiple IPs, load balancing must be done separately. We touch on this below in the best practices section.
2. Protocol issues
Some protocols rely on the hostname that a client uses to work. Since external services “hide” the real hostname behind the service’s own name, connections over these protocols could break.
The most common example of such a protocol is TLS, as the server’s certificate does not match what the client expects based on the DNS name. This means protocols like HTTPS, WSS, MQTTS and more are all affected, and care should be taken if they are used with external services.
3. IPs don’t work
Although external services use DNS, they don’t support direct resolution to an IP – there must always be a canonical DNS name first. You can’t use an IP value in the service’s ExternalName field.
Even though the IP string looks correct, it’s treated as a DNS name composed of digits, not an IP address. As such domain names are not valid on the public internet, the result is probably a malfunctioning service.
Best practices for Kubernetes external services
1. Understand your traffic patterns
You can maximize performance and reliability by customizing your application’s architecture and interactions with the external service to thoroughly understand your traffic patterns.
For example, you can design your application to reuse connections wherever possible or use connection pooling if you know it will connect to an external service frequently and briefly.
Knowing your traffic patterns may help you make sound decisions about load balancing or failover solutions for external services, among other things, that are not related to Kubernetes.
2. Make DNS resolution dependable and quick
Each call to an external service requires at least two DNS hops. This means DNS performance could become a problem in high-traffic or low-latency scenarios.
Use the NodeLocalDNS project or implement caching in CoreDNS for internal DNS resolution. This method can significantly speed up access by reducing the time it takes to resolve the ExternalName service on the first hop.
Also, ensure third-party DNS providers outside your Kubernetes cluster are dependable and quick in translating the ExternalName value (such as my.database.example.com) to the real IP address. Since these providers are often outside the cluster owner’s control, care should be taken to measure performance and reliability before going live.
Applications that depend on the external service may see poor performance if DNS resolution is sluggish or inconsistent. Any problems or delays in DNS resolution will negatively impact here since the external service relies on it for traffic routing.
3. Make adjustments in advance
The endpoint address of a Kubernetes external service may vary over time, especially if it isn’t under your control. Make sure that changing the ExternalName field in your service is easy to do – this is how you can prepare for these changes.
You can eliminate this manual effort by automating this process using Kubernetes operators or configuration management tools.
4. Put security first
Ensure all communications between the external endpoint and the Kubernetes cluster are secure. An external name can point to any DNS endpoint, so if the endpoint is changed to a malicious value, you’re looking at a severe security issue. Also, application Pods may connect to an unauthorized endpoint outside the cluster due to malicious alterations.
To improve security, consider implementing network restrictions, encryption, or even a service mesh if the data is sensitive.
To reduce this risk, ensure that the service object is protected from unauthorized access by non-admin users. Apply strict controls over who can edit service objects and use role-based access control (RBAC) to limit access to Kubernetes objects.
5. Manage traffic and load balance efficiently
Since external services do not provide load balancing, care must be taken to distribute traffic properly.
- Implement load balancing client-side – if the external service DNS resolves to multiple IPs, clients can do local load-balancing and choose which IP to use.
- Implement load balancing server-side – sometimes, the external service is also under your ownership; it just happens to live outside the cluster. This means all common ways to load balance the traffic on the server side are applicable (for example, using a load balancer and pointing the external service to it).
- Use multiple external services – this is similar to client-side load balancing but allows to map multiple DNS names and not just IPs.
6. Test your external services
Test your external services regularly to ensure they perform well and are connected. Have a plan for handling situations where the external endpoint is unreliable or slow.
7. Monitoring and troubleshooting
Keep an eye on the functionality and accessibility of both your external services and the external services they link to. For efficient monitoring, integrate with Prometheus and Grafana for metrics and alerts.
You should be able to diagnose connectivity and performance issues and easily debug misconfigurations and network problems.
Wrap up
By following the guidelines and best practices outlined in this guide, you can ensure that your Kubernetes clusters effectively utilize ExternalName services, leading to smoother operations and improved application performance.
If you stay proactive in monitoring and adjusting configurations as needed, your external services will remain reliable and robust components of your overall infrastructure.
