In the good old days of virtualization, networking was relatively simple. Each virtual machine (VM) had an IP address, and each application in that VM was mapped to a TCP or UDP port. If one app had to talk to another app, basic TCP/IP networking provided all that was needed.  

Then along came containers, and the world as we knew it vanished. In some ways, container networking is merely an extension of VM networking; in other ways, it’s different. But once you add on a container orchestrator like Kubernetes, you have the potential of thousands of Nodes and millions of Pods being spun up and torn down at all hours of the day or night. This is galactically different for networking.  

This blog will look at some essential Kubernetes networking tools to make networking work without having your brain melt.   

Kubernetes Recap (Ultra Brief Version) 

Before jumping into Kubernetes networking, we need to recap some of the basic concepts and architecture that make Kubernetes so powerful and introduce some significant potential roadblocks for networking. Let’s start with two key objects and two key concepts.  

Need the full recap? Skim through our article: What is Containerization? A Quick Tutorial and History.

Key Objects

The key objects are Pods and Nodes.

Pods

A Pod is the closest thing to a container in Kubernetes. It holds a container and provides it with the resources it needs to run, such as – RAM, storage, access to CPU, IP address, TCP/UDP port, etc. 

(In fact, if you have a couple of containers that can work with the same set of resources, you can put all those containers into a single Pod.) The graphic below illustrates a Pod with two containers in it and an IP, RAM and storage. It’s a good spot to mention that every Pod receives its IP address dynamically when it’s created (like DHCP.)  

Nodes

A Node is a compute platform (for example, a VM). There are two types of Nodes in Kubernetes: 

• Worker Nodes: Platforms for running Pods 
• Master Nodes: Manage a related collection of Worker Nodes with their Pods (called a Cluster). 

As illustrated, end users generally interact with Worker Nodes to connect with applications running in Pods, while administrators interact with Master Nodes to manage the Cluster.   

Key Concepts

The two key concepts are “desired state” and “current state.” 

The desired state is the ideal state you want your Cluster to be in.

The current state is the actual status of your Cluster at any given time.

Kubernetes allows you to define your Cluster’s desired state – for example, 100 Nginx Pods running on Ottawa Nodes, 200 Nginx Pods running on Chicago Nodes, 50 MySQL Pods running on Toronto Nodes, etc. Once that is defined, the Kubernetes Master Nodes monitor all the Workers and Pods in the Cluster to ensure that the current state of the Cluster matches the configured desired state. If one of the Nginx Pods in Ottawa stops responding, the Master Nodes will take action to rectify that situation automatically.  

To achieve this goal of aligning the current state to the desired state, Kubernetes uses a layered architecture that is both clever and complicated. Each object has its own responsibilities and works together with other objects with other responsibilities. At the lowest level, a Pod’s role in life is to run the container(s) inside it. Furthermore, Pods are intended to be mortal; don’t expect the Pods you spin up today to be around tomorrow. If a container inside a Pod falls over, the Pod won’t actively tell anybody else about that problem; that is someone else’s job.  

One layer up from Pods is a Deployment, which runs a collection of Pods and monitors their health. In our scenario above, we could create a Deployment to keep those 100 Ottawa Pods up and running. If one of those Pods stops responding, rather than wasting effort trying to repair it, the Pod is torn down, and a new Pod is created to replace it (at the Deployment level). This is like PC support in a big organization, where the fastest solution for laptop problems is often to re-image or even replace the laptop.

This approach is excellent for creating highly available and scalable applications based on lots and lots of Pods, but it also introduces some challenges for networking. In particular, every Pod receives its IP address dynamically upon creation. A Pod networking scheme based on IP addresses would be impossible in an enterprise situation.  

Kubernetes Networking Services  

Kubernetes Services provides a sensible approach for networking with mortal Pods. A Service is a Kubernetes object that enables you to access multiple identical Pods based on labels attached to those Pods, regardless of the Pods’ IP addresses. It’s helpful to think of Services as using “names” (loosely defined) rather than IPs or ports. We’ll investigate five types of Kubernetes Services in the rest of this blog.

ClusterIP Service

ClusterIP is the default Service type in Kubernetes. The ClusterIP Service is designed for app-to-app communication within a Cluster. For example, consider a web app and an SMB app that need to work together. In the diagram, the web app Pods have label X, and the SMB Pods have label Y. 

If the web app Pod at 10.0.0.2 needs something from the SMB app, it could send a request to the Pod at 10.0.0.4:445 … until that Pod inevitably dies and is replaced by an identical Pod at another IP address. 

To resolve that issue, we can define two ClusterIP Services: “Service X” for the web app and “Service Y” for the SMB app. Each ClusterIP Service presents a stable virtual IP address and port value where Pods can contact it. The other end of the Service connects to Pods with the correct label value (“endpoints”), regardless of the Pod’s IP address. If there are multiple Pods with the valid label value, the ClusterIP Service does some load balancing across those Pods.  

Note that the ClusterIP Service can connect to relevant Pods across multiple Nodes in the Cluster. Note also that the Label is defined in the Pod’s manifest file, so every new instance of that Pod will have that same Label, which will make it available to the Service regardless of its IP address. 

An important detail is that the IP address presented by the ClusterIP Service is called the “ClusterIP,” and the presented port is called the “Port .” (This is where I always ask why they couldn’t have called this the “ClusterPort!”) 

To look a little deeper, here is a sample ClusterIP Service manifest file: 

apiVersion: v1
kind: Service
metadata:
name: service-x-cip #ClusterIP for web app
spec:
  type: ClusterIP #optional for a ClusterIP Service
  ports:
    - port: 8080 #Port for the Service (IP is assigned) 
      protocol: TCP 
      targetPort: 80 #port on the Pod
  selector:
    app: web-app #Service will look for pods with this Label 

The “type” field is optional for a ClusterIP Service; if there is no “type” field in a Service manifest, Kubernetes defaults to ClusterIP. Also, the ClusterIP address is automatically assigned by Kubernetes when it creates the Service. To view that IP address, use this command: 

# kubectl get svc service-x-cip

For more detail, add the “-o wide” option. For a thorough display of the functional properties of the Service, use this command: 

# kubectl describe svc service-x-cip

NodePort Service 

The NodePort Service type makes the containerized app accessible from outside the Cluster. The NodePort Service exposes the application Pods at a static port (the NodePort) on each Node. In keeping with the layering theme, the NodePort Service automatically creates a corresponding ClusterIP Service to handle any app-to-app networking needs; the NodePort Service does not replicate the ClusterIP functionality. 

The diagram illustrates a NodePort Service that exposes the Nginx Pod on Node 1. To access that Pod, the user can browse (or curl) to 10.0.0.2:30001 (that is, NodeIP:NodePort). While the diagram shows only one Nginx Pod, multiple endpoints are also possible. As with a ClusterIP Service, a NodePort Service defines its endpoints through a selector in the Service manifest file and matching labels in the Pod manifest file. So, if there are multiple Nginx Pods on this Node, with the correct Label, the NodePort Service can send the request to any of those Pods.

You need to access the Node by IP address, which means the NodePort Service is Node-specific. This is different from a ClusterIP Service, which can access relevant Pods on many other Nodes in the Cluster. This also means that the NodePort Service is accessible only to people who can ping the Node; it is not wide open for internet access. 

Let’s look at a NodePort Service manifest file: 

apiVersion: v1
kind: Service
metadata:
  name: service-x-np
spec:
  type: NodePort # this is required for NodePort Service
  ports: 
    - port: 8080 #Port for the ClusterIP Service
      protocol: TCP
      targetPort: 80 #port on the Pod
  selector:
    app: web-app #Service will look for pods with this Label 

This manifest file is pretty similar to the ClusterIP manifest. However, you must specify the type as “NodePort”; if you don’t, Kubernetes will default to a ClusterIP Service. And the “port” line is used for the automatically generated ClusterIP Service. 

So what about the NodePort, the port for this NodePort Service?  

Kubernetes gives you the option of specifying the NodePort (port) value yourself or letting it be set automatically. Either way, the value of the NodePort is always between 30,000 and 32,767. In the case above, we let Kubernetes pick the NodePort value for us and ended up with 30,001.  

As with our ClusterIP example, to view the NodeIP and NodePort values for this service, enter:

# kubectl get svc service-x-cip

For more details, enter:

# kubectl describe svc service-x-cip

While the NodePort Service is glued to a particular Node, you can extend it to other Nodes in the Cluster by setting aside the NodePort port value on the other Nodes and proxy them over to the NodePort Service. For example, you could set a rule that any traffic going to 10.0.03:30001 will get redirected to 10.0.0.2:30001. But that hardly seems like a proper way to make an app available on the web…  

LoadBalancer Service 

A LoadBalancer Service is the proper way to expose a Service to the internet. This Service type rides on top of the load balancer function of a cloud provider like AWS or Azure. 

Let’s look at a sample manifest file for a LoadBalancer Service:

apiVersion: v1
kind: Service
metadata:
  name: service-x-lb
spec:
  type: LoadBalancer # definitely required!
  ports:
    - port: 8080 #Port for the ClusterIP Service
      protocol: TCP
      targetPort: 80 #port on the Pod
  selector:
    app: web-app #Service will look for pods with this Label

The structure should be familiar by now. As with the NodePort Service, be sure you specify the Service type, or you’ll end up with a ClusterIP Service. Not surprisingly, a LoadBalancer Service will automatically create a NodePort Service and a ClusterIP Service to handle other parts of the equation. But the cloud provider will decide how those backend Services are provisioned. A LoadBalancer Service generates multiple provider-managed VMs that handle the actual traffic. And the fundamental load balancing decisions are managed by the provider – not by Kubernetes. This may be good or bad depending on your point of view.  

The main downside is that each LoadBalancer Service needs its own public IP address; if you want multiple LoadBalancers, that can get expensive quickly. Some cloud providers allow you to specify the LoadBalancer Service IP address, so at least there’s that.   

Other Kubernetes Networking Service Types 

The above three Services all work towards a common goal of making distributed apps available on the internet and ensuring they can communicate together. While that covers most use cases, some other scenarios come into play from time to time.   

ExternalName Service 

The ExternalName Service is designed to point to a canonical domain name for DNS resolution. This Service type redirects traffic through an internal alias and then to an external resource. For example, you might use an ExternalName Service to direct traffic from Pods to an external database, as in the following manifest file: 

kind: Service
apiVersion: v1
metadata:
  name: external-db
spec:
  type: ExternalName # don’t forget this line!
  externalName: mysql.cengn.ca # note the FQDN structure
  ports:
    - protocol: TCP
      port: 80 

Headless Service  

A Headless Service maps to one specific Pod without creating a virtual IP (entirely different from a ClusterIP Service). Headless Services are valid for stateful applications, where each Pod stores unique, persistent data. Additionally, they allow users to interact directly with the target Pod.  

Here’s a sample manifest file for a Headless Service. Notice that the usual “type” field is replaced by the line “clusterIP: None.” 

apiVersion: v1
kind: Service
metadata:
  name: headless-svc
spec:
  clusterIP: None # this is the crucial bit here
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  selector:
    app: myheadlessapp

Recap

So, there you have it.

Kubernetes enables you to run highly available containerized apps using millions of Pods on thousands of Nodes around the world. Along the way, we ran into a serious problem with networking those Pods together, but Kubernetes also provides a clever set of solutions through its networking Services.

Want to learn more about Kubernetes? Go from intro to expert with CENGN’s Docker + Kubernetes Basics and Advanced courses.

Build your confidence with hands-on experience in online labs and learn on your own schedule.

About the Author

Peter Heath is CENGN’s Senior Manager for Training Programs, driving CENGN Academy’s strategic focus and leading the training development/delivery team. Throughout his career, Peter has developed expertise in multiple telecom, networking and information technology specializations. Before joining CENGN, Peter had broad experience in teaching, training, and program development across the higher education and commercial education sectors. Peter holds a Ph.D. in Mathematics from Carleton University in Canada and has achieved multiple networking and IT industry certifications.

More by Peter Heath