We often find ourselved required to route traffic from external sources towards internal services deployed to a Kubernetes cluster. There are several ways of doing this, but the most common is to use the
Service resource, or, for HTTP(S) workloads, the
Kubernetes Ingress API. The latter is finally going to be marked GA in K8s 1.19, so let’s take this opportunity to review what it can offer us, what alternatives there are, and what the future of ingress in general could be in upcoming Kubernetes versions.
Usually, we use the Service resource to expose an application internally or externally: define an entry point for the application which automatically routes distributed traffic to available pods. Since pods tend to come and go – the set of pods running in one moment in time might be different from the set of pods running that application at some later point – the Service resource groups them together with a label selector.
Service resources are broken down by type for more versatile usage. The three most commonly used types are
LoadBalancer. Each provides a different way of exposing the service and is useful in different situations.
ClusterIP is typically used to expose services internally, NodePort and LoadBalancer to expose them externally. The
Service resource essentially provides
Layer-4 TCP/UDP load balancing, simply exposing the application ports as they are.
LoadBalancer-type service could also be used for
Layer-7load balancing if the provider supports it through custom annotations.
Ingress resource is the most commonly used way of getting more fine-grained
Layer-7 load balancing for HTTP protocol-based applications.
The basic idea behind the
Ingress is to provide a way of describing higher level traffic management constraints, specifically for HTTP. With
Ingress, we can define rules for routing traffic without creating a bunch of Load Balancers or exposing each service on the node. It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL/TLS, and offer name-based virtual hosting and content-based routing.
The name Ingress may be a bit misleading, since, at first glance it makes it seem as if it were only for north-south traffic, but actually it can be used for east-west traffic as well.
Lets see a basic example:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: echo-service namespace: default spec: rules: - host: echo.example.com http: paths: - backend: serviceName: echo servicePort: 80 path: /
Ingress is one of the built-in APIs which doesn’t have a built-in controller, and an ingress controller is needed to actually implement the
Ingress is made up of an Ingress API object and an Ingress controller. As mentioned earlier, Kubernetes Ingress is an API object that describes the desired state for exposing services deployed to a Kubernetes cluster. So, to make it work an Ingress controller you will require the actual implementation of the Ingress API to read and process the Ingress resource’s information.
An ingress controller is usually an application that runs as a pod in a Kubernetes cluster and configures a load balancer according to Ingress Resources. The load balancer can be a software load balancer running in the cluster or a hardware or cloud load balancer running externally. Different load balancers require different ingress controllers.
Since the Ingress API is actually just metadata, the Ingress controller does the heavy lifting. Various ingress controllers are available and it is important to choose the right one carefully for each use case.
It’s also possible to have multiple ingress controllers in the same cluster and to set a desired ingress controller for each Ingres. Usually, we end up using a combination of these controllers for different scenarios in the same cluster. For example, we may have one for handling the external traffic coming into the cluster which includes bindings to SSL certificates, and have another internal one with no SSL binding that handles in-cluster traffic.
The Ingress spec contains all the information needed to configure a load balancer or a proxy server. Most importantly, it contains a list of rules matched against all incoming requests. Ingress resources only support rules for directing HTTP traffic. Besides the base functionality the resource provides, the various ingress controller implementations usually provide several advanced features through custom resource annotations.
An Ingress with no rules sends all traffic to a single default backend. If none of the hosts or paths match the HTTP request in the Ingress objects, the traffic is routed to the default backend.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: echo-service namespace: default spec: backend: serviceName: echo servicePort: 80
A fanout configuration routes traffic based on the HTTP URI of the request. It allows us to use a single load balancer and IP address to serve multiple backend services.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-service namespace: default spec: rules: - host: services.example.com http: paths: - backend: serviceName: service1 servicePort: 80 path: /service1 - backend: serviceName: service2 servicePort: 80 path: /service2
Hostname-based routing supports having one load balancer to handle traffic for different hostnames pointing to the same IP address.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: public-services namespace: default spec: rules: - host: service1.example.com http: paths: - backend: serviceName: service1 servicePort: 80 path: / - host: service2.example.com http: paths: - backend: serviceName: service2 servicePort: 80 path: /service2
Ingress can also provide TLS support, but it is limited to port 443 only. If the TLS configuration section in an Ingress specifies different hosts, they are multiplexed on the same port according to the hostname that’s been specified through the SNI TLS extension (if the Ingress controller supports SNI). The TLS secret must contain keys named
tls.key, which contain the certificate and private key for TLS.
apiVersion: v1 kind: Secret metadata: name: public-services-tls namespace: default data: tls.crt: base64 encoded cert tls.key: base64 encoded key type: kubernetes.io/tls
Referencing this secret in an Ingress tells the Ingress controller to secure the channel from the client to the load balancer using TLS. We need to make sure the TLS secret we have created came from a certificate that contains a Common Name (CN), also known as a Fully Qualified Domain Name (FQDN) for
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: services-with-tls namespace: default spec: tls: - hosts: - services.example.com secretName: public-services-tls rules: http: paths: - backend: serviceName: service1 servicePort: 80 path: /service1 - backend: serviceName: service2 servicePort: 80 path: /service2
As was briefly mentioned previously, it is possible to run multiple ingress controllers within a cluster. Each Ingress should specify a class to indicate which ingress controller should be used, if more than one exists within the cluster.
Before Kubernetes 1.18 an annotation (
kubernetes.io/ingress.class) was used to specify the ingress class. In 1.18 a new
ingressClassName field has been added to the Ingress spec that is used to reference the
IngressClass resource used to implement the Ingress.
apiVersion: networking.k8s.io/v1beta1 kind: IngressClass metadata: name: external-lb spec: controller: example.com/ingress-controller parameters: apiGroup: k8s.example.com/v1alpha kind: IngressParameters name: external-lb
IngressClass resources contain an optional parameters field. This can be used to reference additional configurations for the class in question. We can mark a particular IngressClass as the default for the cluster. Setting the
ingressclass.kubernetes.io/is-default-class annotation to
true on an IngressClass resource will ensure that new Ingresses, without an
ingressClassName field specified, will be assigned to the default IngressClass.
Looking for an in-depth blogpost about Istio ingress? Read on: An in-depth intro to Istio Ingress.
Ingress resource is relatively easy to use for a wide variety of use cases with simple HTTP traffic, which makes it very popular and commonly used nowadays.
On the other hand this simplicity limits its capabilities as well. The various Ingress controller implementations try to extend the feature set with custom annotations, which can help to get the job done, but they are a really awkward way to extend an API.
Istio for example can also be configured to act as an Ingress API implementation, but that diminishes its broad features, because there is just no way to properly configure a bit more complex traffic routing scenario with Ingress API.
Along with support for
Ingress, Istio offers another configuration model, using the Istio Gateway, VirtualService and DestinationRule resources. This model provides a way better and extensive customisation and flexibility than
Ingress, and allows Istio features such as monitoring, tracing, authz and route rules to be applied to traffic entering the cluster.
Gateway resource describes the port configuration of the gateway deployment, which operates at the edge of the mesh and receives incoming or outgoing HTTP/TCP connections. The specification describes a set of ports that should be exposed, the type of protocol to use, TLS configuration – if any – of the exposed ports, and so on.
VirtualService defines a set of traffic routing rules to apply when a host is addressed. Each routing rule defines matching criteria for the traffic of a specific protocol. If the traffic matches a routing rule it is sent to a named destination service that’s defined in the registry. For example, these rules can route requests to different versions of a service or to a completely different service than the one that was requested. Requests can be routed based on the request source and destination, HTTP paths and header fields, and weights associated with individual service versions.
DestinationRule defines policies that apply to traffic intended for a service, after routing has occurred. These rules specify load balancing configurations, connection pool sizes from the sidecar, and outlier detection settings to detect and evict unhealthy hosts from the load balancing pool.
Besides having a more fine-grained traffic routing configuration, this model also supports a better security by providing an option to restrict each of these resources separately, with Kubernetes RBAC.
To read more about this approach and in detail, check our other post about ingress/egress gateways with Istio. Looking for an in-depth blogpost about Istio ingress? Read the An in-depth intro to Istio Ingress.
There is ongoing work on what is probably going to be the next iteration of ingress, called
Service APIs. It basically comes from the experience the community has gained with the current Ingress implementation.
When Ingress was first designed the focus was on simplicity and centered around the developer, but nowadays it’s much more of a multi-role environment. One common example of this is when an infrastructure or SRE team provides ingress as a service to application developers.
Service APIs take a similar approach as the Istio model does. It splits up the API into different resources for different purposes in order to provide separation and role-based control of who can create ingress, consume an ingress and also provide support that specifies more complex routing and load balancing features on both L4 and L7 protocols. Service APIs also make the model more extendable for future use-cases.
Gateway define the type of LB infrastructure that the user’s Service will be exposed on. This is modelled on existing work in the Kubernetes IngressClass types.
Route describes a way to handle traffic given protocol-level descriptions of the traffic. The route is a pure configuration and can be referenced from multiple gateways. There are different route resources for each protocol, e.g.
The combination of
Service(s) will define a load-balancer configuration. The diagram below illustrates the relationships between the different resources:
These new APIs are not intended to replace any existing APIs, but instead provide a more configurable alternative for complex use cases. To read more about the current state of
Service APIs check Kubernetes github page or its docs.
The Ingress API is on its way toward graduating from beta to a stable API in Kubernetes 1.19. After that, it’ll most probably go into maintenance mode and will continue to provide a simple way of managing inbound network traffic for Kubernetes workloads. We should note that there are other approaches to ingress on Kubernetes, and that work is also currently underway on a new, highly configurable set of APIs that will provide an alternative to Ingress in the future.