Cisco Tech Blog CiscoTech Blog Close
Home Contact

Inside Kubernetes Networking

Author

Kubernetes Networking is a core abstraction of Kubernetes. In a nutshell, the Kubernetes Networking Model guarantees that all Kubernetes Pods on a cluster are able to communicate. In addition, on top of the Kubernetes Networking Model, Kubernetes provides additional core abstractions, Kubernetes Services and Kubernetes Ingress. Using a systems modeling approach, this blog post will explore Kubernetes Networking. Here, we will develop a concise mental model to reason about Container-to-Container and Pod-to-Pod communication. Future blog posts will explore Kubernetes Services and Kubernetes Ingress.

All models are wrong, but some models are useful ⎯ George Box

This blog post discusses Kubernetes Networking on a conceptual level, not an implementation level This blog post discusses Kubernetes Networking on a conceptual level, not an implementation level

How To Think About Networking πŸ”—︎

Without a doubt, networking is a vast and complex field that demands years of theory and practice to claim proficiency. However, here we reason about networking on a conceptual level, skipping a host of details that would be necessary if we reasoned on an implementation level.

Figure 1. An idealized model of a network Figure 1. An idealized model of a network

Figure 1. illustrates a network as a Network Graph that consists of a set of Nodes and a set of Links between the Nodes: One Node may exchange a message with another Node if and only if there exists a link between the Nodes.

Figure 2. Message Exchange, Structural View Figure 2. Message Exchange, Structural View

A Node, the Source, exchanges a message with another Node, the Target, by placing the message in the Target’s input queue. The message exchange is represented by a Send Event, Send●M, observed at the Source, and a corresponding Receive Event, Recv●M, observed at the Target.

Figure 3. Message Exchange, Behavioral View Figure 3. Message Exchange, Behavioral View

A Node in the network is either a Process or a Switch. Processes produce and consume messages, Switches process messages according to their Forward Information Base (FIB).

Figure 4. Forward Information Base of S1 and S2 Figure 4. Forward Information Base of S1 and S2

Figure 2. illustrates the Forward Information Base of Switch S1 and S2. On receiving a message, each Switch consults its Forward Information Base to determine whether to deliver, forward, or discard the message.

The Switch

  • matches the message’s header, that is, the source address, source port, target address, and target port with its Forward Information Base
  • performs the associated action, discarding being the default.

In the next blog post Kubernetes Services, we will extend the model of a Switch to allow for Network Address Translation.

Kubernetes Networking Model πŸ”—︎

The Kubernetes Networking Model is a descriptive networking model, that is, any network that satisfies the specification of the Kubernetes Networking Model is a Kubernetes Network.

However, Kubernetes does not prescribe how to implement the Networking Model. In fact, many alternative implementations, called Network Plugins, exist today. This section describes the Kubernetes Networking Model in terms of a set of constraints on message exchange.

Constraint ● Network Addressable Entities πŸ”—︎

The Kubernetes Networking Model defines three Addressable Entities, K8s Pods, K8s Nodes, and K8s Services, where each unique entity is associated with a unique IP Address.

∧ (K8s-Pod(E₁) ∨ K8s-Node(E₁) ∨ K8s-Service(E₁))
∧ (K8s-Pod(Eβ‚‚) ∨ K8s-Node(Eβ‚‚) ∨ K8s-Service(Eβ‚‚)):
  addr(E₁, a) ∧ addr(Eβ‚‚, a)β‚‚
   ⟺ E₁ = Eβ‚‚

However, the Networking Model does not make any further statements about these IP Addresses. For example, the Kubernetes Networking Model does not make any further statements about the IP Address Spaces these IP Addresses are drawn from.

Constraint ● Container to Container Communication πŸ”—︎

The Kubernetes Networking model requires that a Container C₁ executing in the context of a Pod P can communicate with any other Container Cβ‚‚ executing in the context of P via localhost.

K8s-Pod(P) ∧ K8s-Container(C₁, P) ∧ K8s-Container(Cβ‚‚, P):
 open(Cβ‚‚, p) 
  ⟹
   Send(e, C₁, 127.0.0.1, _, 127.0.0.1, p)
    ⟹
     Recv(e, Cβ‚‚, 127.0.0.1, _, 127.0.0.1, p)

Constraint ● Pod to Pod πŸ”—︎

The Kubernetes Networking model requires that a Container C₁ executing in the context of a Pod P₁ can communicate with any other Container Cβ‚‚ executing in the context of a Pod Pβ‚‚ via the address of the Pβ‚‚.

∧ K8s-Pod(P₁) ∧ K8s-Container(C₁, P₁)
∧ K8s-Pod(Pβ‚‚) ∧ K8s-Container(C2, Pβ‚‚):
 addr(P₁, sa) ∧ addr(P₁, ta) ∧ open(Cβ‚‚, tp)
  ⟹
   Send(e, C₁, sa, sp, ta, tp)
    ⟹
     Recv(e, Cβ‚‚, sa, sp, ta, tp)

Constraint ● Process to Pod πŸ”—︎

The Kubernetes Networking model requires that a Process, called a Daemon D, hosted on a Node N, can communicate with any Container C executing in the context of a Pod P hosted on N via the address of the P.

K8s-Node(N) ∧ K8s-Daemon(D) ∧ K8s-Pod(P) ∧ K8s-Container(C, P):
 host(N, D) ∧ host(N, P) ∧ addr(P, a) ∧ open(C, p)
  ⟹
   Send(e, D, _, _, a, p)
    ⟹
     Recv(e, C, _, _, a, p)

Kubernetes Networks as Network Graphs πŸ”—︎

This section describes the Kubernetes Networking Model in terms of a Kubernetes Network Graph, an idealized model. Figure 5. illustrates the example used in this section: A Kubernetes Cluster K₁ that consists of two Nodes. Each Node hosts two Pods. Each Pod executes two Containers, one Container listening on port 8080, one Container listening on port 9090. Additionally, each Node hosts one Daemon.

Figure 5. Kubernetes Cluster K₁ Figure 5. Kubernetes Cluster K₁

We can model a Kubernetes Cluster Network as a Graph with a set of Nodes and a set of Links.

Nodes πŸ”—︎

Every K8s Container C maps to a Network Process C

K8s-Pod(P) ∧ K8s-Container(C, P):
  Process(C)

Every Daemon D maps to a Network Process C

K8s-Daemon(D):
  Process(D)

Every K8s Pod P maps to a Network Switch P, the Pod Switch

K8s-Pod(P):
  Switch(P)

Every K8s Node N maps to a Network Switch N, the Node Switch

K8s-Pod(N):
  Switch(N)

Each Container C is linked to its Pod Switch P

K8s-Pod(P) ∧ K8s-Container(C, P):
 link(C, P)

Each Daemon D is linked to its Node Switch N

K8s-Node(N) ∧ K8s-Daemon(D):
 host(N, D)
  ⟹
   link(D, N)

Each Pod Switch P is linked to its Node Switch N

K8s-Node(N) ∧ K8s-Pod(P):
 host(N, P)
  ⟹
   link(P, N)

Each Node Switch N₁ is linked to every other Node Switch Nβ‚‚

K8s-Node(N₁) ∧ K8s-Node(Nβ‚‚):
 N₁ β‰  Nβ‚‚
  ⟹
   link(N₁, Nβ‚‚)

Forward Information Base at the Pod Switch πŸ”—︎

Figure 6. Forward Information Base of P₁ Figure 6. Forward Information Base of P₁

  • Delivery on localhost
K8s-Pod(P) ∧ K8s-Container(C, P):
 open(C, p) 
  ⟹
   [* * 127.0.0.1 p Deliver(C)] in FIB[P]
  • Delivery on Pod Address
K8s-Pod(P) ∧ K8s-Container(C, P):
 addr(P, a) ∧ open(C, p) 
  ⟹
   [* * a p Deliver(C)] in FIB[P]
  • Local Forwarding Rule
K8s-Node(N) ∧ K8s-Pod(P):
 host(N, P) 
  ⟹
   [* * * * Forward(N)] in FIB[P]

Forward Information Base at the Node Switch πŸ”—︎

Figure 7. Forward Information Base N₁ Figure 7. Forward Information Base N₁

  • Node to Pod Forwarding Rule
K8s-Node(N) ∧ K8s-Pod(P):
 host(N, P) ∧ addr(P, a)
  ⟹
   [* * a * Forward(P)] in FIB[N]
  • Node to Node Forwalding Rule
K8s-Node(N₁) ∧ K8s-Node(Nβ‚‚) ∧ K8s-Pod(P):
 N₁ β‰  Nβ‚‚ ∧ host(Nβ‚‚, P) ∧ addr(P, a)
  ⟹
   [* * a * Forward(Nβ‚‚)] in FIB[N₁]

Examples πŸ”—︎

This section walks through some examples and follows the Life of a Message in the Kubernetes Cluster Network K₁.

Container to Container πŸ”—︎

Here, Container C₁.₁ needs to communicate with Container C₁.β‚‚:

  • C₁.₁ executes in the context of Pod P₁
  • C₁.β‚‚ executes in the context of Pod P₁

Figure 8. C₁.₁ to C₁.β‚‚ via 127.0.0.1:9090 Figure 8. C₁.₁ to C₁.β‚‚ via 127.0.0.1:9090

Pod to Pod Communication, Intra Node πŸ”—︎

Here, Container C₁.₁ needs to communicate with Container C₃.₁:

  • C₁.₁ executes in the context of Pod P₁ which is hosted on Node N₁.
  • C₃.₁ executes in the context of Pod P₃ which is hosted on Node N₁.

Figure 9. C₁.₁ to C₃.₁ via 10.1.1.2:8080 Figure 9. C₁.₁ to C₃.₁ via 10.1.1.2:8080

Pod to Pod Communication, Inter Node πŸ”—︎

Here, Container C₁.₁ needs to communicate with Container Cβ‚‚.₁:

  • C₁.₁ executes in the context of Pod P₁ which is hosted on Node N₁.
  • Cβ‚‚.₁ executes in the context of Pod Pβ‚‚ which is hosted on Node Nβ‚‚.

Figure 10. C₁.₁ to Cβ‚‚.₁ via 10.1.2.1:8080 Figure 10. C₁.₁ to Cβ‚‚.₁ via 10.1.2.1:8080

Deamon to Pod Communication πŸ”—︎

Here, Daemon D₁ needs to communicate with Container C₁.₁:

  • D₁ is hosted on Node N₁.
  • C₁.₁ executes in the context of Pod P₁ which is hosted on Node N₁.

Figure 11. D₁ to C₁.₁ via 10.1.1.1:8080 Figure 11. D₁ to C₁.₁ via 10.1.1.1:8080

Conclusion πŸ”—︎

The Kubernetes Networking Model is a permissive networking model, that is, any network that satisfies the constraints of the Kubernetes Networking Model is a valid Kubernetes Network.

Mapping the Kubernetes Networking Model to a Networking Graph enables us to reason about the network on a conceptual level, skipping a host of details that would be necessary if we reasoned on an implementation level.