Realtime Vulnerability Management for Containers: Time to Step Things up!

Securing Kubernetes environments remains a major challenge. In recent years we’ve seen a number of container vulnerabilities, data breaches, and plain misconfigurations. In the SANS “Endpoint Protection and VResponse” survey from 2018, ”’42% of respondents indicated that at least one of their endpoints had been compromised”’. A lot of development and devops teams create their container-based infrastructure locally before pushing into a repository. It’s incredibly common to set up Docker on a local workstation, yet if an attacker can take over that workstation, it becomes an immediate avenue for exploitation. There’s been staggering growth in use of containers and container technology recently. ”’Every time there’s a widespread adoption in technologies, there are bugs. Container technology is no different”’. These are just some recent examples of vulnerabilities container runtime engines and cloud service platforms have experienced:

• Docker Race Condition: CVE-2018-15664 gave root file system access
• Docker Container Escape: CVE-2019-5736 for runc runtime is a full escape
• Docker Hub database breach affected over 190,000 accounts in April 2019

There has been a recent escalation in attacks on container technology as is seen here:

It will come as no surprise then that we need better visibility in container management, auditing needs to be turned on and we need a better model on detection and response capabilities. Many containers only run on a short timeframe, frequently less than an hour. The highly ephemeral nature of the way that container based infrastructure is built, deployed and operated doesn’t work with traditional security systems we’ve had in the past. As organizations come to rely more and more on containers and Kubernetes, they’ll need to put processes in place to vulnerability scan container images, secure orchestration tools, and leverage secure code repositories ”’all in real-time”’.

The Problems with the ‘Traditional’ and ‘Current’ Approach to Container Security

There are three different approaches to container security – the traditional approach, the one currently used by the masses and Cisco Cloud Native Security Solution’s approach. The traditional approach relies on tightening the container’s code to address vulnerabilities and sandboxing images and packages. During deployment and runtime, the configuration includes defined and controlled access privileges, including on volume and OS/Host. This then needs to be incorporated in the orchestration side by applying RBAC/API authorization and Secrets. There are significant issues with the traditional approach such as not being able to fine-tune roles and not maintaining networking security sufficiently. The current approach involves hooking the vulnerability scanning into the CI/CD. It is a good solution because it enables the developers to get feedback BUT it is not enough. Flaws in the approach include:

• It requires preliminary integrations, which someone needs to take responsibility for
• It relies on external repositories & Sidecars
• New Vulnerabilities can be found after deployment, which leads to a continuous struggle to keep the system secure
• The Kubernetes Infrastructure itself may be left vulnerable e.g. – For example there were 2 new Kubernetes vulnerabilities discovered in June, CVE-2020-10749 (container networking plugins), and CVE-2020-8555 (kube-controller-manager), which were only fixed in versions 115/116 or above.
• Constant monitoring of Kubernetes clusters is critical

Pods Deployment

When containers are under attack, the attacker often moves from the breached container to the host to gain exposure to other containers that share the same host. To stay protected, it’s vital to have a comprehensive pod security context in place. It enables users to set up the needed configurations that minimize the blast radius of an attack. The Kubernetes Pod Security Policy is a cluster-level resource that controls security sensitive aspects of the pod specification. The PodSecurityPolicy objects define a set of conditions that pods must run with in order to be accepted, as well as defaults for the related field.

Network Policies

Network policies reduce the impact of potential breaches by limiting exposure. They can be used to:

• Create networking segments in the cluster by restricting communication between applications
• Control the incoming and outgoing traffic – eliminating malicious activities (e.g. crypto-mining) A great way to help network security is to use a service mesh, such as Istio. Istio allows users to create policies based on Kubernetes Services (not pods) and offers integrated options to solve all exceptions. Ingress and Egress policies are simpler with Istio. Istio allows the Kubernetes network policies to be expanded to include non-containerized elements. A newer approach is called for, one that is able to secure containers ”’in runtime”’ without leaving Kubernetes vulnerable to attack, without relying on external repositories, and without relying on pre-deployment measures.

Introducing Cisco Cloud Native Security Solution

To tackle this challenge, we created Cisco Cloud Native Security Solution, an open source runtime vulnerability scanner tool for Kubernetes. The new approach, based on open-source Kubei, tackles security ”’from a runtime perspective”’. The big change to the traditional container security approach, is that instead of relying on CI/CD security, it scans ”’all runtime images”’, regardless of their origin, and without requiring an integration.

• Detects malicious pods and provide an extra layer of security to your cluster
• Scans accessed public images not hosted in your registry
• Gains visibility into Kubernetes system elements
• Runs inside the cluster and provides real-time assessment of deployed containers’ vulnerabilities and cluster’s health

Advanced Risk Protection

Cisco Cloud Native Security Solution’s approach adds a new security check step that’s baked into the CI/CD pipeline and runtime. It assesses the risk potential of a container prior to and during deployment. By reviewing the Role-Based Action Control (RBAC) permissions and examining the roles specified in the deployments/statefulsets. It eliminates risks prior to them occurring. Cisco Cloud Native Security Solution carries out a runtime audit on all APIs invoked towards the API server that have an associated risk. It blocks commands that create risks or violate security best practices and it detects abnormal scenarios. By doing this it adds an extra level of security, keeping your Kubernetes safe and secure.

Try Cisco Cloud Native Security Solution Now

Ready to get started? To use Kubei, run the following command: kubectl apply -f https://raw.githubusercontent.com/Portshift/kubei/master/deploy/kubei.yaml Then, run this command to configure the port for communication: Run the following command to verify that Kubei is up and running: kubectl -n kubei get pod -lapp=kubei That’s it! Simple and easy real time vulnerability scanning!