Government organizations and institutions have similar requirements and goals regarding their IT infrastructure as commercial enterprises: it must be flexible enough to adapt to the changing needs of the organization, easy to maintain and monitor, scalable to meet the changing workload requirements, highly available and resistent to errors, and of course secure to protect the various sensitive data such organizations must process. In addition, they must meet the requirements of various national and state-level regulations, like the Federal Risk and Authorization Management Program (FedRAMP), the Department of Defense (DoD) Cloud Computing Security Requirements Guide (SRG), the Federal Information Security Management Act (FISMA), and other legislation.
As we all know, modern IT infrastructures are increasingly cloud-based, and managing large deployments in the cloud is a challenge in itself. Cloud providers have been trying to assist these efforts by offering cloud solutions that comply with these requirements, for example, AWS GovCloud, Google Cloud for government, or Azure Government Cloud Computing. However, since most of these solutions are lacking in higher-level integrations, it is useful to run higher-level management tools on top of these cloud solutions to automate the provisioning of Kubernetes clusters using a versatile control plane, especially if you need to manage a hybrid cloud, or manage clusters in a multicloud environment.
In addition to government organizations, other entities such as contractors, research organizations, educational institutions, and other U.S. customers that run sensitive workloads in the cloud can also be affected by the above-mentioned regulations.
Banzai Cloud Pipeline is a management platform and control plane that provide lots of components (40+) that are required for these tasks, and makes them work together seamlessly. It provides all the necessary glue code of configuration, resiliency, security, scaling, external integrations, and also provides a rich UI, CLI and API to manage them with ease.
This blog post describes how to run the Banzai Cloud Pipeline platform in the AWS GovCloud US regions. Note that this is a lengthy and somewhat complicated process involving several steps. If you run into problems, feel free to ask for help on our #pipeline-support Slack channel.
The procedure involves the following high-level steps:
The latter is needed because AWS product and price information is only available in us-east-1, even for the GovCloud regions.
Note: This guide expects you to run every command in the same terminal session, because some steps depend on the output of earlier steps. (You can supply those outputs manually in a new terminal, just keep that in mind.)
curl https://getpipeline.sh | sh
For other installation options, see the banzai CLI documentation.
mkdir pipeline-govcloud-poc
cd pipeline-govcloud-poc
Note: the control plane installer of Pipeline Enterprise supports HA EKS and EC2 PKE installation with RDS/Aurora provisioning. For the sake of simplicity this blog post covers using EKS but other Kubernetes distributions (like PKE) are also supported.
To install Pipeline into the AWS GovCloud regions, you need an already existing Kubernetes cluster that you install Pipeline on. Complete the following steps to manually provision an EKS cluster for Pipeline. For more details or customization options of the manual provisioning, see the official AWS documentation.
Make sure that you have your local environment set up for running AWS commands (for example, the default region is properly configured). You will also need an access-secret key pair for accessing the Pipeline cluster.
First, you have to create certain global resources required for provisioning EKS clusters.
Note: This guide uses CloudFormation for creating most of the cloud resources.
aws cloudformation create-stack --stack-name eks-cluster-role --template-body {{< posturl >}}/amazon-eks-cluster-role.yaml --capabilities CAPABILITY_IAM
aws cloudformation wait stack-create-complete --stack-name eks-cluster-role
export CLUSTER_ROLE=$(aws cloudformation describe-stacks --stack-name eks-cluster-role | jq -r '.Stacks[0].Outputs[0].OutputValue')
aws cloudformation create-stack --stack-name eks-node-group-instance-role --template-url https://amazon-eks.s3.us-west-2.amazonaws.com/cloudformation/2020-06-10/amazon-eks-nodegroup-role.yaml --capabilities CAPABILITY_IAM
aws cloudformation wait stack-create-complete --stack-name eks-node-group-instance-role
export WORKER_NODE_ROLE=$(aws cloudformation describe-stacks --stack-name
eks-node-group-instance-role | jq -r '.Stacks[0].Outputs[0].OutputValue')
aws cloudformation create-stack --stack-name pipeline-eks-vpc --template-url https://amazon-eks.s3.us-west-2.amazonaws.com/cloudformation/2020-06-10/amazon-eks-vpc-sample.yaml
aws cloudformation wait stack-create-complete --stack-name pipeline-eks-vpc
export PIPELINE_VPC_SECURITYGROUP=$(aws cloudformation describe-stacks -- stack-name pipeline-eks-vpc | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "SecurityGroups") | .OutputValue')
export PIPELINE_VPC_SUBNETS=$(aws cloudformation describe-stacks --stackname pipeline-eks-vpc | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "SubnetIds") | .OutputValue')
aws eks create-cluster \
--name pipeline-eks \
--kubernetes-version 1.17 \
--role-arn $CLUSTER_ROLE \
--resources-vpc-config subnetIds=$PIPELINE_VPC_SUBNETS,securityGroupIds=$PIPELINE_VPC_SECURITYGROUP
aws eks wait cluster-active --name pipeline-eks
aws eks create-nodegroup \
--cluster-name pipeline-eks \
--nodegroup-name pool0 \
--kubernetes-version 1.17 \
--node-role $WORKER_NODE_ROLE \
--subnets $(echo $PIPELINE_VPC_SUBNETS | sed 's/,/ /g') \
--instance-types c5.large \
--ami-type AL2_x86_64 \
--scaling-config minSize=3,maxSize=5,desiredSize=3
aws eks wait nodegroup-active --cluster-name pipeline-eks --nodegroup-name pool0
Pipeline-managed EKS clusters use self-managed node groups, because they give better flexibility and more control over the nodes.
At this point you should already have the cluster up and running. Before moving on to installing Pipeline itself, you need to configure access to the cluster. This requires some manual editing, as described in the following steps.
export KUBECONFIG=$PWD/kubeconfig aws eks update-kubeconfig --name pipeline-eks
Note: Make sure to get the export right, otherwise the following commands might mess up your
$HOME/.kube/config
file.
kubeconfig
file and find the users
part in it. It should look something like this:
users:
- name: arn:aws-us-gov:eks:us-gov-west-1:ACCOUNTID:cluster/pipeline-eks
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- --region
- us-gov-west-1
- eks
- get-token
- --cluster-name
- pipeline-eks
command: aws
exec
section with the following:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- token
- -i
- pipeline-eks
command: aws-iam-authenticator
env:
- name: AWS_REGION
value: YOUR_REGION (e.g. us-gov-west-1)
- name: AWS_ACCESS_KEY_ID
value: YOUR AWS ACCESS KEY ID
- name: AWS_SECRET_ACCESS_KEY
value: YOUR AWS SECRET ACCESS KEY
env
section with your own values.kubectl get pods --all-namespaces
You should see a bunch of pods, some of them beginning with aws-node-
:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-8tb7l 1/1 Running 0 17m
kube-system aws-node-m2hjq 1/1 Running 0 17m
kube-system aws-node-mmz6x 1/1 Running 0 17m
kube-system coredns-78dbddd469-qbq96 1/1 Running 0 31m
kube-system coredns-78dbddd469-tnfxw 1/1 Running 0 31m
kube-system kube-proxy-dhbwg 1/1 Running 0 17m
kube-system kube-proxy-j22z8 1/1 Running 0 17m
kube-system kube-proxy-rqcvq 1/1 Running 0 17m
Note: This step was required, because the Kubernetes provider of the Pipeline installer expects a self-contained configuration that doesn’t require any additional parameters, but it will still use
aws-iam-authenticator
under the hood for requesting the actual token from AWS.This is the end of the Provisioning EKS cluster for Pipeline section. If everything went well, you can move on to installing Pipeline (it will be much shorter, promise :)).
Now you are ready to install the Banzai Cloud Pipeline platform on your EKS cluster hosted in an AWS GovCloud region.
Note: Make sure your KUBECONFIG environment variable still points to the correct file.
banzai pipeline init --provider=k8s --workspace pipeline-govcloud-p
Technically this step is not necessary, but it gives you an opportunity to check and modify some configuration which is actually necessary in this case.
The workspace in the installer terminology is a directory that contains all configuration and state required to install and upgrade Pipeline.
❯ ll $HOME/.banzai/pipeline/pipeline-govcloud-poc
total 8
drwx------ 4 mark staff 128B Jul 6 00:04 ./
drwx------ 12 mark staff 384B Jul 6 00:04 ../
drwx------ 3 mark staff 96B Jul 6 00:04 .kube/
-rw------- 1 mark staff 269B Jul 6 00:04 values.yaml
❯ cat $HOME/.banzai/pipeline/pipeline-govcloud-poc/values.yaml
defaultStorageBackend: mysql
externalHost: auto
ingressHostPort: false
installer:
image: banzaicloud/pipeline-installer@sha256:815eaaf19da91e5fedd1bc5f76b32c050f33116c0ed440136ec4cd6a2726f7b3
provider: k8s
tlsInsecure: true
uuid: b23002c9-caf2-47b0-97da-28fa705dda92
Note: This directory will hold the state of your Pipeline installation, so make sure it doesn’t get lost. Since it contains credentials, it’s not suitable to store in a git repository. The production version of the Pipeline installer supports remote state storage and secret encryption using KMS.
pipeline:
configuration:
cloud:
amazon:
defaultRegion:
"YOUR GOVCLOUD REGION (eg. us-gov-west-1)"
distribution:
pke:
amazon:
globalRegion:
"YOUR GOVCLOUD REGION (eg. us-gov-west-1)"
cloudinfo:
enabled: true
image:
app:
vault:
enabled: false
providers:
amazon:
enabled: true
region: "YOUR GOVCLOUD REGION (eg. us-gov-west-1)"
accessKey: "YOUR ACCESS KEY ID"
secretKey: "YOUR SECRET ACCESS KEY"
pricing:
accessKey:
"YOUR ACCESS KEY ID with access to us-east-1"
secretKey:
"YOUR SECRET ACCESS KEY with access to
us-east-1"
telescopes:
enabled: true
The latter is required because AWS product and price information is only available in us-east-1, even for the GovCloud regions.
values.yaml
file, you can start the installation process:
banzai pipeline up --workspace pipeline-govcloud-poc
It’s going to take a few minutes, so feel free to grab a cup of coffee 🙂
Once the installer is ready, you should see the following output with an interactive question. Take note of this information as this is how you will be able to access Pipeline. (To display this information later, run the banzai pipeline up
command again.)
Apply complete! Resources: 20 added, 0 changed, 0 destroyed.
...
Outputs:
pipeline-address = https://a7da3d02615692c87ad7d20666653101-219213321.us-gov-west-1.elb.amazonaws.com/
pipeline-password = bgMPtmFFG3ALp5gc
pipeline-username = admin@example.com
INFO[0181] Pipeline is ready at https://a7da3d02615692c87ad7d20666653101-219213321.us-gov-west-1.elb.amazonaws.com/.
? Do you want to login this CLI tool now? (Y/n)
That’s it, now you should be logged in to Pipeline in the CLI tool. You can also log in to the dashboard using the URL in the output. Now that you have installed Pipeline, you can create your first cluster from Pipeline.
The point of provisioning the EKS cluster and installing Pipeline on it was to be able to manage clusters from Pipeline. So in this section, you will create your first PKE cluster on Pipeline. For more details, check the official Pipeline documentation.
```bash
banzai secret create --name=aws --magic --type=amazon
```
You can read more about credentials and access in the [Pipeline documentation](https://banzaicloud.com/docs/pipeline/secrets/providers/pke_aws_auth_credentials/).
Note: Normally we use pre-built PKE images that already contain PKE and the necessary components. When using a plain Ubuntu image, PKE installs everything on the fly. In a production environment, we can provide instructions for building custom images.
wget {{< posturl >}}/cluster.json
banzai cluster create --name first-pke-cluster -f cluster.json
It usually takes about 5-10 minutes to launch a cluster.
banzai cluster shell --cluster-name first-pke-cluster
It gives you a new shell preloaded with the appropriate kube config.
Now that you have a cluster managed by Banzai Cloud Pipeline, you can play around with it, for example, deploy some workload, or enable some of the integrated services on the cluster – either from the command line, or using the Pipeline web interface.
Every once in a while (especially during evaluation) you might want to start over, for example, to move to a different account. This section shows you how to deprovision everything in the correct order, without leaving any leftover resources.
Note: For this you don’t need to remain in the same terminal (as mentioned at the beginning of this guide).
Before destroying the Pipeline instance, make sure that every cluster is deleted (if you don’t want to keep them), because once Pipeline is down, there is no way to get the state back, and you need to delete any remaining clusters manually.
❯ banzai cluster list
Id Name Distribution Location CreatorName CreatedAt Status StatusMessage
1 first-pke-cluster pke us-gov-west-1 admin@example.com 2020-07-06T13:58:54Z RUNNING Cluster is running
banzai cluster delete --cluster-name first-pke-cluster
You can remove Pipeline from the EKS cluster with the following command:
banzai pipeline down --workspace pipeline-govcloud-poc
This will deprovision all Pipeline components from the cluster, including the database, so make sure this is what you want.
You can delete the EKS cluster that hosted Pipeline with a series of AWS commands.
aws eks delete-nodegroup --cluster-name pipeline-eks --nodegroup-name pool0
aws eks wait nodegroup-deleted --cluster-name pipeline-eks --nodegroup-name pool0
Delete the EKS cluster itself:
aws eks delete-cluster --name pipeline-eks
aws eks wait cluster-deleted --name pipeline-eks
aws cloudformation wait stack-delete-complete --stack-name pipeline-eks-vpc
aws cloudformation wait stack-delete-complete --stack-name pipeline-eks-vpc
Note: Sometimes EKS fails to delete load balancers, causing the network deprovisioning to fail. If that happens, delete the load balancers manually and run the stack delete command again.
aws cloudformation delete-stack --stack-name eks-node-group-instance-role
aws cloudformation delete-stack --stack-name eks-cluster-role
pke-global
stack that Pipeline creates the first time you launch a PKE cluster in an account:
aws cloudformation delete-stack --stack-name pke-global
Pipeline will detect if it’s missing and will recreate it if necessary.
This blog post has demonstrated how to install and run the Banzai Cloud Pipeline container management platform in a restricted AWS GovCloud region, often used by government organizations, contractors, research organizations, educational institutions, and other U.S. customers that run sensitive workloads in the cloud. Using Pipeline gives your organization the speed of delivering clusters and the extended automated solutions, the integrated services, and flexibility provided by the platform.
If you are interested in testing Banzai Cloud Pipeline in AWS GovCloud and you need help, or your organization uses the government solution of a different cloud provider, contact us.
Banzai Cloud Pipeline Kubernetes Engine (PKE) is a simple, secure and powerful CNCF-certified Kubernetes distribution, the preferred Kubernetes run-time of the Pipeline platform. It was designed to work on any cloud, VM or on bare metal nodes to provide a scalable and secure foundation for private clouds. PKE is cloud-aware and includes an ever-increasing number of cloud and platform integrations.
Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform..