The Kubernetes backend

Upcoming release

You are reading the reference manual of an upcoming release. Read the current manual »

The Kubernetes backend runs and is tested against all currently actively maintained Kubernetes versions. For ContainerSSH version 0.4.1 these are: 1.21, 1.20, and 1.19.

Tip

This is the documentation for the Kubernetes backend. For deploying ContainerSSH inside Kubernetes please see the installation guide.

The base configuration structure¶

In order to use the Kubernetes backend you must specify the following configuration entries via the configuration file or the configuration server:

backend: kubernetes
kubernetes:
  connection:
    <connection configuration here>
  pod:
    <pod configuration here>
  timeouts:
    <timeouts configuration here>

Configuring connection parameters¶

In order to use Kubernetes you must provide the credentials to authenticate with the Kubernetes cluster. There are several supported authentication methods:

Username and password (HTTP basic auth).
x509 client certificates.
Bearer token.

These options should be specified like this:

kubernetes:
  connection:
    host: <...>
    <...>

Tip

See the Securing Kubernetes section below for a detailed walkthrough for provisioning limited service accounts for ContainerSSH.

Base configuration¶

Name	Type	Description
`host`	`string`	The hostname or ip + the port of the Kubernetes API server. Set this to `kubernetes.default.svc` to run inside a Kubernetes cluster, otherwise set it to the host name of your Kubernetes API.
`path`	`string`	This is the API path of the Kubernetes API. Defaults to `/api` and you will typically not need to change this.
`cacertFile`	`string`	Points to the file that contains the CA certificate in PEM format that signed the server certificate.
`cacert`	`string`	Directly contains the CA certificate in PEM format that signed the server certificate. Set to `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt` when running with a service account.
`serverName`	`string`	Sets the hostname of the server that should be sent to the Kuberentes API in the TLS SNI. This is useful when the Kubernetes API has a hostname that is not resolvable from the server ContainerSSH is running on.
`qps`	`float32`	Indicates a maximum queries per second from this client.
`burst`	`int`	Indicates the maximum burst for query throttling.

HTTP basic authentication (username and password)¶

Name	Type	Description
`username`	`string`	Username for authenticating against the Kubernetes API. This is only used for HTTP basic auth and does not work with other authentication methods (e.g. OAuth2)
`password`	`string`	Password for authenticating against the Kubernetes API. This is only used for HTTP basic auth and does not work with other authentication methods (e.g. OAuth2)

x509 certificate authentication¶

Name	Type	Description
`certFile`	`string`	Points to a file that contains the client certificate for x509 authentication against the Kubernetes API in PEM format.
`cert`	`string`	Directly contains the certificate for x509 authentication against the Kubernetes API in PEM format.
`keyFile`	`string`	Points to a file that contains the client key for x509 authentication against the Kubernetes API in PEM format.
`key`	`string`	Directly contains the client key for x509 authentication against the Kubernetes API in PEM format.

Bearer token authentication¶

This authentication method is primarily used with service accounts.

Name	Type	Description
`bearerTokenFile`	`string`	Points to the file that contains the bearer token for authenticating against the Kubernetes API. Set to `/var/run/secrets/kubernetes.io/serviceaccount/token` to use the service account when running ContainerSSH inside a Kubernetes cluster.
`bearerToken`	`string`	Directly contains the bearer token for authenticating against the Kubernetes API.

Pod configuration¶

The pod configuration contains the information which pod to run. The structure is very similar to the Pod object in Kubernetes, and we add a few extra options:

kubernetes:
  pod:
    metadata:
      <metadata configuration here>
    spec:
      <pod spec here>
    <ContainerSSH-specific options here>

Note

Do not include the apiVersion, kind, or status types from the Kubernetes structure.

Tip

Did you know? You can get a full description of the Pod type by running kubectl explain pod, kubectl explain pod.spec, and kubectl explain pod.metadata.

Basic pod configuration¶

ContainerSSH defaults to running pods in the default namespace with the containerssh/containerssh-guest-image container image. You can change these settings with the following options:

kubernetes:
  pod:
    metadata:
      namespace: default
    spec:
      containers:
        - name: shell
          image: containerssh/containerssh-guest-image
          env:
           - name: VAR
             value: Hello world!

Running multiple containers¶

When running multiple containers ContainerSSH defaults to attaching to the first container. You can change this behavior by specifying the consoleContainerNumber option. This number is 0-indexed.

kubernetes:
  pod:
    consoleContainerNumber: 1
    spec:
      containers:
        - name: container1
          image: ...
        - name: container2
          image: ...

Mounting volumes¶

In Kubernetes volumes of various types can be mounted into pods. This is done as follows:

kubernetes:
  pod:
    consoleContainerNumber: 1
    spec:
      volumes:
        - name: <volume name here>
          <mount type here>:
            <mount options here>
      containers:
        - name: shell
          image: <image name here>
          volumeMounts:
            - name: <volume name here>
              mountPath: <where to mount>

For example, mounting a path from the host machine can be done as follows:

kubernetes:
  pod:
    consoleContainerNumber: 1
    spec:
      volumes:
        - name: home
          hostPath:
            path: /home/ubuntu
            type: Directory
      containers:
        - name: shell
          image: containerssh/containerssh-guest-image
          volumeMounts:
            - name: home
              mountPath: /home/ubuntu

Tip

Use kubectl explain pod.spec.volumes for details on how to configure the volume driver for your storage.

Forcing the pod to run on a specific node¶

In Kubernetes pod scheduling can be influenced either by node affinity or by explicitly binding a pod to a node.

Node affinity lets you schedule pods based on various features of the node, e.g. the presence of a GPU, a disk type, etc. As the configuration can be quite complex we will not discuss it here, please read the Kubernetes manual.

Binding a pod to a specific node on the other hand is rather simple:

kubernetes:
  pod:
    spec:
      nodeName: <insert node name here>

Other options¶

Apart from the metadata and spec options ContainerSSH has the following options on a Pod-level. These should not be changed unless required.

Name	Type	Description
`consoleContainerNumber`	`uint`	Specifies the number of the container to attach to. Defaults to the first container.
`mode`	`string`	Specify `connection` to launch one pod per SSH connection or `session` to run one pod per SSH session (multiple pods per connection). In connection mode the container is started with the `idleCommand` as the first program and every session is launched similar to how `kubectl exec` runs programs. In session mode the command is launched directly.
`idleCommand`	`[]string`	Specifies the command to run as the first process in the container in `connection` mode. Parameters must be provided as separate items in the array. Has no effect in `session` mode.
`shellCommand`	`[]string`	Specifies the command to run as a shell in `connection` mode. Parameters must be provided as separate items in the array. Has no effect in `session` mode.
`agentPath`	`string`	Contains the full path to the ContainerSSH guest agent inside the shell container. The agent must be installed in the guest image.
`disableAgent`	`bool`	Disable the ContainerSSH guest agent. This will disable several functions and is not recommended.
`subsystems`	`map[string]string`	Specifies a map of subsystem names to executables. It is recommended to set at least the `sftp` subsystem as many users will want to use it.

Configuration restrictions¶

In connection mode the idleCommand and shellCommand options are required.
In session mode the restart policy must be empty or Never.

Configuring timeouts¶

The timeouts section has the following options. All options can use time units (e.g. 60s) and default to nanoseconds without time units.

Name	Description
`podStart`	The time to wait for the pod to start.
`podStop`	The time to wait for the pod to stop.
`commandStart`	The time to wait for the command to start in `connection` mode.
`signal`	The time to wait to deliver a signal to a process.
`window`	The time to wait to deliver a window size change.
`http`	The time to wait for the underlying HTTP calls to complete.

Securing Kubernetes¶

Securing the Kubernetes installation is beyond the scope of this document. We will describe how to deploy and configure ContainerSSH for security in a Kubernetes environment

Creating a service account¶

When deploying ContainerSSH with a Kubernetes backend you should never an admin account for interacting with a Kubernetes cluster. ContainerSSH can run inside the same Kubernetes cluster or it can run as a standalone. When deploying inside the same Kubernetes cluster it is strongly recommended that ContainerSSH runs in a different namespace as the guest pods ContainerSSH launches.

The setup below assumes you are creating a service account in the default namespace and the ContainerSSH pods will run in the containerssh-guests namespace

First, we need to create the service account. The following fragment can be applied with kubectl apply -f:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: containerssh
automountServiceAccountToken: false

Then we create the role and rolebinding resources in the containerssh-guests namespace to allow the service accounts to create pods:

kubectl create role containerssh \
  -n containerssh-guests \
  --verb="*" \
  --resource=pods \
  --resource=pods/logs \
  --resource=pods/exec
kubectl create rolebinding containerssh \
  -n containerssh-guests \
  --serviceaccount=containerssh:containerssh

Let's test if the permissions are correct:

$ kubectl auth can-i create pod --as containerssh
no
$ kubectl auth can-i create pod --namespace containerssh-guests --as containerssh
yes

Docker Desktop

Docker Desktop Kubernetes contains a cluster role binding called docker-for-desktop-binding that allows all service accounts to perform every action. To secure your Docker Desktop installation you will need to delete this CRB.

Deploying inside of Kubernetes¶

When deploying ContainerSSH inside the same Kubernetes cluster you can simply use the service account when making your deployment:

kubernetes:
  connection:
    host: ...
    cacertFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

Deploying outside of Kubernetes¶

Now, if you are running ContainerSSH outside of the Kubernetes cluster you can fetch the secret belonging to the service account by first looking at the service account itself:

kubectl describe serviceaccount containerssh

This command will output the name of the secret for this service account, which can then be extracted:

kubectl get secret containerssh-token-2jrnc -o yaml

The output will look as follows:

apiVersion: v1
data:
  ca.crt: <base64-encoded CA certificate here>
  namespace: ZGVmYXVsdA==
  token: <base64-encoded bearer token here>
kind: Secret

Base64-decode both the ca.crt and the token fields and insert them into your ContainerSSH config as follows:

kubernetes:
  connection:
    bearerToken: <insert token here>
    cacert: |
      <insert ca.crt here>

Preventing root escalation¶

Under normal circumstances a user running as root inside a container cannot access resources outside the container. However, in the event of a container escape vulnerability in Kubernetes it is prudent not to run container workloads as root. You can prevent forcibly prevent any container from running as root by configuring the following setting:

kubernetes:
  pod:
    spec:
      securityContext:
        runAsNonRoot: true

However, this will fail starting any container image that wants to run as root. In addition to the option above, you can also force the container to a specific UID:

kubernetes:
  pod:
    spec:
      securityContext:
        runAsUser: 1000

Preventing storage exhaustion¶

A malicious user could cause the Kubernetes host to run out of disk space with a simple attack:

cat /dev/zero > ~/zerofill

There are two cases here:

If the directory the user can write to is mounted using a volume the attacker can fill up the storage that is mounted. You can pass per-user mounts from the configuration server that mount volumes that are unique to the connecting user. This way the user can only fill up their own disk. The specific implementation depends on your volume driver.

If the directory the user can write to is not mounted the user can fill up the container image. This is a much more subtle way of filling up the disk. Current Kubernetes does not support preventing this kind of attack, so it is recommended to only allow users to write to paths mounted as volumes. The readOnlyRootFilesystem PodSecurityPolicy can be applied to the namespace or the whole cluster preventing writes to the container root filesystem filling up the disk.

Preventing memory exhaustion¶

Users can also try to exhaust the available memory to potentially crash the server. This can be prevented using the following configuration:

kubernetes:
  pod:
    spec:
      resources:
        limits:
          memory: "128Mi"

You can read more about memory requests and limits in the Kubernetes documentation.

Preventing CPU exhaustion¶

A malicious user can also exhaust the CPU by running CPU-heavy workloads. This can be prevented by setting the following options:

kubernetes:
  pod:
    spec:
      resources:
        limits:
          cpu: "500m"

In this setting 1000m corresponds to a full core or vCPU of the host. You can read more about memory requests and limits in the Kubernetes documentation.

Limiting network access¶

Depending on which Container Network Interface is installed on the Kubernetes cluster you may have access to Network Policies. These work very similar to how traditional Linux firewalling works. The following network policy disables all network access within a namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: containerssh-guest-policy
  namespace: containerssh-guests
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress: []
  egress: []