$ kubectl proxy
Starting to serve on 127.0.0.1:8001

The purpose behind proxy WebSocket

Architectural overview

• CLI Proxy A small go application that runs on the local computer. It's a simple proxy to the Kubernetes API Server's endpoint. As the UI is javascript running inside a browser, in this case it's running from the same host as the CLI.
• Kubernetes Api Server The entry point to the Kubernetes cluster.
• Ingress Ingress server, which proxies the requests to a particular backend-component. We use Nginx-ingress at the moment.
• Loki The Loki backend, which collects all the logs in the cluster and provides HTTP & GRPC API.

About LogQL:

• One is to call the /loki/api/v1/query endpoint over HTTP with query args, which returns results in a simple HTTP response
• The other is to use the /loki/api/v1/tail endpoint where the query is similar (relatively speaking) to the first one, but it upgrades the HTTP connection to a WebSocket and continuously feeds this live connection with the most recent data (e.g. POSIX tail)

query={pod="one-eye-event-tailer-0"}&limit=10

If you are not familiar with LogQL check out our post Loki, the tail and grep for Kubernetes

Crack some eggs

$ curl -i localhost:8080/loki/ready
HTTP/1.1 200 OK
Content-Length: 6
Content-Type: text/plain; charset=utf-8
Date: Tue, 30 Jun 2020 09:02:05 GMT
Server: nginx/1.17.8
X-Content-Type-Options: nosniff

ready

curl localhost:8080/loki/loki/api/v1/label | jq

Please note the double /loki URL:

• The first /loki URL belongs to the ingress, which makes the ingress route the request to the loki backend
• The second /loki URL is actually part of the Loki service HTTP API

{
	"status": "success",
	"data": [
		"__name__",
		"app",
		"app_kubernetes_io_component",
		"app_kubernetes_io_instance",
		"app_kubernetes_io_name",
		"chart",
		"container",
		"container_id",
		"controller_revision_hash",
		"heritage",
		"host",
		"name",
		"namespace",
		"pod",
		"pod_id",
		"pod_template_hash",
		"prometheus",
		"release",
		"statefulset_kubernetes_io_pod_name"
	]
}

curl 'localhost:8080/loki/loki/api/v1/query_range?query=\{name="loki"\}&limit=1' 2>/dev/null | jq

Notice the escaped curly brackets (\{ \}). We need these because bash tries to evaluate them otherwise

{
  "status": "success",
  "data": {
    "resultType": "streams",
    "result": [
      {
        "stream": {
          "namespace": "default",
          "pod": "one-eye-loki-0",
          "pod_id": "78a4c513-06b9-449d-aca6-10a1266fc116",
          "statefulset_kubernetes_io_pod_name": "one-eye-loki-0",
          "app": "loki",
          "container": "loki",
          "container_id": "c7a9c62eb8e5f5629a04aecf54c39156b4903ecbc5e93ac729de335dbfcf6dd8",
          "controller_revision_hash": "one-eye-loki-7874db668c",
          "host": "kind-control-plane",
          "name": "loki",
          "release": "one-eye-loki"
        },
        "values": [
          [
            "1593529265837448400",
            "{\"stream\":\"stderr\",\"logtag\":\"F\",\"message\":\"level=info ts=2020-06-30T15:01:05.8367595Z caller=table_manager.go:292 msg=\\\"synching tables\\\" expected_tables=116\"}"
          ]
        ]
      }
    ],
    "stats": {
      "summary": {
        "bytesProcessedPerSecond": 14001628,
        "linesProcessedPerSecond": 43852,
        "totalBytesProcessed": 53321,
        "totalLinesProcessed": 167,
        "execTime": 0.0038082
      },
      "store": { ... }
    }
  }
}

Websocket tools

curl

curlws() {
    setopt local_options BASH_REMATCH;
    local uri="${1}"
    [[ "${uri}" =~ ^((http|https)[:]//)?([^/]+)(.*)$ ]] || return
    local protocol=${BASH_REMATCH[2]:-http://}
    curl -N \
    -H "Connection: Upgrade" \
    -H "Upgrade: websocket" \
    -H "Host: ${BASH_REMATCH[4]}" \
    -H "Origin: $protocol${BASH_REMATCH[4]}" \
    -H "Sec-WebSocket-Key: $(head -c 16 /dev/urandom | base64)" \
    -H "Sec-WebSocket-Version: 13" \
    $@ --output - ;
}

The setopt local_options BASH_REMATCH; required on zsh, to use bash regex format

javascript

let socket = new WebSocket( "ws://localhost:8080/loki/loki/api/v1/tail?query={name=`loki}`" ); socket.onopen = socket.onclose = socket.onmessage = socket.onerror = console.log;

The issue

curlws 'http://localhost:8080/loki/loki/api/v1/tail?query=\{name="loki"\}'

�-�parse error : syntax error: unexpected $end

Discovering the problem

curlws 'http://localhost:8080/loki/loki/api/v1/tail?query=\{name="loki"\}' -i

HTTP/1.1 101 Switching Protocols
Server: nginx/1.17.8
Date: Tue, 30 Jun 2020 08:16:35 GMT
Connection: upgrade
Sec-Websocket-Accept: CTk59tUEO+X3dzYN3yElMnoK4Cc=
Upgrade: websocket

�-�parse error : syntax error: unexpected $end%

curlws 'http://localhost:8080/loki/loki/api/v1/tail?query=\{name="loki"\}' 2>/dev/null | hexdump -C

00000000  88 2d 03 f3 70 61 72 73  65 20 65 72 72 6f 72 20  |.-..parse error |
00000010  3a 20 73 79 6e 74 61 78  20 65 72 72 6f 72 3a 20  |: syntax error: |
00000020  75 6e 65 78 70 65 63 74  65 64 20 24 65 6e 64     |unexpected $end|
0000002f

Checking Loki endpoint directly

kubectl port-forward svc/one-eye-loki 3100:3100

Loki service port 3100 forwarded to localhost 3100

curlws 'http://localhost:3100/loki/api/v1/tail?query=\{name="loki"\}&limit=10' -i

Note the singlelokiURL; our request goes directly to the Loki service, skipping the ingress.

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=

�~�{"streams":[{"stream":{"app":"loki","container":"loki","container_id":"809458c447120553ec55bf7159fa1b5eab71e1d541077633d91f5511f48b14d5","controller_revision_hash":"one-eye-loki-7874db668c","host":"k3d-k3s-default-server","name":"loki","namespace":"default","pod":"one-eye-loki-0","pod_id":"89f61a2a-5baa-44bd-be2e-d6e4a7b79156","release":"one-eye-loki","statefulset_kubernetes_io_pod_name":"one-eye-loki-0"},"values":[["1593507930108519100","{\"stream\":\"stderr\",\"logtag\":\"F\",\"message\":\"level=info ts=2020-06-30T09:05:30.1080634Z caller=table_manager.go:292 msg=\\\"synching tables\\\" expected_tables=116\"}"]]},{"stream":{"app":"loki","container":"loki","container_id":"809458c447120553ec55bf7159fa1b5eab71e1d541077633d91f5511f48b14d5","controller_revision_hash":"one-eye-loki-7874db668c","host":"k3d-k3s-default-server","name":"loki","namespace":"default","pod":"one-eye-loki-0","pod_id":"89f61a2a-5baa-44bd-be2e-d6e4a7b79156","release":"one-eye-loki","statefulset_kubernetes_io_pod_name":"one-eye-loki-0"},"values":[["1593508049970165600","{\"stream\":\"stderr\",\"logtag\":\"F\",\"message\":\"level=info ts=2020-06-30T09:07:29.9696504Z caller=table_manager.go:292 msg=\\\"synching tables\\\" expected_tables=116\"}"]]}]}
���^C

Checking Loki through ingress service

kubectl logs -l app=nginx-ingress,app.kubernetes.io/component=controller -f

kubectl port-forward svc/one-eye-nginx-ingress-controller 8000:80

Ingress service port 80 forwarded to localhost 8000

curlws 'http://localhost:8000/loki/loki/api/v1/tail?query=\{name="loki"\}&limit=2' -i

HTTP/1.1 101 Switching Protocols
Server: nginx/1.17.8
Date: Tue, 30 Jun 2020 09:13:01 GMT
Connection: upgrade
Sec-Websocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
Upgrade: websocket

�~d{"streams":[{"stream":{"app":"loki","container":"loki","container_id":"809458c447120553ec55bf7159fa1b5eab71e1d541077633d91f5511f48b14d5","controller_revision_hash":"one-eye-loki-7874db668c","host":"k3d-k3s-default-server","name":"loki","namespace":"default","pod":"one-eye-loki-0","pod_id":"89f61a2a-5baa-44bd-be2e-d6e4a7b79156","release":"one-eye-loki","statefulset_kubernetes_io_pod_name":"one-eye-loki-0"},"values":[["1593508211818521600","{\"stream\":\"stderr\",\"logtag\":\"F\",\"message\":\"level=error ts=2020-06-30T09:10:11.8182047Z caller=tailer.go:97 org_id=fake traceID=7939cf98b55a337f msg=\\\"Error writing to tail client\\\" err=\\\"rpc error: code = Internal desc = transport: transport: the stream is done or WriteHeader was already called\\\"\"}"]]},{"stream":{"app":"loki","container":"loki","container_id":"809458c447120553ec55bf7159fa1b5eab71e1d541077633d91f5511f48b14d5","controller_revision_hash":"one-eye-loki-7874db668c","host":"k3d-k3s-default-server","name":"loki","namespace":"default","pod":"one-eye-loki-0","pod_id":"89f61a2a-5baa-44bd-be2e-d6e4a7b79156","release":"one-eye-loki","statefulset_kubernetes_io_pod_name":"one-eye-loki-0"},"values":[["1593508289700192000","{\"stream\":\"stderr\",\"logtag\":\"F\",\"message\":\"level=info ts=2020-06-30T09:11:29.6975065Z caller=table_manager.go:292 msg=\\\"synching tables\\\" expected_tables=116\"}"]]}]}
��^C%

127.0.0.1 - - [30/Jun/2020:09:27:49 +0000] "GET /loki/loki/api/v1/tail?query={name=\x22loki\x22}&limit=2 HTTP/1.1" 101 1245 "-" "curl/7.64.1" 267 3.778 [default-one-eye-ui-80] [] 10.42.0.20:80 0 3.777 101 478b2dfd61a5a38bd6ef2e0d57226ad3

Requesting Loki through the Kubernetes API server

kubectl proxy --port 8001

Running application-level gateway between localhost:8001 and the Kubernetes API Server.

curlws 'localhost:8001/api/v1/namespaces/default/services/one-eye-nginx-ingress-controller:80/proxy/loki/loki/api/v1/tail?query=\{name="loki"\}' -i

HTTP/1.1 101 Switching Protocols
Server: nginx/1.17.8
Date: Tue, 30 Jun 2020 09:23:29 GMT
Connection: upgrade
Sec-Websocket-Accept: wrWIcq4LOt2fDoU9xsJxH0zBMx0=
Upgrade: websocket

�-�parse error : syntax error: unexpected $end

10.42.0.1 - - [30/Jun/2020:09:28:24 +0000] "GET /loki/loki/api/v1/tail HTTP/1.1" 101 47 "-" "curl/7.64.1" 283 0.006 [default-one-eye-ui-80] [] 10.42.0.20:80 0 0.006 101 aa853734e432cd429a242169045c1881

Verifying the error

curlws 'http://localhost:3100/loki/api/v1/tail' -i

Please note the single loki URL, since our request is skipping the ingress.

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: bFKsYQ4fN7yQPofaObI65dx1mek=

�-�parse error : syntax error: unexpected $end

Possible solutions overview

• Kubectl proxy would strip out the parameter for WebSocket request
• service proxy for WebSocket is missing forwarded headers and query params
• API proxy strips URL query params when proxying WebSocket connections

Update: a solution to this issue has since been submitted, but is yet to be approved: Fixed bug where query string was lost on connection upgrade without using request location

• transfering the query args in request headers
• building a new URL with query args (somehow) placed in the URL path, and restoring them when they were no longer needed (which we named composition/decomposition)

• CLI Proxy composites the URL
• One-Eye-Backend decomposites the URL
• We also had some issues about Loki base path handling (shout out to the Loki devs for the quick response in regards to that), therefore we did some proxy path rule rewriting at the Ingress level. With the chosen solution (proxying the Loki requests and responses through the One Eye backend) we could redesign the rewrites and simply place them in the backend.
• Any element of the proxy-chain was able to proxy standard LogQL queries this way (except when the Kubernetes API server called directly)

An in-depth exploration of the chosen solution

Compositing query args on the CLI Proxy

import ("encoding/base64"
        "net/url")
...
const QueryArgsURLPart string = "queryargs"
...
func paramsToPath(uri *url.URL) *url.URL {
   // cloning url
   result, err := url.Parse(uri.String())
   if err != nil {
      return uri
   }
   if len(result.RawQuery) > 0 {
        // composite all the query args into the path
        result.Path += fmt.Sprintf("/%s/%s/", QueryArgsURLPart, base64.URLEncoding.EncodeToString([]byte(result.RawQuery)))
        // clear query args to avoid future problems
      result.RawQuery = ""
   }
   return result
}

...
    // only websocket connections
    if req.Header.Get("Connection") == "Upgrade" &&
        req.Header.Get("Upgrade") == "websocket" &&
        // only with query args presented
      len(req.URL.RawQuery) > 0 {
      req.URL = paramsToPath(req.URL)
    }
...

curlws 'localhost:8080/loki/loki/api/v1/tail?query=\{name="loki"\}&limit=2'

10.42.0.1 - - [30/Jun/2020:12:58:17 +0000] "GET /loki/loki/api/v1/tail/queryargs/cXVlcnk9e25hbWU9Imxva2kifSZsaW1pdD0y/ HTTP/1.1" 101 1242 "-" "curl/7.64.1" 350 1.923 [default-one-eye-ui-80] [] 10.42.0.20:80 0 1.923 101 e4425bd0a2c4462ca1746e6757e5cc86

echo "cXVlcnk9e25hbWU9Imxva2kifSZsaW1pdD0y" | base64 -d

query={name="loki"}&limit=2

Decompositing query args on One-Eye-Backend

const QueryArgsURLPart string = "queryargs"

var (
   QueryArgsRegxPattern string = fmt.Sprintf("/%s/(%s)/", QueryArgsURLPart, "[^/]*")
   QueryArgsRegx               = regexp.MustCompile(QueryArgsRegxPattern)
)

func PathToParams(uri *url.URL) *url.URL {
   if QueryArgsRegx.MatchString(uri.String()) {
      // cloning url
      result, err := url.Parse(uri.String())
      if err != nil {
         return uri
      }

        // checking the existence of composited query args
      subResults := QueryArgsRegx.FindStringSubmatch(result.String())
      if len(subResults) != 2 {
         return uri
        }

        // decomposites query args
      decodedArgs, err := base64.URLEncoding.DecodeString(subResults[1])
      if err != nil {
         return uri
      }

        // restores the original query args
      result.RawQuery = string(decodedArgs)

        // removing the composited query args from the path
      result.Path = QueryArgsRegx.ReplaceAllLiteralString(result.Path, "")

      return result
   }

   return uri
}

Summary