Open Policy Agent (OPA)

Motivation

In Kubernetes, Gloo stores its configuration as Custom Resource Definitions (CRDs). You can use normal Kubernetes Role Based Access Control (RBAC) to create a policy that grants users the ability to create a Gloo VirtualService. RBAC only allows to grant permissions entire objects. With the Open Policy Agent, one can specify very fine grain control over Gloo objects. For example, with RBAC you can say, “user john@example.com is allowed to create virtual service” With OPA, in addition to specifying access, you can say “virtual services must point to the domain example.com”.

You can of-course combine both, as you see fit.

In this document we will show a simple OPA policy that dictates that all virtual services must not have a prefix re-write.

Prereqs

Setup

First, setup OPA as a validating web hook. In this mode, OPA validates the Kubernetes objects before they are visible to the controllers that act on them (Gloo in our case).

You can use the setup.sh script for that purpose. Note this script follows the docs outlined in official OPA docs with some small adaptations for the Gloo API.

For your convenience, here’s the content of setup.sh (click to reveal):

setup.sh

#!/bin/bash

kubectl create namespace opa
# create Roles as they may need a second to propagate
kubectl --namespace=opa apply -f - <<EOF
# Grant OPA/kube-mgmt read-only access to resources. This lets kube-mgmt
# replicate resources into OPA so they can be used in policies.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: opa-viewer
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts:opa
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gloo-view
rules:
- apiGroups:
  - gateway.solo.io
  resources:
  - virtualservices
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: opa-gloo-viewer
roleRef:
  kind: ClusterRole
  name: gloo-view
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts:opa
  apiGroup: rbac.authorization.k8s.io
---
# Define role for OPA/kube-mgmt to update configmaps with policy status.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: opa
  name: configmap-modifier
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["update", "patch"]
---
# Grant OPA/kube-mgmt role defined above.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: opa
  name: opa-configmap-modifier
roleRef:
  kind: Role
  name: configmap-modifier
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts:opa
  apiGroup: rbac.authorization.k8s.io
EOF

openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"

cat >server.conf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOF

openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=opa.opa.svc" -config server.conf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf

kubectl --namespace=opa create secret tls opa-server --cert=server.crt --key=server.key

kubectl --namespace=opa apply -f - <<EOF
kind: Service
apiVersion: v1
metadata:
  name: opa
  namespace: opa
spec:
  selector:
    app: opa
  ports:
  - name: https
    protocol: TCP
    port: 443
    targetPort: 443
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: opa
  namespace: opa
  name: opa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
      name: opa
    spec:
      containers:
        # WARNING: OPA is NOT running with an authorization policy configured. This
        # means that clients can read and write policies in OPA. If you are
        # deploying OPA in an insecure environment, be sure to configure
        # authentication and authorization on the daemon. See the Security page for
        # details: https://www.openpolicyagent.org/docs/security.html.
        - name: opa
          image: openpolicyagent/opa:0.13.2
          args:
            - "run"
            - "--server"
            - "--tls-cert-file=/certs/tls.crt"
            - "--tls-private-key-file=/certs/tls.key"
            - "--addr=0.0.0.0:443"
            - "--addr=http://127.0.0.1:8181"
          volumeMounts:
            - readOnly: true
              mountPath: /certs
              name: opa-server
          readinessProbe:
            httpGet:
              path: /health
              scheme: HTTPS
              port: 443
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              scheme: HTTPS
              port: 443
            initialDelaySeconds: 3
            periodSeconds: 5
        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:0.8
          args:
            - "--replicate-cluster=v1/namespaces"
            - "--replicate=gateway.solo.io/v1/virtualservices"
      volumes:
        - name: opa-server
          secret:
            secretName: opa-server
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: opa-default-system-main
  namespace: opa
data:
  main: |
    package system

    import data.kubernetes.admission

    main = {
      "apiVersion": "admission.k8s.io/v1beta1",
      "kind": "AdmissionReview",
      "response": response,
    }

    default response = {"allowed": true}

    response = {
        "allowed": false,
        "status": {
            "reason": reason,
        },
    } {
        reason = concat(", ", admission.deny)
        reason != ""
    }
EOF

kubectl label ns kube-system openpolicyagent.org/webhook=ignore
kubectl label ns opa openpolicyagent.org/webhook=ignore

kubectl apply -f - <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1beta1
metadata:
  name: opa-validating-webhook
webhooks:
  - name: validating-webhook.openpolicyagent.org
    namespaceSelector:
      matchExpressions:
      - key: openpolicyagent.org/webhook
        operator: NotIn
        values:
        - ignore
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["gateway.solo.io"]
        apiVersions: ["v1"]
        resources: ["virtualservices"]
    clientConfig:
      caBundle: $(cat ca.crt | base64 | tr -d '\n')
      service:
        namespace: opa
        name: opa
EOF

Policy

OPA Policies are written in Rego. A language specifically designed for policy decisions.

Let’s apply this policy, forbidding virtual service with prefix re-write:

package kubernetes.admission

operations = {"CREATE", "UPDATE"}

deny[msg] {
	input.request.kind.kind == "VirtualService"
	operations[input.request.operation]
	input.request.object.spec.virtualHost.routes[_].routePlugins.prefixRewrite
	msg := "prefix re-write not allowed"
}

Let’s break this down:

operations = {"CREATE", "UPDATE"}

This policy only applies to objects that are created or updated.

deny[msg] {

Start a policy to deny to object creation \ update, if all conditions in the braces hold.

The conditions are:

	input.request.kind.kind == "VirtualService"

(1) This object is a VirtualService

	operations[input.request.operation]

(2) This object is created or updated.

	input.request.object.spec.virtualHost.routes[_].routePlugins.prefixRewrite

(3) This object has a prefixRewrite stanza.

If all these conditions are true, the object will be denied with this message:

	msg := "prefix re-write not allowed"

Apply Policy

You can use this command to apply the policy, by writing it to a configmap in the opa namespace

kubectl --namespace=opa create configmap vs-no-prefix-rewrite --from-file=vs-no-prefix-rewrite.rego

Give it a second, and you will see the policy status changes to ok:

kubectl get configmaps -n opa vs-no-prefix-rewrite -o yaml
apiVersion: v1
data:
  vs-no-prefix-rewrite.rego: "package kubernetes.admission\n\noperations = {\"CREATE\",
    \"UPDATE\"}\n\ndeny[msg] {\n\tinput.request.kind.kind == \"VirtualService\"\n\toperations[input.request.operation]\n\tinput.request.object.spec.virtualHost.routes[_].routePlugins.prefixRewrite\n\tmsg
    := \"prefix re-write not allowed\"\n}\n"
kind: ConfigMap
metadata:
  annotations:
    openpolicyagent.org/policy-status: '{"status":"ok"}'
  creationTimestamp: "2019-08-20T11:10:55Z"
  name: vs-no-prefix-rewrite
  namespace: opa
  resourceVersion: "39558874"
  selfLink: /api/v1/namespaces/opa/configmaps/vs-no-prefix-rewrite
  uid: 2de8732f-c33b-11e9-8be1-42010a8000dc

Verify

Time to test! we have prepared two virtual services for testing:

vs-ok.yaml

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: default
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matcher:
        exact: /sample-route-1
      routeAction:
        single:
          upstream:
            name: default-petstore-8080
            namespace: gloo-system

vs-err.yaml

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: default
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matcher:
        exact: /sample-route-1
      routeAction:
        single:
          upstream:
            name: default-petstore-8080
            namespace: gloo-system
      routePlugins:
        prefixRewrite:
          prefixRewrite: /api/pets

Try it:

kubectl apply -f vs-ok.yaml
virtualservice.gateway.solo.io/default created
kubectl apply -f vs-err.yaml
Error from server (prefix re-write not allowed): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"gateway.solo.io/v1\",\"kind\":\"VirtualService\",\"metadata\":{\"annotations\":{},\"name\":\"default\",\"namespace\":\"gloo-system\"},\"spec\":{\"virtualHost\":{\"domains\":[\"*\"],\"name\":\"gloo-system.default\",\"routes\":[{\"matcher\":{\"exact\":\"/sample-route-1\"},\"routeAction\":{\"single\":{\"upstream\":{\"name\":\"default-petstore-8080\",\"namespace\":\"gloo-system\"}}},\"routePlugins\":{\"prefixRewrite\":{\"prefixRewrite\":\"/api/pets\"}}}]}}}\n"}},"spec":{"virtualHost":{"routes":[{"matcher":{"exact":"/sample-route-1"},"routeAction":{"single":{"upstream":{"name":"default-petstore-8080","namespace":"gloo-system"}}},"routePlugins":{"prefixRewrite":{"prefixRewrite":"/api/pets"}}}]}}}
to:
Resource: "gateway.solo.io/v1, Resource=virtualservices", GroupVersionKind: "gateway.solo.io/v1, Kind=VirtualService"
Name: "default", Namespace: "gloo-system"
Object: &{map["apiVersion":"gateway.solo.io/v1" "kind":"VirtualService" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"gateway.solo.io/v1\",\"kind\":\"VirtualService\",\"metadata\":{\"annotations\":{},\"name\":\"default\",\"namespace\":\"gloo-system\"},\"spec\":{\"virtualHost\":{\"domains\":[\"*\"],\"name\":\"gloo-system.default\",\"routes\":[{\"matcher\":{\"exact\":\"/sample-route-1\"},\"routeAction\":{\"single\":{\"upstream\":{\"name\":\"default-petstore-8080\",\"namespace\":\"gloo-system\"}}}}]}}}\n"] "creationTimestamp":"2019-08-20T11:09:00Z" "generation":'\x01' "name":"default" "namespace":"gloo-system" "resourceVersion":"39558469" "selfLink":"/apis/gateway.solo.io/v1/namespaces/gloo-system/virtualservices/default" "uid":"e99ba1a0-c33a-11e9-8be1-42010a8000dc"] "spec":map["virtualHost":map["domains":["*"] "name":"gloo-system.default" "routes":[map["matcher":map["exact":"/sample-route-1"] "routeAction":map["single":map["upstream":map["name":"default-petstore-8080" "namespace":"gloo-system"]]]]]]] "status":map["reported_by":"gateway" "state":'\x01' "subresource_statuses":map["*v1.Proxy gloo-system gateway-proxy-v2":map["reported_by":"gloo" "state":'\x01']]]]}
for: "vs-err.yaml": admission webhook "validating-webhook.openpolicyagent.org" denied the request: prefix re-write not allowed

Cleanup

you can use the teardown.sh to clean-up the resources created in this document.

For your convenience, here’s the content of teardown.sh (click to reveal):

teardown.sh

kubectl delete validatingwebhookconfiguration  opa-validating-webhook
kubectl delete namespace opa
kubectl delete clusterrolebinding opa-viewer opa-gloo-viewer

rm ca.crt ca.key ca.srl server.conf server.crt server.csr server.key

kubectl label namespace kube-system openpolicyagent.org/webhook-