question: how to make trino independent of opa restarts

[maxgruber19]

Since OPA is deployed as a daemonset and there is always only one pod of a replicaset on one node. That leads to "Failed to query OPA backend" errors when the daemonset is restarting because of config changes. Is there any way to get rid of that?

Currently every update to the opa cluster leads to query submission errors on trino side - maybe there is a better way? I was wondering about the reasons for OPA to be deployed as a DS?

[dervoeti]

Hi @maxgruber19,

thanks for reporting this! You're right, this is a real problem and we've been looking into it.

OPA is deployed as a DaemonSet because policy decisions need to be fast. Products like Trino make multiple serial OPA queries per request, so having an OPA agent on every node (with internalTrafficPolicy: Local) avoids cross-node network hops. The current downside is exactly what you're seeing: when the local OPA pod restarts, there's no fallback.

We're working on a fix that will tell Kubernetes to start the new OPA pod and wait for it to become ready before terminating the old one. This eliminates the availability gap during rolling updates. We will ship that in a future SDP release, and even further in the future we can leverage a new Kubernetes feature which tells Kubernetes to prefer the local pod, but fall back to pods on other nodes if that pod is unavailable. That would be the proper fix, but we need to rely on Kubernetes 1.35 to be the minimum required version in SDP.

What you can do very soon:
Our release of SDP 26.3.0 is just around the corner, sadly it won't yet include the fix, but I can already tell you that we added the concept of objectOverrides in that release, which will allow you to do the following in your OpaCluster spec:

spec:
  objectOverrides:
    - apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        name: <your-opa-name>-server-<rolegroup>
        namespace: <namespace>
      spec:
        updateStrategy:
          type: RollingUpdate
          rollingUpdate:
            maxSurge: 1
            maxUnavailable: 0

Replace <your-opa-name>, <rolegroup> and <namespace> with the actual values from your DaemonSet(s) (check with kubectl get daemonsets -A).

That does exactly what the fix we're implementing for a future release will do (start the new pod before the old one terminates) and should fix your problem. As said, you'll have to wait a bit for the official SDP 26.3 release and then upgrade to SDP 26.3 to make use of objectOverrides, then you should be able to solve the problem that way.

[maxgruber19]

Thank you for that good and quick reply as usual. We're looking forward to that feature. I'll let the team know

[Maleware]

Hey, I'm late to the party my apologies.

If upgrading to 26.3 is no option for you or the solution given won't fit there is another path.

You could define and deploy a svc similar to:

---
apiVersion: v1
kind: Service
metadata:
  name: opa-cluster
  namespace: <namespace>  
spec:
  internalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 8081
    protocol: TCP
    targetPort: 8081
  selector:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: opa
    app.kubernetes.io/name: opa
  type: ClusterIP

which noteworthy sets internalTrafficPolicy: Cluster and defines the product this svc adheres to.

Along this you'd need to deploy a configMap like:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-cluster
  namespace: <trino-namespace>
data:
  OPA: http://opa-cluster/.<namespace>.svc.cluster.local:8081/

and point trino to it with

authorization:
      opa:
        configMapName: opa-cluster

This will make the opa-cluster svc a LoadBalancer across all pods of a specific DaemonSet. Since load will be distributed, restarts of single nodes won't degrade the whole TrinoCluster.

[maxgruber19]

@Maleware thats a very good point, we'll check on that as well.

Thank you as well! I think waiting for the new release is fine for us, it's not that critical right now