NetworkPolicy in Kubernetes: deny by default, then open only what is needed

A cluster runs a dozen applications in separate namespaces. Namespaces organise resources and permissions, but not network traffic: the frontend pod of one application can connect freely to the database of another, and a pod with a vulnerable library can send data to any address on the internet. Once one application is compromised, the rest of the cluster is a single TCP connection away.

The principle is the same as on an edge firewall: deny by default, with explicit and documented exceptions. Below: a set of policies for a typical application (ingress controller, application, database), the specifics of Cilium, and a rollout order that does not end in an outage.

How NetworkPolicy works

• A pod that no policy selects is not isolated: it accepts and sends all traffic.
• Once a policy with policyTypes: [Ingress] selects a pod, incoming traffic to it is dropped except for what the union of rules of all policies selecting that pod allows. The same applies to Egress.
• Policies only allow. The standard API has no “deny” rules, and policies are additive.
• A connection between two isolated pods needs permission on both sides: an egress rule on the source and an ingress rule on the destination.
• The port in a rule is the container port, not the Service port.
• Traffic from the kubelet on the pod’s own node, including liveness and readiness probes, is not blocked.

Policies are enforced by the cluster network (the CNI), not by the Kubernetes API itself:

Cluster network	Enforces NetworkPolicy	Notes
Flannel on its own	no	the API accepts policies, but nothing is blocked
k3s with the default Flannel	yes	built-in policy controller, unless disabled with --disable-network-policy
Calico	yes	plus its own extended policies
Cilium	yes	plus CiliumNetworkPolicy: L7 rules and DNS names

Step 1: default deny

An empty podSelector selects every pod in the namespace, and the absence of rules means nothing is allowed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Step 2: DNS

After default deny on egress, no pod can resolve a name, and applications return timeouts rather than connection errors. The DNS exception allows only the CoreDNS pods in kube-system, on port 53 UDP and TCP:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Kubernetes adds the kubernetes.io/metadata.name label to every namespace automatically, so nothing needs labelling by hand. If the cluster uses NodeLocal DNSCache, queries go to an address local to the node, and the rule has to account for that.

Step 3: ingress controller, application and database

The application accepts traffic only from the ingress controller and connects only to the database. The example assumes Traefik in the traefik namespace; in k3s it lives in kube-system by default.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: shop-api
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: shop-api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: traefik
          podSelector:
            matchLabels:
              app.kubernetes.io/name: traefik
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: shop-db
      ports:
        - protocol: TCP
          port: 5432

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: shop-db
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: shop-db
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: shop-api
      ports:
        - protocol: TCP
          port: 5432

Policy	Selects pods	Allows
default-deny	all	nothing
allow-dns	all	egress to CoreDNS, port 53 UDP and TCP
shop-api	shop-api	ingress from Traefik on 8080, egress to shop-db on 5432
shop-db	shop-db	ingress from shop-api on 5432

Cilium: DNS names, Hubble and audit mode

A standard NetworkPolicy describes traffic leaving the cluster only through address ranges (ipBlock), and the addresses of external services change. A CiliumNetworkPolicy lets you describe the destination by name. The precondition is a DNS rule with a rules.dns section: Cilium then tracks DNS answers and knows which addresses sit behind the name.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: shop-api-egress-fqdn
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: shop-api
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toFQDNs:
        - matchName: api.payments.example.com
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

Cilium enforces both kinds of policy at once and adds up their rules. Dropped connections are visible in Hubble:

hubble observe --namespace shop --verdict DROPPED --last 50

The safest way to roll out policies with Cilium is audit mode: policies are evaluated, but nothing is blocked, and Hubble shows connections that would have been dropped with the AUDIT verdict. The mode is enabled in the Helm values for the whole cluster, which changes the agents’ configuration and requires restarting them:

helm list -n kube-system   # CHART column: the installed version, e.g. cilium-1.17.4
# --version of the same release, so that Cilium is not upgraded along the way
helm upgrade cilium cilium/cilium -n kube-system --version 1.17.4 --reuse-values --set policyAuditMode=true
kubectl -n kube-system rollout restart daemonset/cilium
hubble observe --namespace shop --verdict AUDIT

Rolling out without an outage

1. Observation. For a few days you collect the namespace’s actual connections: hubble observe --namespace shop. Without Cilium, the sources are the application documentation and connection logs.
2. Policies from observation. Every required connection gets a rule; every unexpected one is explained before it gets a rule.
3. Test. With Cilium, audit mode; without it, a rollout on a copy of the environment.
4. Namespace by namespace. Start with the least critical applications. Allow policies and default deny go into a namespace together, because an allow policy on its own already isolates the pods it selects.
5. Review of drops after every rollout and every new application version.

The emergency way back is deleting the default-deny policy from the namespace (kubectl delete networkpolicy default-deny -n shop), which immediately restores traffic not selected by the remaining policies. Keep policies in the same repository as the application manifests: a new service then arrives together with its rules.

Check your own environment

All policies in the cluster:

kubectl get networkpolicy -A
kubectl get ciliumnetworkpolicy -A   # only in clusters running Cilium
# Bad: no output, or policies in only a few namespaces.

Namespaces without any policy:

comm -23 \
  <(kubectl get ns -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | sort) \
  <({ kubectl get networkpolicy -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}'
      kubectl get ciliumnetworkpolicy -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' 2>/dev/null
    } | sort -u)
# Every application namespace listed here has pods with no isolation at all.

Whether namespaces have a default-deny rule:

kubectl get networkpolicy -A -o json | jq -r '.items[]
  | select(.spec.podSelector == {})
  | "\(.metadata.namespace)\t\(.metadata.name)\t\(.spec.policyTypes | join(","))\tingress=\(.spec.ingress | length)\tegress=\(.spec.egress | length)"'
# Default deny: types Ingress,Egress and zero rules of both kinds (ingress=0, egress=0).

Dropped connections after rolling out policies:

hubble observe --verdict DROPPED --last 100
# Good: only traffic that was meant to be blocked. Dropped DNS or database connections mean a missing rule.