Elasticsearch on Kubernetes: mount errors, uid 1000 permissions and per-index roles

Running a single-node Elasticsearch in Docker takes one docker run command. Move the same setup to Kubernetes and things start to drift: the pod sits in Pending, FailedMount events appear and vanish, the container restarts over and over with AccessDeniedException, and once it finally runs, the Laravel application still reports "No alive nodes found". This article follows a typical case: a single-node Elasticsearch 7.15 with xpack security enabled, shared by several Laravel sites. Each step has a check command, plus a way to grant permissions by index prefix so each site only touches its own data.

The general chart layout and fsGroup are already covered in Shared charts: reusing Helm for Laravel, MySQL and Redis. This article focuses on the errors specific to Elasticsearch.

The problem: one Elasticsearch cluster shared by many sites

Each Laravel site needs full-text search, but a dedicated cluster per site wastes a lot of RAM. A more compact approach is a single-node cluster with xpack security enabled, where each site gets its own role and its own user, allowed to operate only on indices that start with that site's prefix. The cluster runs as a one-replica StatefulSet with a headless Service and a 10Gi volume for data.

A minimal Elasticsearch StatefulSet

The manifest below collects the lessons from the failures described further down. The names search and elasticsearch are only examples.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: search
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      securityContext:
        fsGroup: 1000
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.15.1
        env:
        - name: discovery.type
          value: single-node
        - name: xpack.security.enabled
          value: "true"
        - name: ES_JAVA_OPTS
          value: "-Xms1g -Xmx1g"
        - name: ELASTIC_PASSWORD
          valueFrom:
            secretKeyRef:
              name: elasticsearch-secret
              key: ELASTIC_PASSWORD
        ports:
        - containerPort: 9200
          name: http
        - containerPort: 9300
          name: transport
        resources:
          requests:
            cpu: 500m
            memory: 2Gi
          limits:
            memory: 2Gi
        readinessProbe:
          tcpSocket:
            port: 9200
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: linode-block-storage
      resources:
        requests:
          storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: search
spec:
  clusterIP: None
  selector:
    app: elasticsearch
  ports:
  - port: 9200
    name: http

Environment variable values must be strings, so "true" needs quotes. With xpack security enabled, an httpGet probe on port 9200 receives 401 and counts as a failure, which is why the example uses tcpSocket. The elastic password comes from a Secret and is never written into the manifest. The ES_JAVA_OPTS line is an addition to the original example: it sets the heap to about half of the memory limit. Since 7.11, Elasticsearch picks the heap size from the container's memory on its own; setting it explicitly makes the heap fixed and predictable, but it is not required.

Pod Pending: not enough CPU or RAM

The first error often has nothing to do with Elasticsearch. With a request of a few hundred millicores and a few GiB of RAM, the pod can sit in Pending, and adding one more small node still does not help. The reason is that a node's Allocatable is always lower than the capacity printed on the plan, because the operating system, the kubelet and system pods already take a share. A 2GB node has no room, while a 4GB node lets the pod start. Check the "Allocated resources" section of kubectl describe node to see the real free space; the method is described in Pod Pending with Insufficient cpu: calculating real free capacity from kubectl describe node.

FailedMount "Unable to find device path": transient, not a permissions error

When you run describe pod, you may see a chain of events like this: FailedScheduling with "unbound immediate PersistentVolumeClaims", then SuccessfulAttachVolume, then several FailedMount events with the message "Unable to find device path out of attempted paths", and finally the container is still Started. This is the volume being created and attached to the node; the kubelet retries a few times and succeeds. It is not the reason the pod restarts. If it keeps repeating, read Pod stuck in ContainerCreating on LKE: Multi-Attach and Unable to find device path.

AccessDeniedException on data/nodes: data owned by root, process running as uid 1000

The pod restarts continuously, and kubectl logs shows:

ElasticsearchException[failed to bind service]; nested:
AccessDeniedException[/usr/share/elasticsearch/data/nodes];

According to the Elasticsearch documentation, the image runs as the elasticsearch user (uid 1000, gid 0). A freshly formatted ext4 volume is usually owned by root, so the process cannot create the nodes directory. This is a hypothesis that fits the symptoms and is not verified, because there is no ls -ln output for the data directory. Check it yourself before changing anything:

kubectl -n search exec elasticsearch-0 -- ls -ldn /usr/share/elasticsearch/data
kubectl -n search exec elasticsearch-0 -- id

A common fix is to add a postStart hook that runs chown -R 1000:1000, along with several other changes made at the same time (adding storageClassName, removing subPath), so there is no way to tell which change worked. subPath alone is a second candidate cause: the subdirectory the kubelet creates for a subPath is usually owned by root, so it can also trigger AccessDeniedException. That variable needs to be tested separately; this example has not isolated it. This approach has problems: the postStart hook runs as the container's own user (uid 1000), and a normal user cannot change the owner of a directory owned by root. The hook also runs in parallel with the entrypoint, so it is not guaranteed to finish before Elasticsearch writes data.

Recommendation: use fsGroup: 1000 as in the manifest above, or a root initContainer. The general principle of volume ownership is covered in Permission denied on a volume: why chown in the Dockerfile is not enough.

Granting xpack permissions by index prefix

With the elastic account (a superuser), create a role limited to an index prefix, then create a user bound to that role. Example for a site named site-a:

curl -u "elastic:$ELASTIC_PASSWORD" -X POST \
  "http://localhost:9200/_security/role/site_a_role" \
  -H "Content-Type: application/json" \
  -d '{
    "cluster": ["monitor"],
    "indices": [{
      "names": ["site-a-*"],
      "privileges": ["create_index", "read", "write", "delete"]
    }]
  }'

curl -u "elastic:$ELASTIC_PASSWORD" -X POST \
  "http://localhost:9200/_security/user/site-a" \
  -H "Content-Type: application/json" \
  -d @user-site-a.json

The file user-site-a.json contains password, roles and full_name. This keeps the site user's password out of the shell history; remember to delete the JSON file after creating the user. The elastic password passed with -u is visible in the process list while the command runs; you can put it in a curl config file and use -K instead.

Choose prefixes with a clear separator: the pattern site-a-* also matches site-a-b-..., so do not use one site's prefix as the beginning of another site's name. A common mistake is to grant "all" on both the cluster and the indices: it works, but the site's user becomes an administrator of the whole cluster. Start from monitor and the four privileges above. Some libraries also need view_index_metadata or manage to check whether an index exists; add them when you hit an error, and this is not verified.

Test the permission boundary with the site's own user: creating an index outside the prefix must be rejected.

curl -u "site-a:$SITE_A_PASSWORD" -X PUT "http://localhost:9200/other-index"
# {"error":{"type":"security_exception",
#   "reason":"action [indices:admin/create] is unauthorized for user [site-a] ..."},
#  "status":403}

By contrast, PUT /site-a-products must return acknowledged: true.

Connecting the application and the "No alive nodes found" error

In Laravel, the index prefix in the application configuration must match the role's names; it is a prefix, not a suffix. The connection string has the form user:password@host:9200, and if the password contains special characters such as @, : or /, it must be URL-encoded (percent-encoded), otherwise the string is split in the wrong place.

When the application logs "No alive nodes found in your cluster", a correct nslookup result proves nothing, because DNS is only the first step. In this example, the error went away after creating the role and user and setting a connection string with credentials. Missing authentication is the most likely cause, but this is an inference from the sequence of events, because the client may fold both connection errors and 401 errors into one message. To rule things out in order, run these from inside the application's pod:

1. DNS: nslookup elasticsearch.search.svc.cluster.local.
2. TCP: curl -sS -m 5 http://elasticsearch.search.svc.cluster.local:9200. A 401 means you reached the server and authentication is missing.
3. Authentication: add -u with the site's user and call /_cluster/health.
4. Permissions: try to create or write an index with the correct prefix.
5. Only then look at the client configuration: scheme, port, password encoding.

Yellow indices on a single-node cluster

After you create an index, _cat/indices shows the status yellow. This is normal behavior: an index has one replica by default, and a single-node cluster has no place to put the copy, so the replica stays unassigned. The data is still complete. To get a green status, set number_of_replicas to 0 for the existing indices, and use an index template for indices created later (run this with the elastic account):

curl -u "elastic:$ELASTIC_PASSWORD" -X PUT "http://localhost:9200/site-a-*/_settings" \
  -H "Content-Type: application/json" \
  -d '{"index": {"number_of_replicas": 0}}'

curl -u "elastic:$ELASTIC_PASSWORD" -X PUT "http://localhost:9200/_index_template/site-a" \
  -H "Content-Type: application/json" \
  -d '{"index_patterns": ["site-a-*"], "template": {"settings": {"number_of_replicas": 0}}}'

Removing replicas means there is no second copy of the data, so the cluster needs its own backups; this example has no backup or restore procedure.

Checklist

• Keep the memory request below the free Allocatable of at least one node.
• Separate transient FailedMount events from permission errors in the container log.
• Run ls -ldn and id before changing the owner; prefer fsGroup or an initContainer.
• One role and one user per site, names by prefix; verify with one deliberate 403.
• Do not use all for the cluster or indices; pass passwords through a Secret, not through shell history.
• "No alive nodes found": rule out in order DNS, TCP, authentication, permissions, client.
• Set replicas to 0 on a single-node cluster and plan separate backups.

The role syntax is in the Elasticsearch Create or update roles API documentation, and index templates are in the Index templates documentation. How Kubernetes applies fsGroup to a volume is described in the Security Context documentation. Background on volumes: Volume, PersistentVolume, PVC and StorageClass on Kubernetes.

Next article

When a volume mount hides image content: spotting it and rebuilding directories on a PVC. Mounting a volume over a directory that already contains code hides the content baked into the image; the next article shows how to spot it and rebuild.