kubernetes 1.9.4 breaking universal forwarder with ConfigMap

[erks]

Using ConfigMap to manage splunk universal forwarder's config has been working well until the release of 1.9.4, which included this security fix.

ConfigMap now gets mounted as read-only always, which breaks this container due to the chown commands in entrypoint.sh.

See: https://answers.splunk.com/answers/626964/kubernetes-194-breaking-changes-universal-forwarde.html

[halr9000]

@mchene ^

[halr9000]

@erks I've alerted PM and got ack back that they will triage.

[erks]

@halr9000 if you're okay with doing chown ... || true, which seems to work for me, I can create a quick PR for that.

[rayh0001]

Any updates? Any temporary fix or work-around?

[ffscl]

@rayh0001
Mount your config maps inside /var/opt/splunk/etc instead of directly inside ${SPLUNK_HOME}. This fix works because the entrypoint copies the files over before trying to change the ownership https://github.com/splunk/docker-splunk/blob/master/universalforwarder/entrypoint.sh#L24

[guilhemmarchand]

Thanks @ffscl
I just tested it in a Kubernetes v1.10.2 cluster and this fixes the issue, I used:

        volumeMounts:
        - name: ufconfig
          mountPath: /var/opt/splunk/etc/apps/search/local

And still reference any file with $SPLUNK_HOME

[rayh0001]

Thanks @ffscl @guilhemmarchand that seems to work

[erks]

This definitely should be documented. @halr9000

[vasartori]

@guilhemmarchand Works like a charm!!

[mychalsexton]

@ffscl That worked like a charm! Thank goodness I found this otherwise I might have gone crazy.

[sharmmoh1983]

Sorry I am not getting what solved the issue here... I updated daemonset yaml with:

volumeMounts:
        - name: ufconfig
          mountPath: /var/opt/splunk/etc/apps/search/local

But nothing seems to work. Please let me know what exactly need to be done to resolve the issue

[sforsman]

@sharmmoh1983 If you are using Splunk Cloud, you need to mount the ConfigMap to /var/opt/splunk/etc/apps/splunkclouduf/default

[pgilad]

Can't get this to work with a daemonset and configmaps. Does anyone has any working example with splunkforwarder:7.3.0 ?