Proposal to split containers

All the NGINX files should live on one container
- No file other than NGINX files should exist on this container
- This includes not mounting the service account
All the controller files should live on a different container
- Controller container should have bare minimum to work (just go program)
- ServiceAccount should be mounted just on controller
Inside nginx container, there should be a really small http listener just able to start, stop and reload NGINX

Roadmap (what needs to be done)

Map what needs to be done to mount the SA just on controller container
Map all the required files for NGINX to work
Map all the required network calls between controller and NGINX
- eg.: Dynamic lua reconfiguration
Map problematic features that will need attention
- SSLPassthrough today happens on controller process and needs to happen on NGINX

Ports and endpoints on NGINX container

Public HTTP/HTTPs port - 80 and 443
Lua configuration port - 10246 (HTTP) and 10247 (Stream)
3333 (temp) - Dataplane controller http server
- /reload - (POST) Reloads the configuration.
  - "config" argument is the location of temporary file that should be used / moved to nginx.conf
- /test - (POST) Test the configuration of a given file location
  - "config" argument is the location of temporary file that should be tested

Mounting empty SA on controller container

yaml

kind: Pod
apiVersion: v1
metadata:
  name: test
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
  - name: othernginx
    image: alpine:latest
    command: ["/bin/sh"]
    args: ["-c", "while true; do date; sleep 3; done"]
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: emptysecret
  volumes:
  - name: emptysecret
    emptyDir:
      sizeLimit: 1Mi

Mapped folders on NGINX configuration

WARNING We need to be aware of inter mount containers and inode problems. If we mount a file instead of a directory, it may take time to reflect the file value on the target container

"/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;;"; - Lua scripts
"/var/log/nginx" - NGINX logs
"/tmp/nginx (nginx.pid)" - NGINX pid directory / file, fcgi socket, etc
" /etc/nginx/geoip" - GeoIP database directory - OK - /etc/ingress-controller/geoip
/etc/nginx/mime.types - Mime types
/etc/ingress-controller/ssl - SSL directory (fake cert, auth cert)
/etc/ingress-controller/auth - Authentication files
/etc/nginx/modsecurity - Modsecurity configuration
/etc/nginx/owasp-modsecurity-crs - Modsecurity rules
/etc/nginx/tickets.key - SSL tickets - OK - /etc/ingress-controller/tickets.key
/etc/nginx/opentelemetry.toml - OTEL config - OK - /etc/ingress-controller/telemetry
/etc/nginx/opentracing.json - Opentracing config - OK - /etc/ingress-controller/telemetry
/etc/nginx/modules - NGINX modules
/etc/nginx/fastcgi_params (maybe) - fcgi params
/etc/nginx/template - Template, may be used by controller only

List of modules

ngx_http_auth_digest_module.so    ngx_http_modsecurity_module.so
ngx_http_brotli_filter_module.so  ngx_http_opentracing_module.so
ngx_http_brotli_static_module.so  ngx_stream_geoip2_module.so
ngx_http_geoip2_module.so

List of files that may be removed

-rw-r--r--    1 www-data www-data      1077 Jun 23 19:44 fastcgi.conf
-rw-r--r--    1 www-data www-data      1077 Jun 23 19:44 fastcgi.conf.default
-rw-r--r--    1 www-data www-data      1007 Jun 23 19:44 fastcgi_params
-rw-r--r--    1 www-data www-data      1007 Jun 23 19:44 fastcgi_params.default
drwxr-xr-x    2 www-data www-data      4096 Jun 23 19:34 geoip
-rw-r--r--    1 www-data www-data      2837 Jun 23 19:44 koi-utf
-rw-r--r--    1 www-data www-data      2223 Jun 23 19:44 koi-win
drwxr-xr-x    6 www-data www-data      4096 Sep 19 14:13 lua
-rw-r--r--    1 www-data www-data      5349 Jun 23 19:44 mime.types
-rw-r--r--    1 www-data www-data      5349 Jun 23 19:44 mime.types.default
drwxr-xr-x    2 www-data www-data      4096 Jun 23 19:44 modsecurity
drwxr-xr-x    2 www-data www-data      4096 Jun 23 19:44 modules
-rw-r--r--    1 www-data www-data     18275 Oct  1 21:28 nginx.conf
-rw-r--r--    1 www-data www-data      2656 Jun 23 19:44 nginx.conf.default
-rwx------    1 www-data www-data       420 Oct  1 21:28 opentelemetry.toml
-rw-r--r--    1 www-data www-data         2 Oct  1 21:28 opentracing.json
drwxr-xr-x    7 www-data www-data      4096 Jun 23 19:44 owasp-modsecurity-crs
-rw-r--r--    1 www-data www-data       636 Jun 23 19:44 scgi_params
-rw-r--r--    1 www-data www-data       636 Jun 23 19:44 scgi_params.default
drwxr-xr-x    2 www-data www-data      4096 Sep 19 14:13 template
-rw-r--r--    1 www-data www-data       664 Jun 23 19:44 uwsgi_params
-rw-r--r--    1 www-data www-data       664 Jun 23 19:44 uwsgi_params.default
-rw-r--r--    1 www-data www-data      3610 Jun 23 19:44 win-utf