I have a single Nginx container that handles reverse proxying of all my selfhosted services, and I break every service out into its own configuration file, and use include directives to share common configuration across them. For anyone out there with Nginx experience, my Lemmy configuration file should make it fairly clear in terms of how I handle what I described above:

server {
  include ssl_common.conf;
  server_name lm.williampuckering.com;
  set $backend_client lemmy-ui:1234;
  set $backend_server lemmy-server:8536;
  
  location / {
    set $authentication "Authentication Required";
    include /etc/nginx/proxy_nocache_backend.conf;
    
    if ($http_accept = "application/activity+json") {
      set $authentication off;
      set $backend_client $backend_server;
    }
    if ($http_accept = "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"") {
      set $authentication off;
      set $backend_client $backend_server;
    }
    if ($request_method = POST) {
      set $authentication off;
      set $backend_client $backend_server;
    }
    
    auth_basic $authentication;
    auth_basic_user_file htpasswd;
    proxy_pass http://$backend_client;
  }
  
  location ~* ^/(api|feeds|nodeinfo|.well-known) {
    include /etc/nginx/proxy_nocache_backend.conf;
    proxy_pass http://$backend_server;
  }
  
  location ~* ^/pictrs {
    proxy_cache lemmy_cache;
    include /etc/nginx/proxy_cache_backend.conf;
    proxy_pass http://$backend_server;
  }
  
  location ~* ^/static {
    proxy_cache lemmy_cache;
    include /etc/nginx/proxy_cache_backend.conf;
    proxy_pass http://$backend_client;
  }
  
  location ~* ^/css {
    proxy_cache lemmy_cache;
    include /etc/nginx/proxy_cache_backend.conf;
    proxy_pass http://$backend_client;
  }
}

It’s definitely in need of some clean-up (for instance, there’s no need for multiple location blocks that do the same thing for caching, a single expression can handle all of the ones with identical configuration to reduce the number of lines required), but I’ve been a bit lazy to clean things up. However it should serve as a good example and communicate the general idea of what I’m doing.