I discovered some error logs that indicated I was hitting the limit for concurrent open files. Increasing the limit is actually very easy.
Check the current system-wide limit:
$ cat /proc/sys/fs/file-max
To increase this to 65535 we would do (as root):
$ echo "65535" > /proc/sys/fs/file-max
If you want this new value to survive across reboots you can add it to /etc/sysctl.conf:
# Maximum number of open files permited fs.file-max = 65535
That’s it for system limits, however you should also check user limits:
$ ulimit -n 1024
(use ulimit -a to see all user limits)
To increase this to 65535 for all users (as root), edit /etc/security/limits.conf, and modify or add “nofile” (number of file) entries – note that a userid can be used in place of an asterisk (*)
* soft nofile 65535 * hard nofile 65535
References
Fix ‘Too many open files’ error on Linux by increasing filehandles
Linux: Increasing the number of open file descriptors
Maximum Number of open files and file descriptors in Linux