I'm running into an issue where my system will lock up due to too many open files. It's been happening for a couple months. I upped the open file limit, but that didn't seem to help.

[root@bbs ctrl]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15605
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 950000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 950000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I'm still getting lots of this, and it will refuse to let anyone log in.

May 28 13:45:33 bbs sbbs: evnt BBS Events DAILY: Purging deleted/expired e-mail May 28 13:45:33 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4671 (daily_maint) opening "/sbbs/data/mail" access=4294967096 info=smb_open_fp 24 'Too many open files' opening /sbbs/data/mail.shd
May 28 13:45:33 bbs sbbs: evnt BBS Events DAILY: Running system event
May 28 13:45:33 bbs sbbs: evnt BBS Events !JavaScript : can't open /sbbs/exec/logonlist.js: No such file or directory
May 28 13:45:33 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in exec.cpp line 662 (js_execfile) compiling "/sbbs/exec/logonlist.js" access=0
May 28 13:45:33 bbs sbbs: evnt BBS Events Daily event: '?logonlist -m' returned -1
May 28 13:45:33 bbs sbbs: evnt BBS Events DAILY: System maintenance ended
May 28 13:45:37 bbs sbbs: evnt BBS Events New Day - Prev: Wed Dec 31 1969 06:00 pm
May 28 13:45:37 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in logon.cpp line 603 (logonstats) opening "/sbbs/data/logon.lst" access=577
May 28 13:45:37 bbs sbbs: evnt BBS Events DAILY: System maintenance begun
May 28 13:45:37 bbs sbbs: evnt BBS Events DAILY: Backing-up user data...
May 28 13:45:37 bbs sbbs: evnt BBS Events DAILY: Backing-up mail data...
May 28 13:45:37 bbs sbbs: evnt BBS Events ERROR -200 (smb_open_fp 24 'Too many open files' opening /sbbs/data/mail.shd) opening mail base
May 28 13:45:37 bbs sbbs: evnt BBS Events DAILY: Checking for inactive/expired user records...
May 28 13:45:37 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 1" access=4294967295
May 28 13:45:37 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 2" access=4294967295
May 28 13:45:37 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 3" access=4294967295
<continues for every user>

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

and then it tries to re-run the daily event.. because it thinks it's Epoch:

May 28 13:37:57 bbs sbbs: evnt BBS Events New Day - Prev: Wed Dec 31 1969 06:00 pm
May 28 13:37:57 bbs sbbs: evnt BBS Events DAILY: System maintenance begun
May 28 13:37:57 bbs sbbs: evnt BBS Events DAILY: Backing-up user data...
May 28 13:37:57 bbs sbbs: evnt BBS Events DAILY: Backing-up mail data...
May 28 13:37:57 bbs sbbs: evnt BBS Events DAILY: Checking for inactive/expired user records...
May 28 13:37:57 bbs sbbs: evnt BBS Events DAILY: Purging deleted/expired e-mail May 28 13:37:58 bbs sbbs: evnt BBS Events DAILY: Running system event
May 28 13:37:58 bbs sbbs: evnt BBS Events DAILY: System maintenance ended
May 28 13:38:03 bbs sbbs: evnt BBS Events New Day - Prev: Wed Dec 31 1969 06:00 pm
May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: System maintenance begun
May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: Backing-up user data...
May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: Backing-up mail data...
May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: Checking for inactive/expired user records...
May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: Purging deleted/expired e-mail May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: Running system event
May 28 13:38:03 bbs sbbs: evnt BBS Events DAILY: System maintenance ended
May 28 13:38:07 bbs sbbs: evnt BBS Events New Day - Prev: Wed Dec 31 1969 06:00 pm
May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: System maintenance begun
May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: Backing-up user data...
May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: Backing-up mail data...
May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: Checking for inactive/expired user records...
May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: Purging deleted/expired e-mail May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: Running system event
May 28 13:38:07 bbs sbbs: evnt BBS Events DAILY: System maintenance ended
May 28 13:38:12 bbs sbbs: evnt BBS Events New Day - Prev: Wed Dec 31 1969 06:00 pm
May 28 13:38:12 bbs journal: Suppressed 4062 messages from /system.slice/sbbs.service
May 28 13:38:12 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in logon.cpp line 603 (logonstats) opening "/sbbs/data/logon.lst" access=577
May 28 13:38:12 bbs sbbs: evnt BBS Events DAILY: System maintenance begun
May 28 13:38:12 bbs sbbs: evnt BBS Events DAILY: Backing-up user data...
May 28 13:38:12 bbs sbbs: evnt BBS Events DAILY: Backing-up mail data...
May 28 13:38:12 bbs sbbs: evnt BBS Events ERROR -200 (smb_open_fp 24 'Too many open files' opening /sbbs/data/mail.shd) opening mail base
May 28 13:38:12 bbs sbbs: evnt BBS Events DAILY: Checking for inactive/expired user records...
May 28 13:38:12 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 1" access=4294967295
May 28 13:38:12 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 2" access=4294967295
May 28 13:38:12 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 3" access=4294967295
May 28 13:38:12 bbs sbbs: evnt BBS Events !ERROR 24 (Too many open files) in main.cpp line 4580 (daily_maint) reading "user record 4" access=4294967295


-=-=-=-=-==-=-=-=-=-=-=-




Now, when I checked, lsof | wc -l only showed about 72K open files. the majority were by sbbs/term, and sbbs/http to /sbbs/data/user/user.dat

after killing sbbs and restarting, my open files were down to 12k.


The kicker is, 72k is far less than the 950K my system allows. is there an open file limit built into SBBS?

DaiTengu

... I'd give my right arm to be ambidextrous.

---
þ Synchronet þ War Ensemble BBS - The sport is war, total war - warensemble.com

Re: too many open files - user.dat
By: DaiTengu to All on Sat May 28 2022 02:24 pm

I'm running into an issue where my system will lock up due to too many open files. It's been happening for a couple months. I upped the open file limit, but that didn't seem to help.

[root@bbs ctrl]# ulimit -a
open files (-n) 950000

Unless you're running the bbs as 'root' (and I hope you're not), this isn't really relevant. You need to adjust the open file limit for the user that sbbs is run as:
http://wiki.synchro.net/faq:nix#maximum_open_files

Now, when I checked, lsof | wc -l only showed about 72K open files.

Unfortunately, lsof is not accurate with multi-threaded programs since it'll list/count all open files for the process, repeated for each thread of the process. So lsof will report a much larger number than the actual number of open files.

the
majority were by sbbs/term, and sbbs/http to /sbbs/data/user/user.dat

after killing sbbs and restarting, my open files were down to 12k.

The kicker is, 72k is far less than the 950K my system allows. is there an open file limit built into SBBS?

No. I suspect you just haven't correctly adjusted your open file limit for the sbbs user/process yet.
--
digital man (rob)

Sling Blade quote #17:
Charles Bushman: A shovel just makes too goddamned much racket.
Norco, CA WX: 63.0øF, 77.0% humidity, 0 mph ENE wind, 0.00 inches rain/24hrs ---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

Re: too many open files - user.dat
By: Digital Man to DaiTengu on Sat May 28 2022 12:42 pm

Unless you're running the bbs as 'root' (and I hope you're not), this isn't really relevant. You need to adjust the open file limit for the user that sbbs is run as: http://wiki.synchro.net/faq:nix#maximum_open_files

Found the problem, which the wiki link helped me with. I had to up the open file limit in the systemd start config. I had it set to 950k for both users and root, but didn't update the startup config.

LimitNOFILE=150000

Have I ever mentioned how much I hate systemd? :)

Thanks!

DaiTengu

... DOS never says EXCELLENT command or filename...

---
þ Synchronet þ War Ensemble BBS - The sport is war, total war - warensemble.com

DaiTengu wrote to Digital Man <=-

Re: too many open files - user.dat
By: Digital Man to DaiTengu on Sat May 28 2022 12:42 pm

Unless you're running the bbs as 'root' (and I hope you're not), this isn't really relevant. You need to adjust the open file limit for the user that sbbs is run as: http://wiki.synchro.net/faq:nix#maximum_open_files

Found the problem, which the wiki link helped me with. I had to
up the open file limit in the systemd start config. I had it set
to 950k for both users and root, but didn't update the startup
config.

LimitNOFILE=150000

Have I ever mentioned how much I hate systemd? :)

Slackware.
MX Linux.
Devuan.
<a few more>

;-)



... So easy, a child could do it. Child sold separately.
--- MultiMail/Linux v0.52
þ Synchronet þ Palantir BBS * palantirbbs.ddns.net * Pensacola, FL

Re: Re: too many open files - user.dat
By: Gamgee to DaiTengu on Sun May 29 2022 12:18 pm

Slackware.
MX Linux.
Devuan.
<a few more>

Gentoo can run on either. My BBS runs under CentOS though, as it was just easier to deal with for the most part.

at some point I need to move it to CentOS 8 or 9. Or maybe Rocky.

DaiTengu

... Humour is emotional chaos remembered in tranquillity.

---
þ Synchronet þ War Ensemble BBS - The sport is war, total war - warensemble.com

DaiTengu wrote to Gamgee <=-

Slackware.
MX Linux.
Devuan.
<a few more>

Gentoo can run on either. My BBS runs under CentOS though, as
it was just easier to deal with for the most part.

at some point I need to move it to CentOS 8 or 9. Or maybe Rocky.

Okay..... but <confused look> that isn't going to solve anything
regarding using 'systemd', which is what the whole point of my comment
was...

Are you locked in to running it only on RedHat variants, or something?



... Want to meet new people? Pick up the wrong golf ball.
=== MultiMail/Linux v0.52
--- SBBSecho 3.15-Linux
* Origin: Palantir * palantirbbs.ddns.net * Pensacola, FL * (1:123/115)
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

Re: Re: too many open files - user.dat
By: Gamgee to DaiTengu on Mon May 30 2022 08:33 am

Are you locked in to running it only on RedHat variants, or something?

Maybe he prefers it because of familiarity? I certainly prefer Debian, although I started on Slackware in the early 90's, and use SuSE at work currently (because we're licensed, and run software packaged for it) and Amazon Linux, because that's the best-supported distro to run on an EC2 instance.

o
(O)
BeLLy

---
þ Synchronet þ bbs.brazi.net þ www.brazi.net þ WARNING: May contain nuts

Re: Re: too many open files - user.dat
By: Gamgee to DaiTengu on Mon May 30 2022 08:33 am

Gentoo can run on either. My BBS runs under CentOS though, as
it was just easier to deal with for the most part.

at some point I need to move it to CentOS 8 or 9. Or maybe Rocky.

Okay..... but <confused look> that isn't going to solve anything regarding using 'systemd', which is what the whole point of my comment was...

I can hate systemd and still use it.

RedHat variants are my OS of choice for systems I want to "just work" because I'm familiar with them.

DaiTengu

... A oscillator will oscillate at the wrong frequency ...if it oscillates.

---
þ Synchronet þ War Ensemble BBS - The sport is war, total war - warensemble.com

DaiTengu wrote to Gamgee <=-

Gentoo can run on either. My BBS runs under CentOS though, as
it was just easier to deal with for the most part.

at some point I need to move it to CentOS 8 or 9. Or maybe Rocky.

Okay..... but <confused look> that isn't going to solve anything
regarding using 'systemd', which is what the whole point of my comment was...

I can hate systemd and still use it.

Sure. Just seems a little odd. I also hate systemd and won't use a
distro for that very reason.

RedHat variants are my OS of choice for systems I want to "just
work" because I'm familiar with them.

Okey dokey.



... Gone crazy, be back later, please leave message.
--- MultiMail/Linux v0.52
þ Synchronet þ Palantir BBS * palantirbbs.ddns.net * Pensacola, FL

Re: Re: too many open files - user.dat
By: Gamgee to DaiTengu on Tue May 31 2022 10:28 am

Okay..... but <confused look> that isn't going to solve anything
regarding using 'systemd', which is what the whole point of my
comment was...

I can hate systemd and still use it.

Sure. Just seems a little odd. I also hate systemd and won't use a
distro for that very reason.

For the most part, I don't have that luxury. managing thousands of CentOS boxes is my day job. At least until everything moves over to Kubernetes.

DaiTengu

... Heisenberg may have slept here

---
þ Synchronet þ War Ensemble BBS - The sport is war, total war - warensemble.com

On Sat, 28 May 2022 14:24:51 -0500
"DaiTengu" <daitengu@VERT/ENSEMBLE> wrote:

I'm running into an issue where my system will lock up due to too
many open files. It's been happening for a couple months. I upped
the open file limit, but that didn't seem to help.

Digital Man submitted a fix this afternoon which may address the open
files issue. Mine seem to be at a steady level for now so you may want
to update.
--
End Of The Line BBS - Plano, TX
telnet endofthelinebbs.com 23
---
þ Synchronet þ End Of The Line BBS - endofthelinebbs.com