How to Fix Slow Samba smbd on OpenBSD

So I have been testing out a Samba file server on my OpenBSD 6.7 machine. Its read speed from the OpenBSD Samba server to a Windows 10 client was about 100MB/sec, right where I would expect it with Gigabit Ethernet.

However, the write speed was an abysmal 25 MB/sec! On my (much slower hardware) Linux Samba server I get about 100MB/sec in both directions. But with OpenBSD the write is terrible by default. I thought perhaps it could just be Linux file caching and delayed writes were beating the more conservative OpenBSD.

After some experimentation I found that doing two things helped the BSD machine get up to the level of the Linux machine.

So how do you fix the slow data transfer with Samba running on OpenBSD? Like this:

First increase the buffers for OpenBSD:

# sysctl -w net.inet.divert.sendspace=262144

# sysctl -w net.inet.divert.recvspace=262144

# sysctl -w net.inet6.divert.sendspace=262144

# sysctl -w net.inet6.divert.recvspace=262144

These changes to sysctl will not persist after a reboot. So once you are sure it is all working you can add them to /etc/sysctl.conf by adding just the following lines to /etc/sysctl.conf

net.inet.divert.sendspace=262144
net.inet.divert.recvspace=262144
net.inet6.divert.sendspace=262144
net.inet6.divert.recvspace=262144

Next we adjust the buffers in /etc/samba/smb.conf:

[global]

#put right after global
socket options = SO_SNDBUF=262144 SO_RCVBUF=262144

...
#rest of file

Restart Samba:

# rcctl restart smbd

Try out your new faster Samba on OpenBSD!

I experimented with smaller and larger values for these buffers. On my hardware these were the fastest values. YMMV; so perhaps test it out a little bigger or smaller. It’s probably best to keep these powers of two. It will, after all, affect EVERY Samba file transfer. So any improvement is a good thing!

And that is how you fix slow write speeds on Samba smbd for OpenBSD.

Posted in Uncategorized | Tagged , , , , , , , , , | Leave a comment

ping and traceroute Unable to Resolve Hostnames While Other Programs Work Fine

I was chatting with a fellow OpenBSD user on IRC. He was having some very strange problems with name resolution on his OpenBSD machine.

ping www.openbsd.org
ping: no address associated with name

Name resolution as his non-root user worked fine with host or nslookup. Most programs would work without a hitch. However, he was unable to ping or traceroute any hosts on the internet. Both programs would return that there was no address associated with the name (any name).

We went through and checked his resolv.conf file for accuracy. We checked his /etc/hosts file. He was using a local nameserver, which he could query directly with no problems using tools like host or nslookup. He could query 8.8.8.8 directly as well; so no firewall issues.

However, if he put 8.8.8.8 in /etc/resolv.conf it would fail again.

I had him try it as root to try to rule out permissions issues. It still failed as root. So you would think no permissions issues. The programs ping and traceroute are both setuid root anyway.

Finally on a hunch I had him check the permissions on /etc/resolv.conf

It was like this:

-rw-r-----  1 root  wheel  66 Aug 22 02:10 /etc/resolv.conf

Hopefully you are screaming at the screen that you know the issue… That /etc/resolv.conf is only owner (root) and group (wheel) readable. No user outside the group wheel can read it.

So why would this make host and nslookup work fine and ping and traceroute fail?

Let’s think about that for a moment. If the gentleman I was chatting with on IRC’s local user was in group wheel (it was) then any program run as HIS user would work fine because /etc/resolv.conf was group wheel and was group readable.

Then why wouldn’t ping and traceroute work? They are setuid root. So root should be able to read resolv.conf…

They are setuid when they start, but the programmers are smart enough to change the uid after they get the raw socket needed for sending ICMP. (At least on OpenBSD ping they are.)

How can we see this? Try to ping some website. Start ping in the background as root:

ping www.OpenBSD.org &

Now search the process list for ping to see what user is running ping:

 ps auxww | grep ping
_ping    47674  0.0  0.0   668   596 p5  Sp      1:51AM    0:00.04 ping www.OpenBSD.org

WTF? How is ping now running as the user _ping? The ping program revokes its root privileges early by setting the user id to _ping. So even when run as root the ping program is only running as root for a short time to get the socket, and the changing over to _ping.

Don’t forget to kill that ping process you started in the background.

Well, the user _ping is not in the group wheel! So _ping cannot read /etc/resolv.conf ! And hence our program cannot resolve names. The program traceroute works in basically the same way. So all of this applies to it as well.

His solution was simple (as root):

chmod 644 /etc/resolv.conf

Now /etc/resolv.conf is world readable (other readable) as it should be, and everything works fine because the _ping user can access it, and all the other users can too, like the should be able to.

Problem solved.

What would have been another (possibly more efficient) approach to solving this?

ktrace of course! If we trace our system calls with ktrace we can see the problem our program is facing. In this case we need to run it as root since ping is setuid root.

# ktrace -f ping.out ping openbsd.org
ping: no address associated with name
# kdump -f ping.out | grep -n1 resolv.conf
365- 20730 ping     CALL  stat(0x1069c5f03a6d,0x7f7ffffcbef0)
366: 20730 ping     NAMI  "/etc/resolv.conf"
367- 20730 ping     STRU  struct stat { dev=0, ino=103983, mode=-rw-r----- , nlink=1, uid=0<"root">, gid=0<"wheel">, rdev=420253, atime=1598683868<"Aug 29 01:51:08 2020">.621464705, mtime=1598080210<"Aug 22 02:10:10 2020">.244384888, ctime=1598684275<"Aug 29 01:57:55 2020">.564346572, size=66, blocks=4, blksize=16384, flags=0x0, gen=0x0 }
--
375- 20730 ping     CALL  open(0x1069c5f03a6d,0x10000<O_RDONLY|O_CLOEXEC>)
376: 20730 ping     NAMI  "/etc/resolv.conf"
377- 20730 ping     RET   open -1 errno 13 Permission denied

The first occurrence of /etc/resolv.conf is just to stat() the file to make sure it exists and check permissions, etc. You can even see the bad permissions listed right there in the “STRU struct stat” line:

mode=-rw-r-----

The second occurrence of /etc/resolv.conf is where ping tries to open it. Now running as the user _ping, this has the return (RET) from the open() call:

RET   open -1 errno 13 Permission denied

It’s right there in black and white (green and black?)

So we could have used ktrace to diagnose this (possibly) a little faster. (Took about 10 mins of trying and checking various things.) It’s a great troubleshooting tool!

Hopefully you have learned a little something, or just figured out why ping or traceroute keep returning:

ping: no address associated with name

or

traceroute: no address associated with name

while other programs work fine.

Posted in Uncategorized | Tagged , , , , , , , , , , , , | Leave a comment

Build Alpine 2.23 on OpenBSD to Fix Hang When Sending Mail

Once again, I love the Alpine MUA. Simple and robust, it just works, and is much faster than a GUI mail client.

During a recent OpenBSD install (6.7) I ran into a issue where Alpine 2.21 from Ports (alpine-2.21p4) hangs when “Sending Mail.” Occasionally, but repeatably if you send a number of messages Alpine will hang when it gets to Sending Mail. I tried the usual fixes… I checked the “Sendmail Path” and the “SMTP Server (for sending)” settings in the config and tested that those were working. The code would suggest that Sendmail Path is used first. So if that is not working on your system then you want to remove that setting in the config (it’s down lower in the config for some reason, rather than up at the top) before setting your SMTP Server.

This was not the issue in my case, and unlikely your issue if you have smtpd configured and running. Sending via sendmail -t and the SMTP server on localhost worked fine most of the time… But then every few emails (sometimes as few as two, sometimes ten or more) Alpine would hang on Sending Mail and I would have to kill it with a kill -9.

For the curious (skip this unless you like code), this is the ktrace where it would hang every time. The last line is me killing it:

 53976 alpine   NAMI  "/tmp/pine_p43476"
 53976 alpine   RET   lstat -1 errno 2 No such file or directory
 53976 alpine   CALL  open(0x32ce7654b80,0xa01<O_WRONLY|O_CREAT|O_EXCL>,0600<S_IRUSR|S_IWUSR>)
 53976 alpine   NAMI  "/tmp/pine_p43476"
 53976 alpine   RET   open 6
 53976 alpine   CALL  close(6)
 53976 alpine   RET   close 0
 53976 alpine   CALL  pipe(0x7f7ffffc41b8)
 53976 alpine   STRU  int [2] { 6, 7 }
 53976 alpine   RET   pipe 0
 53976 alpine   CALL  sigaction(SIGCHLD,0x7f7ffffc3d28,0x7f7ffffc3d18)
 53976 alpine   STRU  struct sigaction { handler=0x32a0e994e70, mask=0<>, flags=0x2<SA_RESTART> }
 53976 alpine   STRU  struct sigaction { handler=0x32a0e994e70, mask=0<>, flags=0x2<SA_RESTART> }
 53976 alpine   RET   sigaction 0
 53976 alpine   CALL  futex(0x32cf9b38690,0x81<FUTEX_WAIT|FUTEX_PRIVATE_FLAG>,2,0,0)
 53976 alpine   RET   nanosleep 0
 53976 alpine   CALL  kbind(0x32c364e8868,24,0x400e0443c3a0c61)
 53976 alpine   RET   kbind 0
 53976 alpine   CALL  gettimeofday(0x32c364e8660,0)
 53976 alpine   STRU  struct timeval { 1598335368<"Aug 25 01:02:48 2020">.986756 }
 53976 alpine   RET   gettimeofday 0
 53976 alpine   CALL  futex(0x32cf9b38618,0x81<FUTEX_WAIT|FUTEX_PRIVATE_FLAG>,2,0,0)
 53976 alpine   PSIG  SIGKILL SIG_DFL

A backtrace in gdb showed it was called from _rthread_mutex_timedlock. However, I did not research it further. I am not up on debugging pthread programs, and gdb kept breaking constantly when changing threads.

The Fix

Anyway, I built and installed the latest Alpine 2.23 from the official source. I had to patch it with the patches from the ports collection, as well as with my own highly recommended patch to fix the “[Folder vulnerable – directory /var/mail must have 1777 protection]” issue on OpenBSD detailed here. Without the patch on that other page your mailbox may be subject to corruption. Check it out.

First download the latest source tarball from the official Alpine site. I like to do all this in $HOME/src/ but you can do this wherever you like. Also grab my complete patch (SHA256) which combines the OpenBSD Ports patches plus my Folder Vulnerable patch in one big patch file. (You might want to check the hash on the Alpine distro with the one posted on the Alpine site.)

wget http://alpine.x10host.com/alpine/release/src/alpine-2.23.tar.xz
wget https://sourceopen.com/alpine-2.23-openbsd-patch
wget https://sourceopen.com/alpine-2.23-openbsd-patch.sha256

Note this is an .xz compressed archive for some reason instead of the usual .gzip. So:

unxz alpine-2.23.tar.xz

Then expand the tar archive:

tar xf alpine-2.23.tar

Next we will apply the patch to the source code.

A little security note: Please take a look at the patch and make sure there is nothing untoward. I point this out only because applying random patches can possibly be dangerous. Rest assured, I am a nice guy and not trying to do anything bad, but you want to be in the habit of at least looking at the lines in the patch preceded by “+” to make sure there is nothing fishy going on. This works perfectly here. YMMV.

We first check the hash of the patch before applying it.

sha256 alpine-2.23-openbsd-patch

This should result in output like this (the hash should match exactly):

SHA256 alpine-2.23-openbsd-patch) = e7c6d44678030dd11e6e686da2d8997810408561bf51e461a0914cce6baa2791

Next we apply the patch from the directory one above the alpine-2.23 directory. So if you have /$HOME/src/alpine-2.23 then you should still be in the directory you downloaded to /$HOME/src/. This command and this patch will patch all the various files in the alpine-2.23 directory at once!

patch -p0 < alpine-2.23-openbsd-patch

This should result in a bunch of output from patch. Look over this and make sure all the “hunks” succeeded.

Next we configure and make, but we do not install. You could use make to install, but this will install all the other programs you likely don’t need like pico (use GNU nano instead) and pilot, etc. We will install manually below. It’s easy.

cd alpine-2.23
./configure
make

# Or you can do them both if you want to walk away while it build with:
./configure && make

This will build everything up, hopefully with no errors. It is possible there are prerequisite packages (dependencies) that need to be built or installed. In this case you might be easiest to first install the package for alpine, letting pkg_add install the dependencies, and then uninstall the alpine package without removing dependencies.

pkg_add alpine
#(let it install dependencies)
pkg_delete alpine
#(only deletes alpine by default and not dependencies)

After running this go back and run the ./configure and make again. Probably a good idea to do a make distclean first, in case any of the missing stuff was cached as missing.

make distclean
./configure && make

Now you can manually install just the alpine binary. Make sure you are in the alpine-2.23 directory and run:

doas install -o root -g bin -m 0755 -b alpine/alpine /usr/local/bin/alpine

The -b option will backup any existing alpine to /usr/local/bin/alpine.old. Be sure to delete this when you are sure you don’t need it any more. If you get an error about doas not being enabled then su to root and install it without the “doas” at the start of the line. (Or enable doas if you prefer. I don’t care for doas, and don’t use it.)

Now start up Alpine (just run “alpine”) and make sure you are seeing version 2.23 in the top banner. That’s it! This new update solved my issue of Alpine hanging on Sending Mail in OpenBSD. You should also now not have to worry about the whole Folder Vulnerable issue as well, as this patch is included.

Let me know in the comments how this worked for you.

Posted in Uncategorized | Tagged , , , , , , , , , , , , , | Leave a comment

A Journey with the Alpine Mail User Agent (MUA) in OpenBSD or How To Safely Fix: [Folder vulnerable – directory /var/mail must have 1777 protection]

Let me start by saying how much I love the Alpine mail client. It’s fast and easy. I can hit the “d” key ten times and delete as many messages in the amount of time I can click on one email in a GUI mail client.

So in my foray into OpenBSD I ran into the annoying, but ever-present, issue with mbox locking that you get into with Alpine. The dreaded:

[Folder vulnerable - directory /var/mail must have 1777 protection]

There have been countless arguments about this over the years. You can be sure it is a VERY good idea to use some form of locking when accessing mbox files. Without this you are likely to eventually corrupt your entire mailbox.

Mode 1777 sets the sticky bit on the mail spool directory (/var/mail on OpenBSD) and makes that directory world writable (so users can write lock files here). With the sticky bit this is isn’t as bad as it seems, but it does allow nefarious users to potentially fill your mail directory. Some admins work around this with quotas, blah, blah, blah 😉

The alternative is the default permissions of 0755 with root:wheel ownership. With this setup you must be root to write a lock file in the /var/mail directory. This means that Alpine for normal users cannot properly lock the mbox. This is bad. Don’t leave it like this and assume, “She’ll be right!”

There are a couple of alternatives including Alpine’s suggestion (only if you can’t set /var/mail to 1777) of using an external setgid mail locking program like mlock, which comes with the uw-imap setup (in the Alpine source).

From the uw-imap distribution (with Alpine) FAQ:

      /var/spool/mail must have 1777 protection mean? How can I fix this?

      In order to update a mailbox in the default UNIX format, it is
      necessary to create a lock file to prevent the mailer from
      delivering mail while an update is in progress. Some systems use
      a directory protection of 775, requiring that all mail handling
      programs be setgid mail; or of 755, requiring that all mail
      handling programs be setuid root.

      The IMAP toolkit does not run with any special privileges, and I
      plan to keep it that way. It is antithetical to the concept of a
      toolkit if users can't write their own programs to use it. Also,
      I've had enough bad experiences with security bugs while running
      privileged; the IMAP and POP servers have to be root when not
      logged in, in order to be able to log themselves in. I don't
      want to go any deeper down that slippery slope.

      Directory protection 1777 is secure enough on most well-managed
      systems. If you can't trust your users with a 1777 mail spool
      (petty harassment is about the limit of the abuse exposure),
      then you have much worse problems then that.

      If you absolutely insist upon requiring privileges to create a
      lock file, external file locking can be done via a setgid mail
      program named /etc/mlock (this is defined by LOCKPGM in the
      c-client Makefile). If the toolkit is unable to create a
      <...mailbox...>.lock file in the directory by itself, it will
      try to call mlock to do it. I do not recommend doing this for
      performance reasons.

      A sample mlock program is included as part of imap-2010. We have
      tried to make this sample program secure, but it has not been
      thoroughly audited.

Ok, you say… “Why can’t I just set /var/mail to 1777 and be done with it?”

Well, you should be able to, kind of. Except this doesn’t work with OpenBSD’s built-in MTA, OpenSmtpd (known as just smtpd on OpenBSD). Although the stable release of 6.7 says it should work, alas, it does not.

For (good) security purposes some pledge() code was added to /usr/libexec/lockspool. This is a good thing since locksppol is setuid root! The mail delivery program used for smtpd is mail.local. The mail.local process makes calls to seteuid() if the mode on /var/mail is 1777. (Actually lockspool makes this call, but the code is part of the mail.local src). However, lockspool has pledge()d that it will not call seteuid()! So this simply fails and you end up with this in your mail logs (/var/log/maillog)

Aug 23 16:53:01 <hostname> mail.local: lockspool: unable to get lock

If you are seeing these in your logs you are likely also seeing these in /var/log/messages:

Aug 23 16:53:01 <hostname> /bsd: lockspool[85982]: pledge "id", syscall 183

This is the kernel saying “Whoa there!” to lockspool– it did not include the “id” permission in its pledge() call, so calling syscall 183 (SYS_seteuid) is not allowed. The kernel unpleasantly terminates the lockspool process and the mail never gets delivered.

If you found this article via search, and found just this bit above above the log messages the solution is to remove the sticky bit from /var/mail and to set it back to mode 755. This is not the solution to make Alpine work right, though.

Well this goes against the (stable branch) manpage of mail.local:

     Significant efforts have been made to ensure that mail.local acts as
     securely as possible if the spool directory is mode 1777 or 755.  The
     default of mode 755 is more secure, but it prevents mail clients from
     using username.lock style locking.  The use of 1777 is more flexible in
     an NFS shared-spool environment, so many sites use it.  However, it does
     carry some risks, such as attackers filling the spool disk.  Some of
     these problems may be alleviated by making the spool a separate
     filesystem, and placing quotas on it.  The use of any mode other than
     1777 and 755 for the spool directory is recommended against but may work
     properly.

So either lockspool has a bug and needs to allow the seteuid() from its pledge(), or the documentations is wrong. It seems that now in the current branch of OpenBSD the docs have been changed and mail.local now states that only mode root:wheel 0755 is supported for /var/mail, and that processes need to be root (or setuid) to write to /var/mail. Good or bad, this does resolve the bug into a feature.

But Where does this leave us with Alpine?

Good question. My first thought was to just use mlock from the Alpine folks. This is a setgid program that runs as the mail user. I did get this working fine. I had to set the group of /var/mail and the mailboxes to _smmsp as hard coded in the (OpenBSD port Patched version) source for mlock. Then I had to install mlock in /usr/local/libexec/mlock and set it to the same gid and chmod it g+s. But this is not the ideal solution. At least to me it isn’t. (FYI: If you want to build the ports Alpine with mlock I think you need to set the environment variable SUBPACKAGE to “-imap” before you make. Though you don’t need that if you use my proposed better solution below.)

Even the Alpine docs quoted above claim mlock has not be audited for security.

We have tried to make this sample program secure, 
but it has not been thoroughly audited.

I had modified a few things in my copy (made the strcpy()s use strncpy() and such – as well as added some unveil() and pledge() calls to the start of it, which generally made it safer.) However, I still wasn’t happy. I now had Alpine using this setgid locking program that was not 100% guaranteed to work with the locking of smtpd (pretty sure it did, though).

Hmmm… So how does OpenBSD’s smtpd handle locking by default? It uses /usr/libexec/lockspool. This is a setuid root locking program. Naturally a program that is setuid root should be scarier to the admin than a program that is setgid to the mail user. However, this program is simpler and seems to have better code. It already has the unveil() and pledge() calls for added protection, and it is supported as part of the OpenBSD install. Hopefully it is audited as well. Nevertheless, locksppol is already being used all the time by smtpd. So you are likely already vulnerable if there is an issue.

So in an ideal world we would patch Alpine itself to use lockspool rather than mlock. This let’s OpenBSD keep its /var/mail root:wheel 0755, and “just works.” Sure, running the external locking program is a hint slower than Alpine’s internal locking designed for a mail spool with mode 1777, but this patched Alpine method will work correctly with OpenBSD as is. It is pretty clear from the current branch manpage for mail.local that they want to keep /var/mail as is, and not use 1777 mode for it. From that new current branch mail.local manpage:

Significant effort has been made to ensure that mail.local
acts as securely as possible. It will only deliver to a mail
spool directory that is not world-writable. The default mode
of /var/mail on OpenBSD is 755, which prevents non-root
processes from creating mail spool files. The MTA is 
expected to either create the mail spool file itself, or 
call mail.local as root.

OK. Reasonable approach for best security.

Patches

So here are the patches. I plan on submitting these to the ports maintainer.

This first patch is to modify the patch that came with the ports package of Alpine found in /usr/ports/mail/alpine/patches. All the modification does is adjust the LOCKPGM variable to point to lockspool.

Edit the file:

/usr/ports/mail/alpine/patches/patch-imap_src_osdep_unix_Makefile 

Change this line:

         LOCKPGM=$(PREFIX)/libexec/mlock \

to this:

         LOCKPGM=/usr/libexec/lockspool \

You possibly modify that in your build directory manually if you needed to. May need to” make clean” as well if you do.

This second patch modifies c-client, which is used by alpine to access the mboxes. First, here is a sample of just the changes I made. I am going to provide a patch to reply the ports patch below, but it will be confusing with the ports patches in there as well.

These are my changes to make Alpine use lockspool on OpenBSD. This is a diff against the already patched source file. You want the file below for your machine

--- ./imap/src/osdep/unix/env_unix.c.ports_patched  Mon Aug 24 16:54:40 2020
+++ ./imap/src/osdep/unix/env_unix.c    Mon Aug 24 17:06:26 2020
@@ -1198,14 +1198,12 @@
   case EACCES:                 /* protection failure? */
     MM_CRITICAL (NIL);         /* go critical */
     if (closedBox || !lockpgm);        /* can't do on closed box or disabled */
-    else if ((*lockpgm && stat (lockpgm,&sb)) ||
-            (!*lockpgm && stat (lockpgm = LOCKPGM1,&sb) &&
-             stat (lockpgm = LOCKPGM2,&sb) && stat (lockpgm = LOCKPGM3,&sb) &&
-             stat (lockpgm = LOCKPGM4,&sb)))
+    else if ((*lockpgm && stat (lockpgm,&sb)))
       lockpgm = NIL;           /* disable if can't find lockpgm */
+                            /* /usr/libexec/lockspool for OpenBSD */
     else if (pipe (pi) >= 0) { /* make command pipes */
       long cf;
-      char *argv[4],arg[20];
+      char *argv[2];
                                /* if input pipes usable create output pipes */
       if ((pi[0] < FD_SETSIZE) && (pi[1] < FD_SETSIZE) && (pipe (po) >= 0)) {
                                /* make sure output pipes are usable */
@@ -1214,9 +1212,8 @@
        else if (!(j = fork ())) {
          if (!fork ()) {       /* make grandchild so it's inherited by init */
                                /* prepare argument vector */
-           sprintf (arg,"%d",fd);
-           argv[0] = lockpgm; argv[1] = arg;
-           argv[2] = file; argv[3] = NIL;
+           argv[0] = lockpgm;  /* no args to lockspool for OpenBSD */
+               argv[1] = NIL;
                                /* set parent's I/O to my O/I */
            dup2 (pi[1],1); dup2 (pi[1],2); dup2 (po[0],0);
                                /* close all unnecessary descriptors */
@@ -1238,7 +1235,8 @@
          grim_pid_reap (j,NIL);/* reap child; grandchild now owned by init */
                                /* read response from locking program */
          if (select (pi[0]+1,&rfd,0,0,&tmo) &&
-             (read (pi[0],tmp,1) == 1) && (tmp[0] == '+')) {
+             (read (pi[0],tmp,1) == 1) && (tmp[0] == '1')) {
+                               /* OpenBSD lockspool uses 1 for successful lock */
                                /* success, record pipes */
            base->pipei = pi[0]; base->pipeo = po[1];
                                /* close child's side of the pipes */

First we remove the references to the other locations for the lock program. These are LOCKPGM1, LOCKPGM2, etc. These provided a way for Alpine to fall back on various locations where it might find mlock. Since we are tailoring this to OpenBSD and lockpool is part of OpenBSD and always at /usr/libexec/lockspool we don’t need this extra stuff. The variable lockpgm is already set to LOCKPGM (no number) up above in the code. This was the macro we set in that Makefile above.

Then we adjust our argv array to be just two elements because lockspool doesn’t need any arguments. It just needs the program name (lockspool) in argv[0] and NULL in argv[1]. Not sure why the rest of the code uses NIL rather than NULL, but they are the same (0). We no longer need the sprintf() because again lockspool doesn’t need arguments (it determines the current user’s name to figure out the mbox filename).

Finally we check tmp[0] == ‘1’ rather than equal to “+” like mlock used. OpenBSD locksppol outputs the 1 for a successful lock.

That’s it! Pretty simple patch. To clean this up references to the other LOCKPGM1, LOCKPGM2, etc. in the headers and Makefiles should be removed, but this has no effect on the compiled code.

Here is the link to the new patch file for ports. Replace the file:

/usr/ports/mail/alpine/patches/patch-imap_src_osdep_unix_env_unix_c

with this new one.

Now you should be able to make and make install to install the fixed Alpine. If you encounter any errors try a “make clean” and then try to build again. See the ports docs if you need help.

Testing (Important – Don’t skip)

There are a couple ways to determine that the new alpine is running, and running correctly on your OpenBSD system. First, with the new setup you should no longer see the warning about “[Folder vulnerable – directory /var/mail must have 1777 protection].” Alpine seems to quell this is you have a working locking program. Of course if you already adjusted the conf to hide this message then this check is useless.

The real way to make sure Alpine is calling lockspool is to use ktrace to trace the system calls. Don’t worry. This is much easier than it sounds.

Go to a directory where you have write permission ($HOME is good). Here we run alpine through ktrace. We give ktrace the -f argument to tell it what file to write its output to.

ktrace -f alpine_trace.out alpine

Alpine will start up normally, but ktrace will do its magic in the background logging all the system calls for us! Open your inbox. Perhaps send one email to your self and wait for delivery. Delete that test email, and expunge the mailbox. Now quit Alpine.

Now we are going to examine the trace output with kdump.

kdump -f alpine_trace.out | less

You will see all sorts of system calls in here and other info. What we are looking for is the call to lockspool. So use the slash key “/” in /usr/bin/less to search for lockspool. Just type “/lockspool” right into the less “:” prompt. This will take you to this section (hopefully).

10877 alpine   NAMI  "/usr/libexec/lockspool"
 10877 alpine   STRU  struct stat { dev=5, ino=1166436, mode=-r-sr-xr-x , 
nlink=1, uid=0<"root">, gid=7<"bin">, rdev=4668898, 
atime=1598303198<"Aug 24 16:06:38 2020">.610763358, 
mtime=1588870339<"May  7 11:52:19 2020">, 
ctime=1597953746<"Aug 20 15:02:26 2020">.413726489, 
size=10800, blocks=24, blksize=16384, flags=0x0, gen=0x0 }
 10877 alpine   RET   stat 0
 10877 alpine   CALL  kbind(0x7f7ffffdea18,24,0x7683999d577f74c4)
 10877 alpine   RET   kbind 0
 10877 alpine   CALL  pipe(0x7f7ffffdf018)
 10877 alpine   STRU  int [2] { 5, 6 }
 10877 alpine   RET   pipe 0
 10877 alpine   CALL  pipe(0x7f7ffffdf010)
 10877 alpine   STRU  int [2] { 7, 8 }
 10877 alpine   RET   pipe 0
 10877 alpine   CALL  kbind(0x7f7ffffdea18,24,0x7683999d577f74c4)
 10877 alpine   RET   kbind 0
 10877 alpine   CALL  kbind(0x7f7ffffde9e8,24,0x7683999d577f74c4)
 10877 alpine   RET   kbind 0
 10877 alpine   CALL  fork()
 10877 alpine   RET   fork 41244/0xa11c
 10877 alpine   CALL  kbind(0x7f7ffffdea18,24,0x7683999d577f74c4)
 10877 alpine   RET   kbind 0
 10877 alpine   CALL  wait4(41244,0,0<>,0)
 10877 alpine   PSIG  SIGCHLD caught handler=0xc63a1ac9e70 mask=0<>
 10877 alpine   RET   wait4 RESTART
 10877 alpine   CALL  sigreturn(0x7f7ffffde740)
 10877 alpine   RET   sigreturn JUSTRETURN
 10877 alpine   CALL  wait4(41244,0,0<>,0)
 10877 alpine   RET   wait4 41244/0xa11c
 10877 alpine   CALL  kbind(0x7f7ffffdea18,24,0x7683999d577f74c4)
 10877 alpine   RET   kbind 0
 10877 alpine   CALL  select(6,0x7f7ffffdeee0,0,0,0x7f7ffffdefe0)
 10877 alpine   RET   select -1 errno 22 Invalid argument
 10877 alpine   CALL  read(5,0x7f7ffffdeae0,0x1)
 10877 alpine   GIO   fd 5 read 1 bytes
       "1"

The key things to look for are:

<pid> alpine   NAMI  "/usr/libexec/lockspool"

You can then see the CALL to pipe and the CALL to fork as Alpine executes the child process of lockspool. Then we see the part where the Alpine parent process reads the “1” from lockspool:

<pid> alpine   GIO   fd 5 read 1 bytes
       "1"

That is what we want. We want to see that /usr/libexec/lockspool is executed, and that we read the “1” for a successful lock. If you want to be lazy you could just:

kdump -f alpine_trace.out | grep lockspool

and check for the NAMI line. That is likely sufficient to at least make sure some patch got in there.

Summary

So you never thought there would be this much to write about the admonition “[Folder vulnerable – directory /var/mail must have 1777 protection]” did you? I sure didn’t!

Actually many people have had flame wars about the mail spool permissions for years. Here we have a real solution that works to both quell the warning, but also to truly keep the mbox safe from corruption (at least via Alpine or smtpd). Simply ignoring the warning is unwise as eventually two processes will collide accessing the mbox and chaos will ensue.

Hopefully we can get these changes incorporated into ports and then nobody will have to think about it again. If you manage to submit this into ports before I do please drop me a note here to give me a credit for the changes. Email joe at this domain. I can provide details privately.

Posted in Uncategorized | Tagged , , , , , , , , , , , , , , , , | Leave a comment

Fix T_SPF_HELO_PERMERROR in Spamassassin

So I thought I had my spf (Sender Policy Framework) records setup correctly. They all passed the various test sites.

Nevertheless I kept seeing this one error in my spamassassin produced headers:

T_SPF_HELO_PERMERROR

I was unable to find a lot of info on this specific error. Everyone seemed to keep saying you needed spf records for your domain, blah, blah…

The nice thing is that this does seem to be a zero-weighted rule by default, but who’s to say some admins haven’t added negative weight to this, maybe even some big mail sites like GMail.

The key to finally figuring out the solution to this error (I mean who likes to see “PERMERROR!!!???!!!” That’s quite a permanent Error by the sound of it) was in the name itself– “HELO” When a mail admin sees hello spelled like that he or she has to think of the SMTP conversation where the mail servers starts out with a HELO greeting. (https://tools.ietf.org/html/rfc5321) Indeed when a mail server says HELO (or perhaps EHLO (sic.) for modern servers) it gives its host name.

I was able to get rid of this error message by setting up an spf record for my mail server’s hostname itself, not just for my domain.

For example, you may already have your domain spf record setup to something like this:

example.com  TXT "v=spf1 mx ip4:111.222.333.444 ip4:222.333.444.555 a:mail.example.com a:bkpmail.example.com -all"

This works great for your domain. You only send mail out of those two mail servers identified both by name and by ip. However you will still get the dreaded T_SPF_HELO_PERMERROR header from Spamassassin.

To solve this I also added an spf record for the mail server itself:

mail.example.com  TXT "v=spf1 mx ip4:111.222.333.444 ip4:222.333.444.555 a:mail.example.com a:bkpmail.example.com -all"

Note the first example is an spf TXT record for just the domain name, “example.com”. (The start of the line)

Whereas the second example is an spf TXT record for “mail.example.com”. (You would want one for the backup mail server too.)

Now there is no longer a T_SPF_HELO_PERMERROR from Spamassassin because there is no longer a permanent error on the HELO server name. Now I get the prized SPF_HELO_PASS header instead!

Will this help you at all? I have no idea. Like I said above the rule defaults to a zero weighting, but people change these weights all the time, and future version of SA may change them too. This was all tested with SA 3.4.2 .

Posted in Uncategorized | Tagged , , , , , , , , , , , | Leave a comment

Hello world!

OK. You kinda gotta have a Hello World, right?

So, yes, there was an old version of this blog a few years back. The DB got corrupted, and I have been too busy/lazy to extract it form backups and get it here. So I figured I would instead start a new version.

Posted in Uncategorized | 1 Comment

Fix “File Not Found” with PHP on OpenBSD httpd

When setting up my WordPress site on OpenBSD 6.7 using this nice tutorial I ran into an error during installation of PHP. Every time I tried to load a .php page all I got was a message saying:

File not found.

Not very informative at all! But I guess this goes along with the security consciousness of OpenBSD– an information leak could be used in nefarious ways. It turns out I had chosen a different chroot directory for my httpd web server, rather than the default /var/www. When configuring php-fpm I did not edit enough of the config file.

You see php-fpm communicates with httpd via a unix domain socket. That socket need to be accessible by the webservers AFTER is has chroot-ed into the web directory. So in my httpd.conf I have something like this:

 location "*.php*" {
            fastcgi socket "/run/php-fpm.sock"
    }

This means that httpd should send php requests to php-fpm through the socket at /run/php-fpm.sock. This path is relative to the chroot directory of the web server. So I needed to edit my /etc/php-fpm.conf file to change the chroot directive to point to MY custom chroot directory for my httpd server, rather than to the default one:

<snip other config stuff>
...
; Chroot to this directory at the start. This value must be defined as an
; absolute path. When this value is not set, chroot is not used.
; Note: you can prefix with '$prefix' to chroot to the pool prefix or one
; of its subdirectories. If the pool prefix is not set, the global prefix
; will be used instead.
; Note: chrooting is a great security feature and should be used whenever
;       possible. However, all PHP paths will be relative to the chroot
;       (error_log, sessions.save_path, ...).
; Default Value: not set
chroot = /<my>/<chroot>/<path>

Now I could get php to work and a simple:

 <?php phpinfo(); ?>

page would work just fine! For the record this was PHP-7.4.x with php-frm on OpenBSD 6.7 using the build in httpd.

Posted in Uncategorized | Tagged , , , , , , , | Leave a comment

Fix: Fatal error: Uncaught Error: Call to undefined function mysql_connect() in …

When setting up my WordPress site on OpenBSD 6.7 using this nice tutorial I ran into an error during installation. I enabled Debug mode in WordPress by setting WP_DEBUG to true in wp-config.php

define( 'WP_DEBUG', true );

This allowed me to to see the real errors from PHP, etc. I then was handed the following error:

Fatal error: Uncaught Error: Call to undefined function mysql_connect() in … <path>

I was able to resolve this issue by enabling the mysqli plugin in /etc/php-7.4.ini.

Just remove the semi-colon in front of the line:

;extension=mysqli

So after removing that semicolon in your favorite text editor:

# grep ^extension=mysqli /etc/php-7.4.ini
extension=mysqli
#

After that was smooth sailing. This was with PHP-7.4.x on OpenBSD.

Posted in Uncategorized | Tagged , , , , , | Leave a comment

Fix Let’s Encrypt 403 Errors in OpenBSD 6.7 with acme-client

This is not another tutorial on how to use acme-client or how to setup SSL/TLS with httpd in OpenBSD. There are a number of good ones out there already. This post is about a particular issue during the I was trying to setup Let’s Encrypt certificates in OpenBSD using the built-in acme-client tool. I was able to run the command:

# acme-client -v example.org

I kept getting errors from acme-client about a bad return status, and no certificates were generated. When I clicked the link to Let’s Encrypt’s website that was returned by acme-client I would see 403 Forbidden errors in the message. The weird thing (that I have not figured out) is that the certificates were working fine with the Let’s Encrypt Staging server. However, they would fail when I tried to use the production server to get a real cert.

All my permissions were setup correctly for the directories. It turns out my umask for root was set too restrictive. It was 077, which creates files with no group or other read permissions.

After running:

# umask 022

I was able to run acme-client successfully.

So watch your umask when installing any sort of software, or whenever you encounter some sort of permissions error. I personally think the acme-client should automatically set the challenge files to world readable. Perhaps I should submit a patch.

Posted in Uncategorized | Tagged , , , , , , , , , , | Leave a comment