Web Site Documentation Update

[matt]

The documentation on the web site has undergone a major overhaul. Please spend a little time browsing through the pages and post some feedback.

Thanks!

[LogicX]

This was supposed to be a docfix but turned into a running commentary, with doc fixes, questions, and comments throughout an entire toaster install.

Random notes:
I'd like to use one of the ports tools, and created a published document for the toaster project outlining the implications of each install option in the config --
If we had a tree that shows 'if you chose to install this, then it will install these dependent programs, which will install these dependent programs, etc.' -
maybe even make it a nice cgi script where you can check off which things you chose to install, and it'll generate the tree for you.

That way anyone can know at a glance what the heck installed port xyz, and depends upon it, etc.

As I believe was recently pointed out in a post to the mailing list (by Justin Kulikowski jpk236 [at] jpk236.com)
I think the toaster-watcher.pl needs to reference the install lines in the toaster conf to determine which programs it reports errors on them not running --


http://www.tnpi.biz/internet/mail/toaster/intro/history.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/intro/history.shtm l

Due to the issues were were seeing with sendmail, in late 2000 I needed a mail solution capable of hosting a 100,000 email accounts.

typo 'we were' -- @

I did a bunch of testing and benchmarking and then build a clustered qmail system

s/build/built/

http://www.tnpi.biz/internet/mail/toaster/intro/features.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/intro/features.sht ml

Security - Mail::Toaster supports secure connections from the email client (via POP3, IMAP, SMTP, and webmail) to the server. If the remote (destination) email server supports it, we also also encrypt the email as it travels across the public intenet from server to server.

also is repeated twice, s/intenet/Internet/

The web forums are also frequented by a quite a few helpful folks.

'by a quite' to 'by quite'

http://www.tnpi.biz/internet/mail/toaster/intro/platforms.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/intro/platforms.sh tml

Suggestion: on the platform page make a note that the toaster has classically worked on i386 architecture, but as FreeBSD now supports amd64 as a tier1 supported platform (verbage?); there are users known to use the Mail Toaster on 64bit systems w/o issue

(I run a few)

http://www.tnpi.biz/internet/mail/toaster/intro/contribute.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/intro/contribute.s html

I'd enjoy seeing some clarification on 'submit it'
I know I have emailed you in the past asking exactly how you'd like changes/fixes -- If you could write a little paragraph detailing your preferred way to recieve fixes, I think we'd all enjoy complying with your preference.

http://www.tnpi.biz/internet/mail/toaster/install/toaster.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/install/toaster.sh tml

I'd hope anyone installing the toaster knows how to extract an archive; but its definitely missing from the steps:

"Download the latest released Mail::Toaster.

Once downloaded, install it much like you would any other perl module.

   * perl Makefile.PL"


I've never enjoyed the fact that at this stage:
# make newconf (overwrites installed .conf files)
# make cgi

We have no forewarning that make cgi actually installs stuff all over /usr/local/www

I'd prefer some forewarning, or perhaps doing it AFTER the step where we configure that path in the toaster-watcher.conf

toaster_setup.pl -s ports
should be made to adhered to make.conf settings, if set, such as:
SUP_UPDATE=     yes

SUP=            /usr/local/bin/cvsup
SUPFLAGS=       -g -L 2
SUPHOST=        cvsup17.FreeBSD.org
SUPFILE=        /etc/cvsup/cvsupfile-src
PORTSSUPFILE=   /etc/cvsup/cvsupfile-ports

following these settings would override the cvsup* variables in toaster-watcher.conf (hmm?)


re: toaster.conf

I'd like to see a one-liner/header above the rbl list that explains the format of those entries (that they're rbl_any_valid_host
and that you can add/remove from the list at will)
I understand this is explained in other documentation --I'd just like to see a quick one-liner in the config explaining it also

OPENSSL:
I'd like to see an option where in the config you can specify a path to the (hopefully chmod 400'd) SSL cert files --
so I could put in the config the path to the SSL cert files I've gotten when purchasing the SSL cert --
then throughout various install steps, it will take those files, run the openssl commands on them to make appropriate files (pem)
and install them into the appropriate locations, with the appropriate perms (400 again?), owned by the proper users (root?)
--- instead of prompting for the info to generate a cert.

re: pkgtools.conf

php4 entries should be updated to reflect the new php4/php4-extensions style
also, appropriate php5 entries should be added


http://www.tnpi.biz/internet/mail/toaster/install/programs.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/install/programs.s html

# rehash
# toaster_setup.pl -s pre
# toaster_setup.pl -s ports

Should we have them redo the -s ports again (they were told in a previous step to update their ports)
remove it? put a note that you don't have to do it again if they just did it?

toaster_setup.pl -s mysql

I never before realized that this did more than just install mysql (installs DBI, DBD::mysql) -- I've ALWAYS skipped this, as I already have mysql installed.  Anyway to make a note of that for users such as myself?

same goes for toaster_setup.pl -s apache -- never had any idea it installed cool stuff

Perhaps a reassurance somewhere that the scripts will SKIP and NOT touch things you already have installed (apache, mysql)

or maybe I just need to read and understand this statement:
"If you want something installed "your" way, install it yourself, making sure it's registered in the package database. toaster_setup.pl will detect it and skip that portion of the install. The toaster build scripts make many tweaks, so make sure to read the perl modules to see what (if anything) special we do when installing that program. Then you won't be surprised when a toaster advertised feature doesn't work."


why did ezmlm try to install?

toaster_setup.pl -s ezmlm
ports_check_age: Ports file is current (enough).
===>  Vulnerability check disabled, database not found
=> ezmlm-idx-0.40.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch from ftp://ftp.ezmlm.org/pub/patches/." target="_blank">ftp://ftp.ezmlm.org/pub/patches/.
fetch: ftp://ftp.ezmlm.org/pub/patches/ezmlm-idx-0.40.tar.gz:" target="_blank">ftp://ftp.ezmlm.org/pub/patches/ezmlm-idx-0.40.tar.gz: No address record
=> Attempting to fetch from ftp://gd.tuwien.ac.at/infosys/mail/qmail/ezmlm-patches/." target="_blank">ftp://gd.tuwien.ac.at/infosys/mail/qmail/ezmlm-patches/.
ezmlm-idx-0.40.tar.gz                           3% of  540 kB 7776  Bps^C
fetch: transfer interrupted
syscmd: make  -DWITH_MYSQL  install
child died with signal 2, without coredump
syscmd: make  -DWITH_MYSQL  install
syscmd: result: 2
===>  Cleaning for perl-5.8.6_2
===>  Cleaning for qmail-1.03_3
===>  Cleaning for ezmlm-idx-0.40_3
port_install: ezmlm-idx            FAILED
FATAL FAILURE: Install of ezmlm-idx failed. Please fix and try again.
at /usr/local/lib/perl5/site_perl/5.8.6/Mail/Toaster/Setup.pm line 833
root:step|5:33:30pm|/usr/local/etc:
root:step|5:33:31pm|/usr/local/etc: grep -i ezmlm /usr/local/etc/toaster*
/usr/local/etc/toaster-watcher.conf:install_ezmlm                  = 0   # 0, ver
/usr/local/etc/toaster-watcher.conf:install_ezmlm_cgi              = 0      # 0, 1



why did qmailadmin try to install?

root:step|5:38:08pm|/usr/local/etc: toaster_setup.pl -s qmailadmin
domain autofill: yes
modify spam: yes
modify quotas: yes
install_from_source: looking for existing sources...not found.
qmailadmin-1.2.0.tar.gz                       100% of  315 kB  363 kBps
install_from_source: no patches to fetch.
archive_expand: decompressing qmailadmin-1.2.0.tar.gz....^Csyscmd: /usr/bin/gunzip -c qmailadmin-1.2.0.tar.gz | /usr/bin/tar -xf -
child died with signal 2, without coredump
syscmd: /usr/bin/gunzip -c qmailadmin-1.2.0.tar.gz | /usr/bin/tar -xf -
syscmd: result: 2
archive_expand: FAILURE expanding
Couldn't expand qmailadmin-1.2.0.tar.gz: No such file or directory
at /usr/local/lib/perl5/site_perl/5.8.6/Mail/Toaster/Setup.pm line 2345
root:step|5:38:47pm|/usr/local/etc: grep -i qmailadmin /usr/local/etc/toaster*
/usr/local/etc/toaster-watcher.conf:install_qmailadmin             = 0  # 0, ver, port (1.2.0)



http://www.tnpi.biz/internet/mail/toaster/install/qmail.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/install/qmail.shtm l

I really really really appreciate the trivia note!!!
Something I've wondered about for ages!

http://www.tnpi.biz/internet/mail/toaster/install/filtering.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/install/filtering. shtml

The following note gets lost in the toaster_setup.pl -s filtering install:
It's a good idea to make sure an entry like this exists in /etc/newsyslog.conf:

/var/log/mail/maildrop.log  vpopmail:vchkpw   644  3     1000 *     Z

Is that in documentation for the toaster somewhere? Could we make a note of it somewhere?


http://www.tnpi.biz/internet/mail/toaster/install/logging.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/install/logging.sh tml

I always get frightened when I'm installing the toaster_setup.pl -s services and then it magically starts running things

Could we perhaps instead not start processes (at this stage, and after the apache install)
and instead have a toaster_setup.pl start/stop which references rc.conf variables with toaster installed variables, and starts/stops the appropriate services?

And then just add this 'start' step in at the end of the toaster setup, before testing occurs

http://www.tnpi.biz/internet/mail/toaster/install/cron.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/install/cron.shtml

a good example of my earlier mentioned failure notifications:

/usr/local/sbin/toaster-watcher.pl
       imapd-ssl is not running, FAILED!
       imapd is not running, FAILED!
       pop3d-ssl is not running, FAILED!
       httpd is not running, FAILED!



Ran out of time -- I'll continue later, and potentially post more

[matt]

LogicX wrote on Fri, 29 April 2005 06:03

Random notes: I'd like to use one of the ports tools, and created a published document for the toaster project outlining the implications of each install option in the config -- If we had a tree that shows 'if you chose to install this, then it will install these dependent programs, which will install these dependent programs, etc.' - maybe even make it a nice cgi script where you can check off which things you chose to install, and it'll generate the tree for you.

This will become more feasible when toaster_setup.cgi arrives.

Quote:

As I believe was recently pointed out in a post to the mailing list (by Justin Kulikowski jpk236 [at] jpk236.com) I think the toaster-watcher.pl needs to reference the install lines in the toaster conf to determine which programs it reports errors on them not running --

Already fixed in 4.07

Quote:

http://www.tnpi.biz/internet/mail/toaster/intro/platforms.shtml" target="_blank">  http://www.tnpi.biz/internet/mail/toaster/intro/platforms.sh tml Suggestion: on the platform page make a note that <snip> FreeBSD now supports amd64 as a tier1 supported platform <snip> (I run a few)

done.

Quote:

http://www.tnpi.biz/internet/mail/toaster/intro/contribute.shtml" target="_blank">  http://www.tnpi.biz/internet/mail/toaster/intro/contribute.s html I'd enjoy seeing some clarification on 'submit it'

Done.

Quote:

http://www.tnpi.biz/internet/mail/toaster/install/toaster.shtml" target="_blank">  http://www.tnpi.biz/internet/mail/toaster/install/toaster.sh tml I'd hope anyone installing the toaster knows how to extract an archive; but its definitely missing from the steps:

Done.

Quote:

I've never enjoyed the fact that at this stage: # make newconf (overwrites installed .conf files) # make cgi We have no forewarning that make cgi actually installs stuff all over /usr/local/www

You know how to deal with that. Besides, it's a Makefile target, and is determined thusly:

my $dir = "/usr/local/www/data";
if  ( -d "/usr/local/www/data/mail" ) { $dir = "/usr/local/www/data/mail"     }  # FreeBSD
elsif ( -d "/usr/local/www/mail"    ) { $dir = "/usr/local/www/mail"          }
elsif ( -d "/Library/Webserver/Documents" ) { $dir = "/Library/Webserver/Documents" }  # Mac OS X
elsif ( -d "/var/www/html"       ) { $dir = "/var/www/html"                }; # Linux
return $dir;

Quote:

toaster_setup.pl -s ports should be made to adhered to make.conf settings,<snip> following these settings would override the cvsup* variables in toaster-watcher.conf (hmm?)

Code submissions are welcome.

Quote:

re: toaster.conf I'd like to see a one-liner/header above the rbl list that explains

Documentation submissions are welcome.

Quote:

OPENSSL: I'd like to see an option where in the config you can specify a path to the (hopefully chmod 400'd) SSL cert files <snip> --- instead of prompting for the info to generate a cert.

Excellent idea. Code submissions are welcome.

Quote:

re: pkgtools.conf php4 entries should be updated to reflect the new php4/php4-extensions style also, appropriate php5 entries should be added

Patches are welcome.

Quote:

http://www.tnpi.biz/internet/mail/toaster/install/programs.shtml" target="_blank">  http://www.tnpi.biz/internet/mail/toaster/install/programs.s html # rehash # toaster_setup.pl -s pre # toaster_setup.pl -s ports Should we have them redo the -s ports again

removed.

Quote:

toaster_setup.pl -s mysql I never before realized that this did more than just install mysql (installs DBI, DBD::mysql) -- I've ALWAYS skipped this, as I already have mysql installed.  Anyway to make a note of that for users such as myself?

Note added to MySQL info page.

Quote:

same goes for toaster_setup.pl -s apache -- never had any idea it installed cool stuff

maybe you didn't notice, on the programs page, several of the programs are links. Click them.

Quote:

why did ezmlm try to install? toaster_setup.pl -s ezmlm <snip>

Because you told it to? (duh)

Quote:

why did qmailadmin try to install? root: toaster_setup.pl -s qmailadmin <snip>

same reason?

Quote:

The following note gets lost in the toaster_setup.pl -s filtering install: It's a good idea to make sure an entry like this exists in /etc/newsyslog.conf: /var/log/mail/maildrop.log  vpopmail:vchkpw   644  3     1000 *     Z Is that in documentation for the toaster somewhere? Could we make a note of it somewhere?

I've done better, it now gets added automatically (if it doesn't exist).

Quote:

http://www.tnpi.biz/internet/mail/toaster/install/logging.shtml" target="_blank">  http://www.tnpi.biz/internet/mail/toaster/install/logging.sh tml I always get frightened when I'm installing the toaster_setup.pl -s services and then it magically starts running things Could we perhaps instead not start processes (at this stage, and after the apache install) and instead have a toaster_setup.pl start/stop which references rc.conf variables with toaster installed variables, and starts/stops the appropriate services? And then just add this 'start' step in at the end of the toaster setup, before testing occurs

This idea has merit...but creates other problems. For example, it can't very well do many of the installation parts without starting up MySQL. Testing RRDutil is pointless if snmpd wasn't started.

Maybe a couple additional targets for toaster_setup.pl like "start & stop" that will go through and start or stop every toaster component?

Matt

[LogicX]

http://www.tnpi.biz/internet/mail/toaster/config/watcher.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/config/watcher.sht ml

   * install_mysql = 4 # 0, 1, 2, 3, 4, 5

You can choose from a variety of version of MySQL to install. The code meanings are as follows:

   * 0 - none
   * 1 - install latest package
   * 2 - install latest stable release from ports
   * 3, 323 - 3.23 from ports
   * 4, 40 - 4.0.x from ports
   * 41, 4.1 - 4.1.x from ports
   * 5, 50 - 5.0.x from ports


I'd like to see the config line comment like this:

#0, 1, 2, 3, 40, 41, 5

I think there's enough distinction between 40 and 41 to justify them being listed separately there.


The reference here should link to this: http://www.tnpi.biz/internet/mail/toaster/faq/programs/mysql.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/faq/programs/mysql .shtml :
Set install_mysql to the major version of MySQL server that you have or desire to have installed.

   * install_mysql_linuxthreads = 0

mysql_linuxthreads is considered experimental on FreeBSD although some folks reccomend it.



install_vqadmin                = port      # 0, ver

can be updated to list ,port also


   * qmail_chk_usr_patch = 1

This decides whether to install the chkusr patch as described at http://www.interazioni.it/qmail/" target="_blank">http://www.interazioni.it/qmail/ . Please note that this patch has various implications for how mail is handled on your system-- make sure you really want the functionality it offers.

I believe this was deprecated in a recent mail toaster version, and no longer appears in the configs; should it be removed from the docs?

[LogicX]

Quote from matt:

Quote: toaster_setup.pl -s ports should be made to adhered to make.conf settings,<snip> following these settings would override the cvsup* variables in toaster-watcher.conf (hmm?) Code submissions are welcome.

I'll look over the code that uses those variables and see what I can come up with

Quote from matt:

Quote: re: toaster.conf I'd like to see a one-liner/header above the rbl list that explains Documentation submissions are welcome.

Something like this will do:

#######################################
#      RBLs (spam blacklists)         #
#######################################

rbl_enable                      = 1    # master RBL switch. Disables all RBLs
rbl_enable_fail_closed          = 1    # default is on
rbl_enable_soft_failure         = 1    # default is on (off means bounce immediately (553)
rbl_timeout                     = 60   # default is 60 seconds
rbl_reverse_dns                 = 1    # block based on presence of reverse DNS
rbl_reverse_dns_failure         = soft # soft | hard  (temporary (451) or permanent (553) error)
                                      # currently the only way to block based on DNS is modifying
                                      # your ~vpopmail/etc/tcp.smtp file. See the FAQ for details
                                      # Eventually toaster-watcher will be able to set that up

#This is only a recommended list of RBLs. The following variables are dynamic in nature. You may add or remove from it, and they will automatically be used.
For example, to use the combined SBL-XBL list published by spamhaus, just add "rbl_sbl-xbl.spamhaus.org = 1" and it will be recognized by toaster-watcher.

(stole that last part right out of the perldoc toaster-watcher.conf)

Quote from matt:

Quote: OPENSSL: I'd like to see an option where in the config you can specify a path to the (hopefully chmod 400'd) SSL cert files <snip> --- instead of prompting for the info to generate a cert. Excellent idea. Code submissions are welcome.

Its been a while since I've had to do this -- let me look things over and I'll get back to you with this.

Quote from matt:

Quote: re: pkgtools.conf php4 entries should be updated to reflect the new php4/php4-extensions style also, appropriate php5 entries should be added Patches are welcome.

the new formatted lines would be:
'lang/php*' => 'WITH_APACHE2=1 WITHOUT_DEBUG=1 WITHOUT_IPV6=1',
'lang/php*-extensions' => 'WITH_IMAP=1 WITH_SNMP=1 WITH_GD=1 WITH_OPENSSL=1';

Those would be the updated lines; however I question if we actually need all those options in there
Depends what way you want to look at this for users:
Way 1: Now that you have the supporting binaries for these options installed, you should also install their php extensions so if you ever one day have a script that needs them, it can use them
Way 2: Although you have the supporting binaries, you likely don't have any php scripts which implement said options, so we won't install them by default, especially now that you can just install that particular extension as a port.

I also feel WITH_ZLIB=1 should be in there

There are of course other problems with making generalizations here --
If the user has install courier off, then why should we include WITH_IMAP, or if not installing the openssl port, etc. etc.

one approach could be to modify this file to reflect options set in toaster_watcher --

ugh, so much complication for something so simple

Quote from matt:

Quote: why did ezmlm try to install? toaster_setup.pl -s ezmlm <snip> Because you told it to? (duh)

I feel this behavior is contrary to most other toaster_setup.pl options -- allow me to demonstrate:

toaster_setup.pl -s phpmyadmin
phpMyAdmin install disabled. Set install_phpmyadmin in toaster-watcher.conf and try again.

/usr/local/sbin/toaster_setup.pl script execution complete.


So why does that one (and many of the others)abide by the setting, but ezmlm and qmailadmin do not?

[LogicX]

http://www.tnpi.biz/internet/mail/toaster/install/programs.shtml" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/install/programs.s html

toaster_setup.pl -s vpopmail

Should that step perhaps have a link, or somewhere else -- about the fact that if you already had mysql installed -- and had changed  your root password; then vpopmail will fail to create the vpopmail table, results in error messages similar to:



/usr/local/vpopmail/bin/clearopensmtp
vmysql: error creating table 'relay': MySQL server has gone away
vmysql: sql error[8]: MySQL server has gone away


later in the config when you are told to test the cron scripts.

I was later able to remove the pass; re-run toaster_setup.pl -s vpopmail and get this result:

DBD::mysql::st execute failed: Unknown database 'vpopmail' at /usr/local/lib/perl5/site_perl/5.8.6/Mail/Toaster/Mysql.pm line 699.
couldn't execute: DBI::st=HASH(0x1a8fe10)->errstr
at /usr/local/lib/perl5/site_perl/5.8.6/Mail/Toaster/Setup.pm line 5517
vpopmail: oops, no vpopmail database.
vpopmail: creating MySQL vpopmail database.

where it created the database, tables, and user w/o issue.

Perhaps an option where if it fails, then it prompts for a root user/pass so it can go in and do the mods (similar to the nictool server setup script)

[matt]

How about this:

Quote:

MySQL Passord Note There are several places where the setup scripts will connect to the MySQL database and create tables and set up permissions for applications to connect (vpopmail, squirrelmail, spamassassin, phpMyAdmin, etc).  If you have configured a root password for MySQL, then these connections will fail and your database won't be set up. When this happens, the queries are printed out in your terminal and you must run them manually. There are several ways to deal with this. The first is to simply wait until after you've built your Mail::Toaster to set the root password. The next, and probably best solution is to store your MySQL root password in ~/.my.cnf. Configure the user and pass in the [mysql] section, exactly as you would for the mysql client. For security, make sure the file is chmod 700!

Of course, your idea to prompt for the password is also good, and you're welcome to submit a patch that does that.  If you do write such a patch, then I'd remove the MySQL note from the web page and drop it into the source code as a warning. Then the only folks that see it are the ones it affects.