Drop me a message !
english french

SME FetchMail

Type How To
Version 1.0.11
Tested on SME 6.0, 6.0.1, 6.5RC1, 7.0, 7.1, 7.2, 7.3 and 7.3 utf-8

Subject
Installation
Configuration
Configuration tuning
Notes for peoples upgrading from sme-fetchmail
About smeserver-fetchmail and SME starting at version 7.1
About smeserver-fetchmail and SME 7.3 with update from April 2008
Wish List : all ideas to improve Fetchmail
Contrib authors
Change Log

Subject

SME is a very nice mail server, but we sometime need external mailboxes. In this case, retrieving mails from theses mailboxes can be a lot of work. Moreover, if you configure a lot of P.C. on your network to retrieve a bunch of externals mailboxes, the configuration of each computer becomes complicated.
It could be easier (and maybe safer) to let our SME server drop all mails and store them in each user box(es), allowing a centralized and standardized mail configuration

Here comes fetchmail in the process. Fetchmail is intended just to do that : at regular time, retrieve a lot of externals mailboxes, and throw the contents to internals mailboxes.
You should note, however, that fetchmail only retrieve mail from PO3 an IMAP4 server. You cannot use it to retrieve mail from a webmail server.

Installation

Fetchmail itself is already installed on your SME Server : it is used by the ETRN and multidrop mode of the e-mail configuration (See the SME documentation for more information)

Here we speak about an add-on for SME, who is intended to retrieve several externals mailboxes for several internals accounts


To my knowledge, there are two versions of fetchmail for SME.

prerequisites

To install smeserver-fetchmail, you need :

While it's easy to know your version of SME, it can be a little bit harder to check the version of FormMagick. Two solutions :

'Academic' way

rpm -q perl-CGI-FormMagick
perl-CGI-FormMagick-0.89-08

If your level is below this one, you need to upgrade FormMagick.

'Funny' way

Just start the installation of smeserver-fetchmail. If your system don't reach the FormMagick desired level, it will tell you !

FormMagick Upgrade

FormMagick-0.89-08 is available from all majors SME mirrors, but is still (in June 2004) in the 'devel' subdir.
You can install it directly from my personal site, like this :

rpm -Uvh http://www.schirrms.net/files/perl-CGI-FormMagick-0.89-08.noarch.rpm
Retrieving http://www.schirrms.net/files/perl-CGI-FormMagick-0.89-08.noarch.rpm
Preparing...                ########################################### [100%]
   1:perl-CGI-FormMagick    ########################################### [100%]

smeserver-fetchmail RPM installation

There's now three versions of smeserver-fetchmail : one for SME up to 7.0, one for SME beetween version 7.1.1 and 7.3, and one for version starting at 7.3 with the upgrade form April 2008. You can learn why here for the second version and For the 'Spring 2008' version. The explanations are for the newest version, if you're on a older SME version, just replace the version by 1.3.4-10 or smeserver-fetchmail-1.3.5-02.

Installation is easy. just one RPM to roll :

rpm -Uvh http://www.schirrms.net/files/smeserver-fetchmail-utf8-1.3.5-02.noarch.rpm
Retrieving http://www.schirrms.net/files/smeserver-fetchmail-1.3.5-02.noarch.rpm
Preparing...                ########################################### [100%]
   1:smeserver-fetchmail-utf########################################### [100%]
Rebuilding Web Server Manager Left Panel Cache ... Can take up to a minute.
Done.
Setting defaults values in SME configuration database, if needed. Don't change any existing configuration.
Migrating existing database domains
Migrating existing database spamassassin
Migrating existing database configuration
Migrating existing database hosts
Migrating existing database yum_repositories
Migrating existing database yum_available
Migrating existing database backups
Migrating existing database networks
Migrating existing database mailpatterns
Migrating existing database accounts
Migrating existing database yum_updates
Migrating existing database yum_installed
Done.
Installation finished.

For your information, the source RPM (totally unneeded for installation) is also available

In case of an upgrade

Normally, if you upgrade from sme-fetchmail-1.0.2, everything should be OK. At least, we did numerous tests for that !

As a complementary safety measure, you can backup these two files, who contains the whole fetchmail configuration :
/home/e-smith/configuration or /home/e-smith/db/configuration
/home/e-smith/accounts or /home/e-smith/db/accounts

But be careful ! sme-fetchmail contains a multidrop feature. As this is already embedded in SME, smeserver-fetchmail no more contains this feature. If you did multidrop with sme-fetchmail, you will have to double-check the whole multidrop feature !
During rpm installation, sme-fetchmail is automagically uninstalled.

Two points are to change manually :

Configuration

In the server manager, part 'Collaboration' you should find a 'Mails Retrieval' option.

This let you access on the main page of the configuration.

Main screen

You can see a screenshot here.

Here, you can set some generals options :

Enable/Disable Fetchmail Just stop or start Fetchmail, without any change to the configuration. The following filters are still working !
Enable/Disable External POP and IMAP server Access If you disable that, your internals stations cannot more access externals POP3 and/or IMAP4 servers. Only the SME server can do that. In others words : Fetchmail is mandatory.
Enable/Disable access to external News servers If you select to disable access to news (NNTP) servers, all stations on your LAN loose access to any external News servers. To use if you have a local reflector or if you think that news are not needed.
Enable/Disable access to external SMTP servers If you disable access to external SMTP servers, all stations on your LAN need to use the SME SMTP services to send E-Mail. Note that this has only effect in conjunction with the next parameter.
Use the SME SMTP Proxy (yes/no) SME SMTP Proxy is a new feature of SME 6. From the SME documentation :
- A transparent SMTP proxy has been added, to force all outgoing
SMTP connections to go through the SME server. This feature
enhances security with respect to viruses and worms that use SMTP
to propagate themselves.
The SME SMTP Option is enable by default. This means that every outgoing SMTP connection is in fact 'catched' by SME, and only the SME server does connection to external SMTP server. In most cases, this option is wise. If you let this option on (green light ;-) ), the the upper option has no effect (in fact, stations on your LAN can only connect to SME). If you choose (or need) to deactivate the SME SMTP Proxy, then the upper option becomes valid.

Note : you cannot change the time schedule of fetchmail here. This is because fetchmail is still sharing some parts with the standard SME Fetchmail system.
If the standard schedule (every 15 minutes) doesn't suit your needs, in server manager, go on Configuration -> E-Mail and choose new schedule rules (at the bottom of the page).
But there is a trick : If your server E-mail retrieval mode (at the top of the same page) is set as Standard (SMTP), you cannot save the change !
In this case just select ETRN (SMTP with client request), do your schedule change, and click 'Save'.
Then go again in the page, select back Standard (SMTP), and save again.

You can also select any SME user, to create a new fetchmail ruleset, or to modify or remove the existing ruleset.

User screen 1

Then you go to the first user page (screenshot)
This screen let you set recipients rules for the mail retrieved by fetchmail. In most case, you wont change anything here.

Here, four options :

Is it a 'ghost' account (only to redistribute mails) ? If you choose yes, this account will search externals mail, but will only distribute them on another internals mailboxes and/or externals mails address.
Do you want to debug this account (fetchmail actions are logged in /var/log/maillog) ? Really only a debug option, as it is very verbose !
What do you want to do with E-Mails on externals servers ? Three options :
Retrieve all mails :alls mails still on server are downloaded, and the mail are suppressed on the mail server.
Read mails and let them on the server :news mails are downloaded, alls mails stay on the mail server. Note: depending of the external mail server, the same mail could arrive again at each new fetchmail run (by default, every 15 minutes)
Retrieve news mails only :a mail already tagged as 'read' is not downloaded.
Email address(es) to witch forward mails ( format : xxx@yyy.zz or aaa@bbb.cc : fff.ggg@hhh.jjjj.kk ) If you want that every mail is send to an external e-mail, just set it here.
You can set here more than one external e-mail address. Just set a colon between each email addresses.

Just below, you have a list of internals accounts. Just check the corresponding case to have all incoming mails send also to this(these) account(s).

You can use any combination of local copy (that is "ghost account = no") copy to internal(s) user(s) and copy to an external mailbox. But you need at least one of them, and fetchmail will tell you if you forgot it !

User screen 2

In this screen, you set the information for the mailboxes to fetch. A screenshot is here.

You can retrieve many mailboxes for a single account (I think there is no limit, but if you find one, let me know)

You can create, modify and delete externals mailboxes configuration.
For each external mailbox to fetch, you must give four informations :
Name of the External Mail Server Internet style name, like imap.myisp.net or pop3.mailservice.com
Protocol to use when reading mails Starting with version 1.3.5, four choices : POP3, IMAP4, POP3 via SSL (experimental) and IMAP4 via SSL (very experimental !)
The user account name on the mail server This is the login for the account on the mail system.
The user account password on the mail server This is the associated password to gain access on the mail system. Please note : the password can be seen clearly on this screen and is almost so clear to read in the config file. This file can only be read by a system administrator. If this is an issue for you, you should maybe try another way. But remember : Using POP3 or IMAP4, your password are traveling totally clearly on Internet.

And finally, you'll find a [Validate changes] button. Don't forget it ! Your user modifications are only stored when you click on it !

Configuration tuning

smeserver-fetchmail-1.3.4-06 arrives with some tiny configuration changes, for the mail redistribution logic.
fechtmail retrieves mails using POP3 or IMAP4 protocol, following your configuration choices, and then deliver these mails by using the SMTP protocol.
Older smeserver-fetchmail version ever deliver the mail to the local SMTP server.
Now, the logic is the following :

  1. If the configuration database field 'fetchhost' exists, then the filed value is the destination smtp server. (see later explanation about the 'fetchhost' field)
  2. If not, and if in the Server-manager->Configuration->Email, the field 'Address of internal mail server' has been filled, then this address is used as the destination mail Server. This should permit to use a SME server as a Anti-Spam/Anti Virus mail frontend also in collaboration with fetchmail.
  3. Last, if none of these fields have been filled, then the mail is send at 'SME domain name' witch is for SME another name for itself.

the 'fetchhost' field

In some cases, smeserver-fetchmail cannot guess alone how to deliver the retrieved mail. So I have added a configuration field that you can use to set a destination server (the server who own the final mailboxes).Probably, in the future, you'll be able to feed this field from the server-manager fetchmail panel. But for now, if you need this configuration change, you can do it so :

# to see the current 'fetchhost' value :
/sbin/e-smith/db configuration getprop FetchMails fetchhost

# to set a value (no validity chek, be careful)
/sbin/e-smith/db configuration setprop FetchMails fetchhost smtp.from.mycorp.com

# check the new value
/sbin/e-smith/db configuration getprop FetchMails fetchhost
smtp.from.mycorp.com

# delete the value
/sbin/e-smith/db configuration delprop FetchMails fetchhost

#IMPORTANT ! activate the changes !
/sbin/e-smith/expand-template /etc/fetchmail
As the /etc/fetchmail file is reread at each fetchmail collect, you don't need to restart any services to activate the change.

the 'timeout' field

Starting with version 1.3.5, a 'timeout' parameter has been added. Before, if the remote server didn't answer to a fetchmail query,fetchmail did stop after a timeout hardcoded at compilation time ! Now the timeout is by default of 60 seconds. This parameter is the same for all fetchmail accounts. If this 60 seconds value doesn't suit your needs, you can change it :

# to see the current 'timeout' value :
/sbin/e-smith/db configuration getprop FetchMails timeout
60

# to set a value (no validity chek, be careful) expressed in seconds
/sbin/e-smith/db configuration setprop FetchMails timeout 30

# check the new value
/sbin/e-smith/db configuration getprop FetchMails timeout
30

#IMPORTANT ! activate the changes !
/sbin/e-smith/expand-template /etc/fetchmail

Notes for peoples upgrading from sme-fetchmail

Here again, for most cases, upgrade should work fine. However, even if the general 'look & feel' is fairly the same a lot of underlying work have been done. So you could have smalls changes during opérations.

About smeserver-fetchmail and SME starting at version 7.1

SME 7.0 was the first SME version with an embedded spamfilter and antivirus program (and that's great). But on alot of small systems, the time to accept a mail was toooo long. The SME Core Team choose to suppress the spam filter for mail coming from the LAN. No luck, for SME, mails coming via fetchmails were seen as local. On SME 7.1, I didn't see a way to go trough this change and incoming mails weren't checked. Lucky me, the trouble was the same for mails coming via ETRN or Multidrop, wich are standard functions of SME. So the Core Team put a solution in SME starting at version 7.1.1. I 'just' had to create a new version of smeserver-fetchmail using this new way for incoming mail.

It would probably be a way to create a smeserver-fetchmail package working on all SME versions, but the overwork it would add was to much for me. So we have two versions in the wild : 1.3.4-10 for system up to SME 7.0 and the current version (1.3.5-00 at the time of writing) for SME from 7.1.1.

And what about SME 7.1.0 ? I have to say that this version is not officially supported. Of course, you can install smeserver-fetchmail-1.3.4-10 on it, but this will work without antispam filter, witch seems a strange idea today

This is not only bad, in fact : the new version of smeserver-fetchmail can use new features of fetchmail. So I was able to add support for POP3 and IMAP via SSL ("About time" would you say :-) )

About smeserver-fetchmail and SME 7.3 with the April 2008 Update

Starting with the update of April 2008, SME did a major change (even if hard to see) : the system charset went from ISO-8859-1 to UTF-8 ! What does it mean ? Computers store any information as a succession of bits (that is O or 1). This is not really 'user friendly' :-) To be able to store text in a computer file, we need to associate bit succession with human character. So, in most small systems the letter A is equal to 65 (or 0100 0001) and the number one is aqual to 49 (or 0011 0001). For non american character, there are many coding tables. A fairly recent coding table UTF-8, is for now the most commonly accepted table. This table permit to store most known symbols (Greek, Chinese...). But the change is big, and I don't see a way to create a package runing in the two charset, so I created a specific package for the most updated SME server and another, neeeded until all SME server be updated (can takes years !).

Wish List : Ideas to improve/extend this contribution

smeserver-fetchmail seems to have many users, and some of them gave me some greats ideas to improve or extend the contribution. I'll try to settle all theses ideas here. Don't hesitate to send me others ideas (by mail), I'll try to put them here also.
But I have very little free time, and I cannot say that these ideas will be included in the contribution !
If YOU think that one of theses ideas really need to be included, don't hesitate to do the work ! You're welcome !
The ideas are here without any order.

Fetchmail for groups For now, if you want that a mail 'fetched' is send to more than one user, you'll have to check every recipients by hand. it could be easier to use SME group as recipient : configuration easier, and the change in a group would be automatically active on Fetchmail
Fetchmail configuration backup If you have to migrate a fetchmail configuration from one SME server to another, you have to recreate the whole configuration. It would be marvelous to have an export/import tool, like the great lazy-admin-tools.
External mail account 'enable/disable' checkbox Most useful in test and debug, the idea is to have in from of each POP/IMAP account, a 'enable' checkbox. This would permit to temporary disable an account without needing to remove and recreate it.
Secure pop3 retrieval Some mail services (GMail from Google for instance) can only be accessed via secure POP3 (or maybe secure IMAP). The fetchmail binary send with SME doesn't have the ssl extension embedded. So it's not only a configuration option to add...This feature is now supported since version 1.3.5-00 (yep !)
Custom port number for the collected services TCP/IP gives for each well known service a port number. For example, 110 is for POP3 and 143 is for IMAP4. But some providers use non standard port number. It could be useful to be able to set these non standard port numbers.
IMAP4 folder selection By default, fetchmail retrieve mails in the distant INBOX folder. But it's possible to ask fetchmail to retrieve mails in other folder(s).
Webmail fetching How to fetch webmail servers. It seems to be possible, but it's maybe very 'webmail services' dependant... I think of a kind of 'plug-in mode' witch could permit to add webmail facility 'on the fly' (who said "he's dreaming" ?)
User Panel configuration The possibility for each user to set alone his/her fetchmail rule, without access to the other account configuration. This could/should be included in the Dungog userpanel
Some fine-tune options in the Panel Some options (like the name of the internal mail server) are today 'hard-coded'. This is bad... Theses options should at least be stored in the configuration database (thus permitting change without need to go 'deep in the files'), and could eventually be on the server-manager panel.
Better 'SME Spirit' integration The installation process of smeserver-fetchmail is not in the good SME-Spirit. This prevents smeserver-fetchmail from been included in a first installation CD.
Config reporter The fetchmail configuration process is not ever so straightforward. It could be 'cool' to have a 'report configuration' panel, giving in one page wich mail account(s) is/are fetched via wich SME account, and who are the final recipients. (Actually, it could be cool for me, when questions arise !).
Your great idea here Your marvellous explanations here.

Contrib authors

The initial sme-fetchmail was written by Vincent Filali (his site). The whole fetchmail concept is from him.

The current version, smeserver-fetchmail is a complete interface rework, but the base engine is still the same. This work and the documentation was done by me, Pascal Schirrmann (my site), under supervision of Vincent.

Change Log

April the 20, 2008 bug correction (version 1.0.11) working together with smeserver-fetchmail and smeserver-fetchmail-utf81.3.5-02 rpms.
SME did with the April 2008 update a big step, going from ISO-8859-1 charset to UTF-8. But, as not everybody update his server at the same time, Contribs have to work in the two worlds. So there are now two packages, smeserver-fechmail and smeserver-fetchmail-utf8.
March the 20, 2007 bug correction (version 1.0.10) working together with smeserver-fetchmail-1.3.5-01 rpms.
Some translations typo stop the display of fetchmail in the server-manager panel
March the 8, 2007 bug correction (version 1.0.9) working together with smeserver-fetchmail-1.3.5-00 rpms.
Use of new SME server modifications. Now, incoming fetched mails are again checked by the anti spam filter.
(Still experimental) support of POP3 and IMAP4 via SSL.
Control of the fetchmail timeout value, in case of a non responding remote server (default wait time : 60 seconds).
Warning: this version can only be installed on SME server starting at version 7.1.1 !
October the 25, 2006 bug correction (version 1.0.8) working together with smeserver-fetchmail-1.3.4-10 rpms.
Small bug correction : SME 7 now accept user names with dots ( . ) in it. smeserver-fetchmail didn't like that !
April the 29, 2006 Some small corrections and adds (version 1.0.7) working together with smeserver-fetchmail-1.3.4-09 rpms.
Brand new german version, thanks to Dietmar Berteld.
IMAP is now the default protocol for new fetchmail accounts: IMAP works better than POP3 with fetchmail, and most mail servers have IMAP capabilities.
In fact, smeserver-fetchmail didn't work with SME 7! It only worked because of file expand due to other servicies. Now, smeserver-fetchmail works totally (and bug free, of course :-) ) on SME7.
January the 14, 2006 Some small corrections and adds (version 1.0.6) working together with smeserver-fetchmail-1.3.4-08 rpms.
External mail servers can now by given as name or IP address.
Brand new english version, thanks to Dave Dumolo.
Minor changes to hopefully suppress all errors messages during upgrade and/or uninstall on SME 7 version.
October the 22, 2005 Some small corrections and adds (version 1.0.5) working together with smeserver-fetchmail-1.3.4-07 rpms.
smeserver-fetchmail didn't work with SME 7 beta 4 and newer, due to a change in the FormMagick library.
Warning : the SME 7 support is still very light : there's still errors messages when upgrading or uninstalling the contrib, and I still wasn't able to do all needed sanity tests.
August the 19, 2005 Some small corrections and adds (version 1.0.4) working together with smeserver-fetchmail-1.3.4-06 rpms.
This release contains a bug fix : Preceding version incorrectly send incoming mails to the SmartHost, if a SmartHost has been set. But the smart host is set for Outgoing mails !
August the 2, 2005 Some small corrections and adds (version 1.0.3) working together with smeserver-fetchmail-1.3.4-05 rpms. This release contains theses changes :
  • This version works with SME 7.0 Alpha 26 (a very few tests done, still an annoying message when removing the package).
  • A new config database value has been created, allowing to set a particular destination host for the fetched mails.
  • Bug correction : the SMTP Smart Host, if set in the configuration, is again the default destination server for retrieved mails. Let me know if this is useful or annoying :-).
  • And last but not least : Italian translation added, thanks to Antonio Fragola - MrShark.
May the 20, 2005 1.0.2 : Add a 'wishList' section
May the 15, 2005 Some very small corrections (version 1.0.1) working together with smeserver-fetchmail-1.3.4-04 rpms. This release contains theses three changes :
  • A small bug didn't permit the use of smeserver-fetchmail on SME 6.5 RC 1 .
  • smeserver-fetchmail use internally a name to send all incoming mails to the local mail server. Until now, the internal mail server name was 'mail.{DNS Domain name}' In some cases (especially when using an external DNS server), this wasn't working. Now, the internal mail server name is '{DNS Domain Name}'.
  • Internal distribution list name change : When smeserver-fetchmail has to deliver mails to more than one recipients, it use internal distribution lists. Until now, theses lists where prefixed with the fm% sequence (to be sure they are unique in the system). Some antispam rules don't like the percent sign. Now, the smeserver-fetchmail distribution list prefix is fm_fm- .
May the 16, 2004 Initial version (1.0) corresponding to RPMs smeserver-fetchmail-1.3.4