Archive for June, 2009

Mail fails consistently with timeout or lost connection

Every now and then, mail fails with “timed out while sending end of data — message may be sent more than once”, or with: “lost connection after DATA”. Network outages happen, systems crash. There isn’t much you can do about it. Usually the problem goes away by itself.

However, when you see mail deliveries fail consistently, you may have a different problem: broken path MTU discovery. Or it could be a broken PIX firewall.
Cisco PIX “fixup protocol smtp” bug
The Cisco PIX firewall has a bug when running software older than version 5.2(4) or 6.0(1).

The bug ID is CSCds90792. The “fixup protocol smtp” feature does not correctly handle the case where the “.” and the “CRLF” at the end of mail are sent in separate packets.

How does one recognize a mailer behind a Cisco PIX with “fixup protocol smtp” enabled? As of version 5.1 and later, the fixup protocol smtp command changes the characters in the SMTP banner to asterisks except for the “2”, “0” and “0 SPACE” characters.

When you connect to a mailer behind such a filter you see something like:

220 **************************************0******0*********20 ****200**0*********0*00

IP path MTU discovery
A little background is in order. With the SMTP protocol, the HELO, MAIL FROM and RCPT TO commands and responses are relatively short. When you’re talking to old versions of sendmail, every command and every response is sent as a separate packet, because sendmail didn’t implement ESMTP command pipelining until recently.

The message content, however, is sent as a few datagrams, each datagram typically a kbyte large or even bigger, depending on your local network MTU.

When mail fails consistently due to a timeout, I suspect that the sending machine runs a modern UNIX which implements path MTU discovery. That causes the machine to send packets as large as it would send over the LAN, with the IP DON’T FRAGMENT bit set, preventing intermediate routers from fragmenting the packets that are too big for their networks.

Depending on what network path a message follows, some router on the way responds with an ICMP MUST FRAGMENT message saying the packet is too big. Normally, the sending machine will re-send the data after chopping it up into smaller pieces.

However, things break when some router closer to the sending system is dropping such ICMP feedback messages, in a mistaken attempt to protect systems against certain attacks. In that case, the ICMP feedback message never reaches the sending machine, and the connection times out.

This is the same configuration problem that causes trouble with web servers behind a misconfigured packet filter: small images/files are sent intact, large images/files time out because the server does not see the MUST FRAGMENT ICMP feedback messages.

Workaround: at the sending machine, disable path MTU discovery. Mail will get out, but of course everyone else will still suffer. How to disable path MTU discovery? It depends. Solaris has an ndd command; other systems use different means such as sysctl to control kernel parameters on a running system.

Workaround: at the receiving machine, set a smaller MTU. For example, people using PPPoE (PPP over Ethernet) often have to choose an MTU lightly smaller than the default 1500 for ethernet.

Fix: find the router that drops the ICMP MUST FRAGMENT messages, and convince the person responsible for it to fix the configuration.


Leave a comment

Squid debug ACLs

In your squid.conf add the following parameter:

debug_options ALL,1 28,9

Restart squid and take a look at cache.log. Use your regex skills to find what you want.

Leave a comment

Monitoring your system with sysstat

Sysstat is a package of monitoring tools, these are the tools included in the package.

Reports CPU statistics and input/output statistics for devices, partitions and network filesystems.
Reports individual or combined processor related statistics.
Reports statistics for Linux tasks (processes) : I/O, CPU, memory, etc.
Collects, reports and saves system activity information (CPU, memory, disks, interrupts, network interfaces, TTY, kernel tables,etc.)
Is the system activity data collector, used as a backend for sar.
Collects and stores binary data in the system activity daily data file. It is a front end to sadc designed to be run from cron.
Writes a summarized daily activity report. It is a front end to sar designed to be run from cron.
Displays data collected by sar in multiple formats (CSV, XML, etc.) This is useful to load performance data into a database, or import them in a spreadsheet to make graphs.

To use this tools, you need to install sysstat

-First download it from and perform the usual installation procedures.

One great feature of this tools is that you can configure it to run as a daemon, and it will log a lot of info about your system.

To start it if you are using Debian, edit with your favorite text editor, in my case is vi

vim /etc/default/sysstat

And change the line: ENABLED=”false” to ENABLED=”true”, so that file may look like this

# Default settings for /etc/init.d/sysstat, /etc/cron.d/sysstat
# and /etc/cron.daily/sysstat files

# Should sadc collect system activity informations? Valid values
# are “true” and “false”. Please do not put other values, they
# will be overwritten by debconf!

# Additional options passed to sa1 by /etc/init.d/sysstat
# and /etc/cron.d/sysstat
# By default contains the `-d’ option responsible for
# generating disk statisitcs.

# Additional options passed to sa2 by /etc/cron.daily/sysstat.

sa1 is a variant of sadc, which is designed to work as a cronjob, just like sa2 which is a variant of sar command also designed to be run as cronjob.

If you are not using Debian, you will not have the /etc/default/sysstat, so you will have to add to your root’s cronjob file this lines:

0 8-18 * * 1-5 /usr/lib/sysstat/sa1 1200 3 &
5 19 * * 1-5 /usr/lib/sysstat/sa2 -A &

The -d option in sa1 makes it to store disks data, which by default are not written.

Now you can use sar to see the data stored, in your system, check the page of the project for more information about this performance toolkit


Leave a comment

Postfix limit incoming or receiving email rate

Postfix (smtpd daemon) can enforce a number of limits on incoming email. This will stop email flooding attacks.

A bot connects to your Postfix email server and sends garbage commands or spam, attempting to crash your server. You can limit:

=> The length of lines in a message and so on

=> The size of messages

=> The number of recipients for a single delivery

Try following directives in your postfix config file:
smtpd_error_sleep_time – The SMTP server response delay after a client has made more than $smtpd_soft_error_limit errors, and fewer than smtpd_hard_error_limit errors, without delivering mail.
smtpd_soft_error_limit : The number of errors a remote SMTP client is allowed to make without delivering mail before the Postfix SMTP server slows down all its responses.
smtpd_hard_error_limit : The maximal number of errors a remote SMTP client is allowed to make without delivering mail. The Postfix SMTP server disconnects when the limit is exceeded.

Open config file
# vi
Append following directives:
smtpd_error_sleep_time = 1s
smtpd_soft_error_limit = 10
smtpd_hard_error_limit = 20

Save and restart/reload postfix configuration
# /etc/init.d/postfix restart

Postfix waits one second before each error such as HELO command not provided or FQDN hostname does not exists etc After 10 such errors postfix will start to increase delay. If error limits touches 20 Postfix will disconnect client.


Leave a comment

Password-less logins with OpenSSH

Because OpenSSH allows you to run commands on remote systems, showing you the results directly, as well as just logging in to systems it’s ideal for automating common tasks with shellscripts and cronjobs. One thing that you probably won’t want is to do though is store the remote system’s password in the script. Instead you’ll want to setup SSH so that you can login securely without having to give a password.

Thankfully this is very straightforward, with the use of public keys.

To enable the remote login you create a pair of keys, one of which you simply append to a file upon the remote system. When this is done you’ll then be able to login without being prompted for a password – and this also includes any cronjobs you have setup to run.

If you don’t already have a keypair generated you’ll first of all need to create one.

If you do have a keypair handy already you can keep using that, by default the keys will be stored in one of the following pair of files:

* ~/.ssh/identity and ~/.ssh/
o (This is an older DSA key).
* ~/.ssh/id_rsa and ~/.ssh/
o (This is a newer RSA key).

If you have neither of the two files then you should generate one. The DSA-style keys are older ones, and should probably be ignored in favour of the newer RSA keytypes (unless you’re looking at connecting to an outdated installation of OpenSSH). We’ll use the RSA keytype in the following example.

To generate a new keypair you run the following command:

skx@lappy:~$ ssh-keygen -t rsa

This will prompt you for a location to save the keys, and a pass-phrase:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/skx/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/skx/.ssh/id_rsa.
Your public key has been saved in /home/skx/.ssh/

If you accept the defaults you’ll have a pair of files created, as shown above, with no passphrase. This means that the key files can be used as they are, without being “unlocked” with a password first. If you’re wishing to automate things this is what you want.

Now that you have a pair of keyfiles generated, or pre-existing, you need to append the contents of the .pub file to the correct location on the remote server.

Assuming that you wish to login to the machine called mystery from your current host with the id_rsa and files you’ve just generated you should run the following command:

ssh-copy-id -i ~/.ssh/ username@mystery

This will prompt you for the login password for the host, then copy the keyfile for you, creating the correct directory and fixing the permissions as necessary.

The contents of the keyfile will be appended to the file ~/.ssh/authorized_keys2 for RSA keys, and ~/.ssh/authorised_keys for the older DSA key types.

Once this has been done you should be able to login remotely, and run commands, without being prompted for a password:

skx@lappy:~$ ssh mystery uptime
09:52:50 up 96 days, 13:45,  0 users,  load average: 0.00, 0.00, 0.00

What if it doesn’t work?

There are three common problems when setting up passwordless logins:

* The remote SSH server hasn’t been setup to allow public key authentication.
* File permissions cause problems.
* Your keytype isn’t supported.

Each of these problems is easily fixable, although the first will require you have root privileges upon the remote host.

If the remote server doesn’t allow public key based logins you will need to updated the SSH configuration. To do this edit the file /etc/sshd/sshd_config with your favourite text editor.

You will need to uncomment, or add, the following two lines:

RSAAuthentication yes
PubkeyAuthentication yes

Once that’s been done you can restart the SSH server – don’t worry this won’t kill existing sessions:

/etc/init.d/ssh restart

File permission problems should be simple to fix. Upon the remote machine your .ssh file must not be writable to any other user – for obvious reasons. (If it’s writable to another user they could add their own keys to it, and login to your account without your password!).

If this is your problem you will see a message similar to the following upon the remote machine, in the file /var/log/auth:

Jun  3 10:23:57 localhost sshd[18461]: Authentication refused:
bad ownership or modes for directory /home/skx/.ssh

To fix this error you need to login to the machine (with your password!) and run the following command:

chmod 700 .ssh

Finally if you’re logging into an older system which has an older version of OpenSSH installed upon it which you cannot immediately upgrade you might discover that RSA files are not supported.

In this case use a DSA key instead – by generating one:


Then appending it to the file ~/.ssh/authorized_keys on the remote machine – or using the ssh-copy-id command we showed earlier.

Note if you’ve got a system running an older version of OpenSSH you should upgrade it unless you have a very good reason not to. There are known security issues in several older releases. Even if the machine isn’t connected to the public internet, and it’s only available “internally” you should fix it.

Instead of using authorized_keys/authorized_keys2 you could also achieve a very similar effect with the use of the ssh-agent command, although this isn’t so friendly for scripting commands.

This program allows you to type in the passphrase for any of your private keys when you login, then keep all the keys in memory, so you don’t have password-less keys upon your disk and still gain the benefits of reduced password usage.
If you’re interested read the documentation by running:

man ssh-agent


Leave a comment

IE8 compatibility problem

Rendering engine:

IE8 improves rendering of content authored to various web standards (like HTML, CSS and JavaScript) in standards mode. Such changes might cause it to break compatibility as its behavior differs significantly from that of IE7. In order to maintain backwards compatibility, sites can opt-into IE7-like handling of content by inserting a specially created meta element into the web page, that triggers the “Compatibility mode” in the browser, using:

<meta http-equiv=”X-UA-Compatible” content=”IE=EmulateIE7″ />


Leave a comment

Dspam 3.8.0 with postfix

- Add user and group
groupadd dspam
useradd -g dspam -s/bin/false -c"DSPAM" dspam

( setenv CFLAGS '-O6 -march=i686' ;./configure \
 --with-dspam-home=/var/dspam \
 --with-dspam-home-mode=770 \
 --with-dspam-home-owner=dspam \
 --with-dspam-home-group=postdrop \
 --with-dspam-mode=2510 \
 --with-dspam-owner=dspam \
 --with-dspam-group=postfix \
 --with-delivery-agent=/usr/sbin/sendmail \
 --with-storage-driver=mysql_drv \
 --with-mysql-includes=/usr/local/include/mysql \
 --with-mysql-libraries=/usr/local/lib/mysql \
 --enable-preferences-extension \
 --enable-virtual-users \
 --enable-daemon \
 --enable-large-scale )


make install

- DSPAM.CONF /usr/local/etc/dspam.conf
Home /var/dspam
StorageDriver /usr/local/lib/
TrustedDeliveryAgent "/usr/sbin/sendmail"
DeliveryPort        10026
DeliveryIdent       localhost
DeliveryProto       SMTP
OnFail error
Trust root
Trust httpd
Trust dspam
Trust postfix
Trust mail
Trust mailnull
Trust smmsp
Trust daemon
TrainingMode teft
TestConditionalTraining on
Feature chained
Feature whitelist
Algorithm graham burton
PValue graham
Preference "spamAction=quarantine"
Preference "signatureLocation=headers"  # 'message' or 'headers'
Preference "showFactors=on"
AllowOverride localStore
AllowOverride trainingMode
AllowOverride spamAction spamSubject
AllowOverride statisticalSedation
AllowOverride enableBNR
AllowOverride enableWhitelist
AllowOverride signatureLocation
AllowOverride showFactors
AllowOverride optIn optOut
AllowOverride whitelistThreshold
MySQLServer     /tmp/mysql.sock
MySQLPort       3306
MySQLUser               dspam
MySQLPass               ******
MySQLDb                 dspam
MySQLCompress           true
MySQLVirtualTable          dspam_virtual_uids
MySQLVirtualUIDField       uid
MySQLVirtualUsernameField  username
MySQLUIDInSignature    on

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
#smtp      inet  n       -       n       -       40       smtpd  -o content_filter=filter:dummy
#smtp      inet  n       -       n       -       30       smtpd   -o content_filter=clamav:clamav
smtp      inet  n       -       n       -       30       smtpd  -o content_filter=lmtp:unix:/tmp/dspam.sock
localhost:10026 inet  n -       n       -       50        smtpd
#  -o cleanup_service_name=pre-cleanup
  -o content_filter=clamav:clamav
  -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
  -o smtpd_helo_restrictions=
  -o smtpd_client_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o mynetworks=
  -o smtpd_authorized_xforward_hosts=

Leave a comment

Disable bell in xterm

Put the following in you .bashrc
xset -b
set bell-style none

Or put his in your .Xdefaults file:
xterm*visualBell: false

Leave a comment

Running hundreds of Postfix processes on FreeBSD

With hundreds of Postfix processes, the kernel will eventually run out of file handles; after that, it will run out of sockets.

To set the following kernel parameters at boot time, add the following lines to the /boot/loader.conf file (this is verified with FreeBSD 4.4):


With FreeBSD 4.2, the last three parameters cannot be set from /boot/loader.conf. To set the open file limits, execute the following commands as root:

# sysctl -w kern.maxfiles=16384
# sysctl -w kern.maxfilesperproc=16384

With FreeBSD 4.2, kern.maxproc can be set only by recompiling the kernel with a different maxusers setting in the kernel configuration file.


Leave a comment

Some Oracle’s installation parameters

Add Oracle users and groups for Oracle Database 10g installation.
For a typical Oracle 10g installation, two groups and an Oracle user need to be added.

groupadd -g 499 oinstall;
groupadd -g 502 dba;
useradd  -u 499 -p oracle -g oinstall -G dba oracle;

Configure kernel parameter for database installation.
Ideally, you could write a script that would check the existing kernel parameter settings. If any of the kernel parameters were set higher than the value recommended by Oracle for the installation, the value would not be changed. If the value of the parameter was lower than the recommended minimum value, you would adjust it to meet the required threshold.

The following example simply sets some of the values that needed to be changed on a test server:

echo "# " >> /etc/sysctl.conf
echo "# Required for Oracle 10g " >> /etc/sysctl.conf
echo "# " >> /etc/sysctl.conf
echo "kernel.shmall = 2097152" >> /etc/sysctl.conf
echo "kernel.shmmax = 2147483648" >> /etc/sysctl.conf
echo "kernel.shmmni = 4096" >> /etc/sysctl.conf
echo "kernel.sem = 250 32000 100 128" >> /etc/sysctl.conf
echo "fs.file-max = 65536" >> /etc/sysctl.conf
echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf
/sbin/sysctl -p

See the Oracle Database 10g installation documentation for more details on the configuration of kernel parameters.

Set environmental variables for database installation.
When a user logs in, the .bash_profile in their home directory (typically /home/<username>) is executed and environmental variables such as ORACLE_BASE, ORACLE_HOME, and PATH variables are set.

echo "ORACLE_BASE=/oracle/home" >> /home/oracle/.bash_profile
echo "ORACLE_HOME=/oracle/home/OraHome_1" >> /home/oracle/.bash_profile
echo "export ORACLE_HOME ORACLE_BASE" >> /home/oracle/.bash_profile
echo "PATH=$PATH:$ORACLE_HOME/bin" >> /home/oracle/.bash_profile
echo "export PATH"

There are a variety of other variables that you might want to set in a user’s .bash_profile. For instance, you may also want to set JAVA_HOME or other variables depending upon your installation and the user’s responsibilities.

Other pre-installation tasks. You could add a number of additional verification/configuration steps to the script to perform the following:

  • Check that sufficient memory is installed
  • Verify that the amount of swap spaced configured meets minimum requirements
  • Check for sufficient disk space and in various directories (e.g. /tmp) to accommodate the installation in question
  • Confirm that required software packages are installed
  • Create directories required for the Oracle Database installation


Leave a comment