Fortigate Conserve Mode – Investigations

A Fortigate can enter in Conserve Mode when the remaining free physical memory (RAM) is nearly exhausted. The memory threshold that triggers the conserve mode varies by model but it is around 20-30 % of free memory. For example, I have a 61E with a threshold at 70%. To check if your device is in the conserve mode, you can use this command:

# diagnose hardware sysinfo shm
SHM counter:          139
SHM allocated:      16384
SHM total:     1272930304
conserve mode:  off
system last entered:  n/a
sys fd last entered:  n/a
SHM FS total:  1302220800
SHM FS free:   1302175744
SHM FS avail:  1302175744
SHM FS alloc:       45056

Most content inspection that the Fortigate performs requires that the files, e-mail URL and so on be temporary buffered and examined as a whole. This buffer used the RAM and the opperating system itself needs memory to works correctly. The main objective of the Conserve Mode is to prevent all the component features of the Fortigate from trying to use more memory then it has. The Fortigate will exit this mode only when the available physical memory is under the threshold. 

The main question is : How to reduce the amount of used memory ? The answer is : that depend because we can modify a lot of configuration to reduce memory usage. First, if your Fortigate enters often in Conserve Mode, you must check which service uses a lot of memory. 

# diagnose sys top-summary    --> Press "m" to sort by memory usage

  CPU [||                                      ]   5.4%
   Mem [||||||||||||||||||||||||||              ]  67.0%  1266M/1866M
   Processes: 20 (running=1 sleeping=122)
   
   PID      RSS   CPU% ^MEM%   FDS     TIME+  NAME
 * 7340    411M    0.0 22.0   268  01:01.73  ipsmonitor [x6]
   6096     32M    0.0  1.7    29  09:25.93  dnsproxy
   4499     28M    0.0  1.5    32  03:41.58  cmdbsvr
   6073     27M    0.0  1.5    17  43:52.20  reportd
   6080     22M    0.0  1.2    48  09:19.80  scanunitd [x4]
   6061     21M    0.0  1.1    23  02:28.76  httpsd [x4]
   18274    20M    0.0  1.1    12  00:00.53  pyfcgid [x4]
   6059     17M    0.0  0.9    59  14:16.90  miglogd [x3]
   6068     15M    0.0  0.8    23  19:30.65  forticron
   6090     15M   20.6  0.8    13  07:57.82  sshd [x4]
   6074     13M    0.8  0.7    32  11:54.18  sslvpnd [x4]
   6098      9M    0.0  0.5    16  01:25.32  fgfmd
   6099      8M    0.0  0.5    23  02:52.12  cw_acd
   6086      8M    0.0  0.4    16  54:45.00  src-vis
   6082      6M    0.0  0.3    15  08:04.50  updated
   6076      6M    0.0  0.3    10  00:00.10  guacd
   6105      6M    0.0  0.3    19  01:39.45  cu_acd
   6088      5M    0.0  0.3    19  03:31.49  urlfilter
   6069      5M    0.0  0.3    15  01:40.28  forticldd
   6081      5M    0.0  0.3    31  02:46.85  iked

In this example we can see that ipsmonitor has 6 worker thread and it consume 22% of RAM. Regarding the global configuration, it is possible to perform some tuning. Below some examples:

# config system global
(global) # set tcp-halfclose-timer 60   --> default 120 s
(global) # set tcp-halfopen-timer 5     --> default 10 s
(global) # set tcp-timewait-timer 0     --> default 1 s
# config system global
(global) # set udp-idle-timer 60        --> default 180 s
# config ips global
(global) # set socket-size 4            --> default 32 MB
(global) # set engine count 2           --> default 0 = infinite
# config system dns
(dns) # set dns-cache-limit 300         --> default 1800 s
# config system session-ttl
(session-ttl) # set default 300         --> default 3600 s

Of course all these settings must be configured carefully and tested. Some other commands can be useful in this situation

# diagnose sys kill <signal_number>
# diagnose test application <application-name> <test-level>

To know the possible values for <test-level> simply enter the command without the <test-level> value. Below an example with the application <ipsmonitor>

# diagnose test application ipsmonitor

IPS Engine Test Usage:

    1: Display IPS engine information
    2: Toggle IPS engine enable/disable status
    3: Display restart log
    4: Clear restart log
    5: Toggle bypass status
    6: Submit attack characteristics now
   10: IPS queue length
   11: Clear IPS queue length
   12: IPS L7 socket statistics
   13: IPS session list
   14: IPS NTurbo statistics
   15: IPSA statistics
   16: Display device identification cache
   17: Clear device identification cache
   21: Reload FSA malicious URL database
   22: Reload whitelist URL database
   24: Display Flow AV statistics
   25: Reset Flow AV statistics
   96: Toggle IPS engines watchdog timer
   97: Start all IPS engines
   98: Stop all IPS engines
   99: Restart all IPS engines and monitor

After that, if your firewall still entered in Conserve Mode to often, replace it with higher model 🙂

Bookmark the permalink.

One Comment

  1. Awesome info, many thanks !!!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.