I wrote this article because I had a problem that stuck me for a long time. For the context, I performed a migration from devices that were behind à Fortigate 1500D toward a Fortigate 500E using VDOMs.
One of these devices was a Nagios XI server (172.20.11.158) that monitor 400 various devices. When it was behind the 1500D, all worked fine so I decided to migrate it.
After the migration, some of devices monitored by the Nagios server was unreachable.
Immediately, I think about a routing or policy problem. So I performed some debug tasks using CLI.
diagnose debug disable
diagnose debug flow trace stop diagnose debug flow filter clear diagnose debug reset
diagnose debug flow filter saddr 172.20.11.158 diagnose debug flow filter daddr 172.16.181.56 diagnose debug flow show console enable diagnose debug flow show function-name enable diagnose debug console timestamp enable diagnose debug flow trace start 1000 diagnose debug enable
func=print_pkt_detail line=5311 msg="vd-INFRA received a packet(proto=1, 172.20.11.158:12530->172.16.181.59:2048) from INT_SERVERS. type=8, code=0, id=12530, seq=1."
func=init_ip_session_common line=5470 msg="allocate a new session-29a8ae66"
func=vf_ip_route_input_common line=2576 msg="find a route: flag=04000000 gw-172.16.181.59 via LNK_CUSTOMERS"
func=fw_forward_handler line=743 msg="Allowed by Policy-63:"
func=__icmp_send line=549 msg="Exceeded ICMP rate limit(type=3 code=1 limit=1s), drop"
With this output we can conclude this:
- VDOM “INFRA” received ICMP packet from Nagios server (172.20.11.158) to send to 172.16.181.59
- This packet is received from the INT_SERVERS interface
- The Fortigate allocates a new session for this communication
- A specific route is found via interface LNK_CUSTOMERS
- The traffic is allowed by policy ID 63
These information exclude a routing or policy problem. The last line of the output shows the reason why the packet send by Nagios never reach the destination.
The fortigate has dropped the ICMP packet because of exceeded ICMP rate limit.
I performed a lot of research on Google and friends about this error but I never found a real solution. So I decided to open a case to the Fortinet support. I was a little disappointed with the response of the support.
This is an expected behavior:
The package is dropped since the ICMP is exceeding the rate limit.
The FortiGate team has a limitation for ICMP; the limit is 6 packets per second per sender.
This is based on RFC 1812:
126.96.36.199 Rate Limiting
A router which sends ICMP Source Quench messages MUST be able to
limit the rate at which the messages can be generated. A router
SHOULD also be able to limit the rate at which it sends other sorts
of ICMP error messages (Destination Unreachable, Redirect, Time
Exceeded, Parameter Problem). The rate limit parameters SHOULD be
settable as part of the configuration of the router. How the limits
are applied (e.g., per router or per interface) is left to the
I will recommend to increase the interval between the pings.
Raaaaaaaah. Thank you dear support… Many many thank for your recommendation… After that, I explained that before the migration all worked fine and blahblahblah… I will spare you their answer …
I continued my research and found nothing really interesting. During a discussion with a colleague, we spoke about DoS policy… My colleague asks me about the usage of DoS policy and give me the idea of trying to use them to bypass the hardcoded limitation of ICMP rate limiting…
To be honest I was not very enthusiastic and did not believe too much in this solution. But after all, I had no better idea on hand …
So, I created a DoS policy with the following settings:
config firewall DoS-policy edit 1 set interface "INT_SERVERS" set srcaddr "172.20.11.158" set dstaddr "all" set service "ALL_ICMP" config anomaly edit "icmp_flood" set status enable set log enable set threshold 5000 next edit "icmp_sweep" set status enable set log enable set threshold 5000 next edit "icmp_src_session" set status enable set log enable set threshold 5000 next edit "icmp_dst_session" set status enable set log enable set threshold 5000 next end next end
It’s a bit strange but after doing this confguration, the Nagios server was again able to correctly monitor all the equipment as before. I sent this solution to Fortinet support and I am still waiting for response from them. I would like to receive a more precise answer as to why this solution works.
hoping that this article will help some, I will supplement it with other information as soon as I have a return of the support.