Fortigate, Exceeded ICMP rate limit(type=3 code=1 limit=1s), drop

Hi guys, 

I wrote this article because I had a problem that stuck me for a long time. For the context, I performed a migration from devices that were behind à Fortigate 1500D toward a Fortigate 500E using VDOMs. 

One of these devices was a Nagios XI server (172.20.11.158) that monitor 400 various devices. When it was behind the 1500D, all worked fine so I decided to migrate it. 

The Symptoms

After the migration, some of devices monitored by the Nagios server was unreachable. 

Immediately, I think about a routing or policy problem. So I performed some debug tasks using CLI. 

diagnose debug disable
diagnose debug flow trace stop diagnose debug flow filter clear diagnose debug reset
diagnose debug flow filter saddr 172.20.11.158 diagnose debug flow filter daddr 172.16.181.56 diagnose debug flow show console enable diagnose debug flow show function-name enable diagnose debug console timestamp enable diagnose debug flow trace start 1000 diagnose debug enable

func=print_pkt_detail line=5311 msg="vd-INFRA received a packet(proto=1, 172.20.11.158:12530->172.16.181.59:2048) from INT_SERVERS. type=8, code=0, id=12530, seq=1."
func=init_ip_session_common line=5470 msg="allocate a new session-29a8ae66"
func=vf_ip_route_input_common line=2576 msg="find a route: flag=04000000 gw-172.16.181.59 via LNK_CUSTOMERS"
func=fw_forward_handler line=743 msg="Allowed by Policy-63:"
func=__icmp_send line=549 msg="Exceeded ICMP rate limit(type=3 code=1 limit=1s), drop"

With this output we can conclude this: 

  • VDOM “INFRA” received ICMP packet from Nagios server (172.20.11.158) to send to 172.16.181.59
  • This packet is received from the INT_SERVERS interface
  • The Fortigate allocates a new session for this communication
  • A specific route is found via interface LNK_CUSTOMERS
  • The traffic is allowed by policy ID 63

These information exclude a routing or policy problem. The last line of the output shows the reason why the packet send by Nagios never reach the destination.

The fortigate has dropped the ICMP packet because of exceeded ICMP rate limit. 

The researchs

I performed a lot of research on Google and friends about this error but I never found a real solution. So I decided to open a case to the Fortinet support. I was a little disappointed with the response of the support. 

This is an expected behavior: 
The package is dropped since the ICMP is exceeding the rate limit. 
The FortiGate team has a limitation for ICMP; the limit is 6 packets per second per sender. 

This is based on RFC 1812: 
4.3.2.8 Rate Limiting 

A router which sends ICMP Source Quench messages MUST be able to 
limit the rate at which the messages can be generated. A router 
SHOULD also be able to limit the rate at which it sends other sorts 
of ICMP error messages (Destination Unreachable, Redirect, Time 
Exceeded, Parameter Problem). The rate limit parameters SHOULD be 
settable as part of the configuration of the router. How the limits 
are applied (e.g., per router or per interface) is left to the 
implementor’s discretion. 

I will recommend to increase the interval between the pings. 

Raaaaaaaah. Thank you dear support… Many many thank for your recommendation…  After that, I explained that before the migration all worked fine and blahblahblah… I will spare you their answer …

The workaround

I continued my research and found nothing really interesting. During a discussion with a colleague, we spoke about DoS policy… My colleague asks me about the usage of DoS policy and give me the idea of trying to use them to bypass the hardcoded limitation of ICMP  rate limiting…

To be honest I was not very enthusiastic and did not believe too much in this solution. But after all, I had no better idea on hand …

So, I created a DoS policy with the following settings:

config firewall DoS-policy
    edit 1
        set interface "INT_SERVERS"
        set srcaddr "172.20.11.158"
        set dstaddr "all"
        set service "ALL_ICMP"
        config anomaly
            edit "icmp_flood"
                set status enable
                set log enable
                set threshold 5000
            next
            edit "icmp_sweep"
                set status enable
                set log enable
                set threshold 5000
            next
            edit "icmp_src_session"
                set status enable
                set log enable
                set threshold 5000
            next
            edit "icmp_dst_session"
                set status enable
                set log enable
                set threshold 5000
            next
        end
    next
end

It’s a bit strange but after doing this confguration, the Nagios server was again able to correctly monitor all the equipment as before. I sent this solution to Fortinet support and I am still waiting for response from them. I would like to receive a more precise answer as to why this solution works.

hoping that this article will help some, I will supplement it with other information as soon as I have a return of the support.

Bookmark the permalink.

2 Comments

  1. Hello,

    I found your site after experiencing this myself.
    This spared me the mail to support

    Out of curiosity, what did Fortigate say when you told them it worked correctly under the 1500D but not the 500E. I suspect the issue has to do with the NPU/SPU on the Fortigate.

    • Cyrill Gremaud

      Hello Bobola Oke. I don’t know what my old 1500D said before because I never checked this unfortunately. Which model do you have ? Did you have the same error message ?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.