How to Protect a Self-Hosted Application from Brute Force Attacks with FortiGate

Protect Application from Brute Force Attacks with FortiGate

Understanding Brute Force Attacks: What They Are and Why They Matter for Self-Hosted Applications

Imagine someone trying to unlock your front door by trying every possible key — again and again — until one finally fits. This is what happens in a brute force attack online. Attackers do not use sneaky tricks. Instead, they flood your application with many username and password guesses. They rely on persistence and computing power.

Brute force attacks come in different forms:

Simple password guessing: Trying common passwords like “123456” or “password” on your login page.
Credential stuffing: Using leaked username-password lists from other breaches to gain access.
Distributed brute force: Many machines (a botnet) work together to flood your app with login attempts from different IPs, making blocking harder.

A Real-World Scenario: What Happens When an Attack Succeeds

Say you run a small e-commerce site hosted on your own server. Attackers start bombarding your login page with thousands of guesses. Since there are no limits on login attempts, they eventually guess the admin’s password.

Once inside, attackers can cause serious harm:

Take your site offline: By flooding the server or changing files, causing crashes or slowdowns that frustrate customers.
Steal sensitive data: Copy personal info, credit cards, and order history to sell or misuse.
Inject malicious code: Add backdoors or malware to your site. This can spread infections or ransomware.

Without strong protection, these attacks can cause lost revenue, legal troubles, and damage your reputation.

Why FortiGate is the Right Shield for Your Application

FortiGate works like a vigilant guard at your application’s door. It inspects every visitor and blocks suspicious ones. It is a next-gen firewall for small and medium networks, offering:

Intrusion Prevention System (IPS): Detects and stops malicious activity in real-time.
Connection Limits: Controls how many connections an IP can make to avoid floods.
Application Control and Traffic Shaping: Prioritizes good traffic and restricts harmful behavior.
Threat Intelligence: Uses updated data to spot known attack patterns.

With FortiGate you get more than a firewall – you get a multi-layer defense that stops brute force attacks before they reach your app, keeping it safe and running well.

How Brute Force Attacks Work

Brute force attacks rely on trial and error to break into accounts or systems. Attackers use automated tools to repeatedly try different username and password combinations. These tools can quickly test thousands or even millions of possibilities. The goal is simple: eventually guess the correct credentials and gain unauthorized access.

Common Brute Force Techniques

There are several common techniques attackers use:

Basic brute force: Trying every possible password combination, starting from simple to complex.
Dictionary attacks: Using a list of common passwords or words from dictionaries to guess credentials faster.
Credential stuffing: Using username-password pairs leaked from other data breaches to try on your application, assuming users reuse passwords.
Hybrid attacks: Combining dictionary words with numbers or symbols to increase chances of success.

Attackers often use botnets — networks of compromised computers — to launch distributed attacks. This spreads login attempts across many IP addresses, making it harder for firewalls or security tools to block them.

Common Targets and Methods of Attack on Applications

Applications with login portals, APIs, or admin panels are prime targets for brute force attacks. Attackers focus on these points because gaining access here often means full control over the app.

Common targets include:

Web login pages: User portals where passwords are entered.
Admin panels: Interfaces that control application settings and user data.
API endpoints: Sometimes APIs require authentication and can be attacked similarly.
Remote access services: Such as SSH or RDP, which allow control over the server.

The methods used can vary depending on the target and attacker’s resources:

Rapid-fire login attempts: Flooding the login page with many requests in a short time.
Slow and low attempts: Trying few guesses per minute to avoid detection.
Using stolen credentials: From other sites to gain access without guessing.
Exploiting weak passwords: Targeting users with simple or reused passwords.

Understanding these methods helps in designing effective defenses to detect and block brute force attacks before they cause harm.

FortiGate Configuration Checks

Step 1: Preparing Your FortiGate for Protection

Before setting up protection against brute force attacks, it’s important to ensure your FortiGate device is properly prepared. Start by verifying that the device is correctly connected to your network and that you have administrative access through the web GUI or CLI.

Key preliminary steps include:

Set a strong admin password: Change the default password to a complex one to prevent unauthorized access to the firewall itself.
Configure management access: Limit management interfaces (e.g., HTTPS, SSH) to trusted IP addresses only.
Sync time settings: Ensure the device’s clock is accurate by configuring NTP (Network Time Protocol). Correct timestamps help in log analysis and troubleshooting.
Backup current configuration: Before making any major changes, save your existing configuration for recovery purposes.

Checking Firmware Version and Update Recommendations

Running the latest firmware version on your FortiGate is crucial for security and stability. Firmware updates often include important patches, new features, and improved threat detection capabilities.

To check and update firmware check the following post:

Fortinet FortiGate Firmware Upgrade

Step 2: Creating Firewall Policies to Protect Your Application

When you host your own application (web app, API, etc.) behind FortiGate , setting proper firewall policies is critical. The goal is to allow legitimate traffic while blocking suspicious or repeated malicious connections (such as brute force attempts).

Firewall Policies to Block Suspicious and Repeated Connections

Why block suspicious connections?

Attackers may try multiple connections rapidly to brute force login or exploit vulnerabilities. FortiGate can block or limit this traffic using:

DoS policy
Session limits
IPS signatures (if available)
Firewall policy with Traffic Shaping / Limitations

Example: Block excessive connection attempts from same IP (DoS Policy)

Step 1: Configure DoS Policy

config firewall DoS-policy
    edit 1
        set interface "wan1"
        set srcaddr "all"
        set action block
        set status enable
        set log enable
        set session-limit 50
        set status enable
    next
end

FortiGate DDoS Protection: Configure DoS Policies to Secure Self-Hosted Applications

Step 2: Create firewall policy that blocks known bad IPs (optional)

You can create an address group or use list to block known suspicious IPs.

Step 3. Advanced: Limit Connections Using Firewall Policy Traffic Shaping (Session Limits)

You can limit the number of concurrent sessions per source IP to mitigate brute force:

config firewall policy
    edit 2
        set name "Limit_Connections"
        set srcintf "wan1"
        set dstintf "internal"
        set srcaddr "all"
        set dstaddr "App_Server_IP"
        set action accept
        set service "ALL"
        set schedule "always"
        set logtraffic all
        set per-ip-session-limit 10
    next
end

This limits each IP to max 10 concurrent sessions.

Firewall Policies to Allow Legitimate Traffic

What to Allow

Incoming HTTP/HTTPS traffic to your web server IP
Traffic from trusted IP ranges (e.g., your office, partners)
DNS, if your app needs to resolve names

You can also check my post:

Fortinet FortiGate Firewall Policies Best Practices

Summary and Best Practices

Always allow only the necessary services and ports
Enable logging on firewall policies to monitor traffic
Use DoS policies or session limits to block brute force attacks
Regularly check FortiGate logs for suspicious patterns
Consider additional layers like Fail2Ban or Web Application Firewall (if possible)

Step 3: Configuring Connection Limits to Mitigate Brute Force Attempts

One of the most effective ways to protect your self-hosted application from brute force attacks is by limiting how many connections or login attempts a single IP address can make within a certain timeframe. This helps prevent attackers from flooding your application with endless login tries.

Setting Session Limits and Connection Thresholds

On FortiGate 60D, you can configure session limits and connection thresholds to control the number of simultaneous or total connections per IP. This acts like a speed bump, slowing down or blocking excessive requests that indicate a brute force attempt.

Key parameters to configure include:

Session timeout: Ensures stale sessions are closed quickly to free resources.

Session TTL (Time To Live): Defines how long a session remains active.

Maximum concurrent sessions per IP: Limits how many sessions an IP can have at the same time.

Session rate limits: Controls how many new sessions can be established from a single IP within a set time window.

Example: Limiting Connections per IP Address

Here’s a basic example of how to set a limit on the number of concurrent sessions per IP address using FortiGate CLI:

config firewall policy
    edit <policy_id>
        set srcintf "wan1"
        set dstintf "internal"
        set srcaddr "all"
        set dstaddr "your_app_server"
        set action accept
        set schedule "always"
        set service "HTTP" "HTTPS"
        set session-ttl 1800
        set per-ip-concurrent-session 10
    next
end

Explanation:

set session-ttl 1800 limits each session to 30 minutes.
set per-ip-concurrent-session 10 restricts each IP to 10 concurrent sessions to your application

Additional Rate Limiting with DoS Policies

You can also configure DoS (Denial of Service) policies to limit the rate of new connections from the same IP. For example:

config firewall DoS-policy
    edit 1
        set interface "wan1"
        set srcaddr "all"
        set status enable
        set service "HTTP" "HTTPS"
        set dstaddr "your_app_server"
        set anomaly enable
        set syn_flood enable
        set tcp_session_limit 20
        set tcp_session_limit_action block
    next
end

This example blocks IP addresses that exceed 20 concurrent TCP sessions, providing an additional layer against brute force flooding.

Step 4: Enabling and Using FortiGate’s DoS and IPS Features

Denial of Service (DoS) protection is essential for stopping attackers from overwhelming your self-hosted application with excessive traffic. Brute force attacks often involve flooding the login page with rapid, repeated connection attempts. Without DoS protection, this flood can exhaust server resources and cause legitimate users to lose access.

Using DoS Protection to Prevent Overload

FortiGate 60D offers built-in DoS protection that monitors traffic patterns and automatically blocks IPs that exceed defined thresholds. You can configure thresholds for new connection rates, concurrent sessions, and protocol-specific anomalies like SYN floods.

Key Steps to Enable DoS Protection:

Access the FortiGate GUI or CLI.
Define DoS policies on the WAN interface where external traffic arrives.
Set limits on connection rates per IP, such as maximum new sessions per second.
Enable anomaly detection features like SYN flood protection.
Specify the action (block, detect only, or quarantine) when thresholds are exceeded.

By tuning these settings, you reduce the risk of your server becoming unresponsive during an attack.

Activating the IPS Module to Detect and Block Brute Force Patterns

The Intrusion Prevention System (IPS) in FortiGate 60D scans network traffic for known attack signatures and suspicious behaviors. It is highly effective at identifying brute force attack patterns, such as repeated failed login attempts or abnormal connection bursts.

How to Enable IPS:

Navigate to Security Profiles > Intrusion Prevention in the FortiGate GUI.
Create or edit an IPS sensor and enable relevant signatures related to brute force attacks and authentication failures.
Apply the IPS sensor to the firewall policy protecting your application.
Optionally, configure IPS logging and alerting to monitor detected attacks.

IPS keeps your application safe by proactively blocking attacks before they can cause harm.

You can check my post about FortiGate Hidden Features:

Fortinet FortiGate: Enable Hidden Features

Step 5: Monitoring and Logging Brute Force Attempts

Effective defense starts with visibility. FortiGate provides detailed logging capabilities that help you monitor brute force attacks in real-time and analyze past events. By regularly reviewing logs, you can identify suspicious activities such as repeated failed login attempts, high connection rates from specific IP addresses, or blocked traffic patterns.

Tracking Attacks Through FortiGate Logs

To monitor brute force attempts, focus on:

Event logs: Show security events like intrusion attempts and policy violations.
Traffic logs: Detail connections and sessions, useful to spot abnormal spikes.
System logs: Provide device-level information including warnings and errors.

Logs can be viewed directly in the FortiGate GUI under Log & Report or exported to a centralized syslog server or SIEM for deeper analysis.

Setting Up Alerts and Automated Responses

To respond promptly to attacks, configure notifications and automated rules:

Email alerts: Set FortiGate to send emails when suspicious events or threshold breaches occur.
Syslog forwarding: Forward logs to external monitoring tools that can trigger alarms or ticketing systems.
Automatic blocking: Use firewall policies or DoS profiles to temporarily block offending IPs based on log events.
Scripts and automation: FortiGate supports automation triggers to run scripts or actions when specific log patterns are detected.

Example: Enabling Email Alerts for Security Events

Go to Log & Report > Alert Email in the GUI.
Configure your SMTP server settings.
Set which events should trigger alerts (e.g., blocked connections, intrusion attempts).
Enable alerts and test email delivery.

This setup ensures you’re informed immediately when a brute force attack is detected, enabling fast mitigation.

Step 6: Advanced Techniques: Geo-IP Filtering and Botnet Protection

Geo-IP filtering allows you to block or allow traffic based on the geographic origin of IP addresses. If your application only serves users from a specific country or region, there’s often no reason to allow connections from the rest of the world — especially from regions known for frequent cyberattacks.

Blocking Connections from Suspicious or Unwanted Countries

FortiGate lets you create firewall policies that deny traffic from selected countries using built-in Geo-IP data. This significantly reduces exposure to brute force attacks and other malicious traffic originating from high-risk locations.

Steps to Configure Geo-IP Blocking in FortiGate GUI:

Go to Policy & Objects > Addresses.
Click Create New > Address.
Set the type to Geography and select the country you want to block.
Repeat for all unwanted countries.
Go to Firewall Policy, and create a new policy that blocks traffic from those addresses to your application.

Tip: Use Geo-IP blocking with caution — avoid blocking countries where your legitimate users or partners might reside.

You can check this article for more details:

Fortinet FortiGate: Restrict Access to the Open Ports for Specific Country

Using FortiGuard Services for Extra Protection

FortiGuard is Fortinet’s cloud-based security intelligence service. It provides real-time updates and reputation-based data to enhance your FortiGate’s ability to detect and stop threats — including brute force and botnet activity.

FortiGuard features relevant to brute force protection:

Botnet IP Reputation: Automatically blocks known malicious IPs used in botnets or distributed brute force attacks.
Threat Intelligence Feeds: Keeps your firewall up to date with global attack patterns and high-risk sources.
Application Control & Web Filtering: Can prevent malicious scripts or traffic from reaching your application.

How to Enable FortiGuard Protection:

Navigate to Security Profiles > Web Filter / DNS Filter / IPS.
Make sure FortiGuard updates are enabled and up to date.
Enable Botnet C&C blocking under Threat Feeds or DNS Filtering.
Apply the updated security profiles to your active firewall policies.

By combining Geo-IP filtering with FortiGuard threat intelligence, you create a smarter, more adaptive defense system that can block brute force attacks before they even reach your application.

Step 7: Testing Your Configuration

How to Test the Effectiveness of Your Protection

Once your firewall rules, connection limits, and security profiles are in place, it’s critical to validate that they work as expected. Testing allows you to confirm that brute force attempts are detected, rate-limited, or blocked, and that legitimate users are not affected.

Start with basic checks:

Try logging in with incorrect credentials multiple times in a row. Does the system respond with lockouts or rate-limiting?
Attempt multiple concurrent connections from a single IP. Are they capped or throttled?
Review logs to see if your FortiGate is capturing and classifying these attempts correctly.

Using Brute Force Simulation Tools

To simulate real-world attacks in a controlled environment, consider using safe and legal penetration testing tools:

Hydra: A popular tool for testing login pages and network services like SSH, FTP, and HTTP authentication.
Burp Suite (Community Edition): Can be used to automate login attempts against web applications.
Nmap Scripts: Some Nmap NSE scripts simulate login attempts to test rate-limiting and DoS protection.
Fail2Ban (on target host): You can use it together with FortiGate to observe log-based bans.

⚠️ Note: Never run brute force simulations against production systems or third-party applications without permission. Always test in an isolated, controlled lab environment.

By running these tests, you’ll gain confidence that your FortiGate 60D is actively detecting and blocking brute force behavior, while allowing normal traffic to pass unhindered.

Conclusion and Best Practices

Brute force attacks are simple but powerful, and self-hosted applications are especially vulnerable without proper defenses.

Recap of Key Steps to Protect Your Application

With FortiGate, you have the tools needed to create a strong line of defense. Here’s a quick recap of the key steps:

Harden your FortiGate setup: Update firmware, limit admin access, and secure the management interface.
Set connection limits: Cap the number of sessions per IP and configure DoS thresholds.
Enable IPS and DoS protection: Automatically detect and block brute force attack patterns.
Use logging and alerts: Monitor traffic, identify attack attempts, and receive real-time notifications.
Apply Geo-IP and botnet filters: Reduce attack surface by blocking traffic from high-risk regions and known malicious IPs.
Test your configuration: Use tools to simulate attacks and validate that protection mechanisms are effective.

Additional Tips for Long-Term Security

Use strong, unique passwords for all accounts, especially admin interfaces.
Enable 2FA (Two-Factor Authentication) wherever possible, even for internal admin access.
Regularly audit logs and firewall policies to keep them up to date with your application’s needs.
Stay informed by subscribing to Fortinet advisories and updating your threat intelligence profiles.
Document your firewall policies and response plans so your team knows how to react during an attack.

By combining layered security features with smart configuration and ongoing monitoring, you can effectively protect your self-hosted application against brute force attacks — and ensure it stays online, secure, and trusted by your users.