How to Install and Configure ModSecurity with NGINX

ModSecurity 3.x (libmodsecurity) requires several dependencies. Install them based on your distribution:

Ubuntu/Debian:

sudo apt update
sudo apt install -y apt-utils autoconf automake build-essential \
  git libcurl4-openssl-dev libgeoip-dev liblmdb-dev libpcre2-dev \
  libtool libxml2-dev libyajl-dev pkgconf wget zlib1g-dev

RHEL/CentOS/Rocky:

sudo dnf install -y gcc-c++ flex bison yajl curl-devel zlib-devel \
  pcre2-devel autoconf automake git curl make libxml2-devel \
  pkgconfig libtool httpd-devel lmdb-devel GeoIP-devel

Clone the ModSecurity repository and build the library:

cd /opt
sudo git clone --depth 1 -b v3/master --single-branch https://github.com/owasp-modsecurity/ModSecurity
cd ModSecurity

# Initialize and update submodules
sudo git submodule init
sudo git submodule update

# Build and install
sudo ./build.sh
sudo ./configure
sudo make
sudo make install

The NGINX ModSecurity connector integrates libmodsecurity with NGINX as a dynamic module:

# Clone the connector
cd /opt
sudo git clone --depth 1 https://github.com/owasp-modsecurity/ModSecurity-nginx.git

# Get your NGINX version
nginx -v

# Download matching NGINX source (replace version as needed)
wget http://nginx.org/download/nginx-1.24.0.tar.gz
tar -xzf nginx-1.24.0.tar.gz
cd nginx-1.24.0

# Build the dynamic module
./configure --with-compat --add-dynamic-module=../ModSecurity-nginx
make modules

# Copy module to NGINX modules directory
sudo cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules/

Load the ModSecurity module in your NGINX configuration:

Add this line at the very top of /etc/nginx/nginx.conf (before the events block):

load_module modules/ngx_http_modsecurity_module.so;

Test the configuration:

sudo nginx -t

You should see: syntax is ok and test is successful.

Set up the ModSecurity configuration directory and base configuration:

# Create directory structure
sudo mkdir -p /etc/nginx/modsec

# Copy the recommended configuration
sudo cp /opt/ModSecurity/modsecurity.conf-recommended /etc/nginx/modsec/modsecurity.conf

# Copy the Unicode mapping file
sudo cp /opt/ModSecurity/unicode.mapping /etc/nginx/modsec/

Edit /etc/nginx/modsec/modsecurity.conf and change the engine from detection to blocking:

# Change this line:
SecRuleEngine DetectionOnly

# To (when ready for production):
SecRuleEngine On

The OWASP Core Rule Set provides comprehensive protection against common attacks:

cd /etc/nginx/modsec
sudo git clone https://github.com/coreruleset/coreruleset.git
cd coreruleset

# Create configuration from example
sudo cp crs-setup.conf.example crs-setup.conf

Create a main rules file at /etc/nginx/modsec/main.conf:

Include /etc/nginx/modsec/modsecurity.conf
Include /etc/nginx/modsec/coreruleset/crs-setup.conf
Include /etc/nginx/modsec/coreruleset/rules/*.conf

Enable ModSecurity for your sites by adding directives to your server blocks:

server {
    listen 80;
    server_name example.com;

    # Enable ModSecurity
    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;

    location / {
        proxy_pass http://backend;
        # ... other directives
    }
}

Reload NGINX to apply changes:

sudo nginx -t && sudo systemctl reload nginx

Verify ModSecurity is working by sending a test attack:

# This should be blocked (SQL injection attempt)
curl -I 'http://your-server.com/?id=1%20OR%201=1'

# This should be blocked (XSS attempt)
curl -I 'http://your-server.com/?q=<script>alert(1)</script>'

Check the ModSecurity audit log:

sudo tail -f /var/log/modsec_audit.log

You should see entries showing the blocked requests with rule IDs and attack details.

Configure ModSecurity logging in /etc/nginx/modsec/modsecurity.conf:

# Audit log settings
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial
SecAuditLog /var/log/modsec_audit.log

# Debug log (disable in production)
SecDebugLog /var/log/modsec_debug.log
SecDebugLogLevel 0

Create the log files with correct permissions:

sudo touch /var/log/modsec_audit.log
sudo chown www-data:www-data /var/log/modsec_audit.log

After running in DetectionOnly mode, review logs for false positives. Common tuning approaches:

Exclude specific rules:

# In /etc/nginx/modsec/crs-setup.conf or a custom file:
SecRuleRemoveById 942100  # Disable specific rule
SecRuleRemoveByTag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"  # Disable by tag

Exclude rules for specific URLs:

SecRule REQUEST_URI "@beginsWith /api/upload" \
  "id:1000,phase:1,pass,nolog,\
   ctl:ruleRemoveById=200002"

Adjust paranoia level:

In crs-setup.conf, set the paranoia level (1=low false positives, 4=maximum security):

SecAction "id:900000,phase:1,pass,t:none,\
  setvar:tx.blocking_paranoia_level=1"