Securing a Compromised Linux Web Server: A Practical Forensic Investigation Guide

 
KIRAN KUMAR K3
Overview

When an attacker can upload an image containing PHP code (e.g., image.jpg that actually has <?php … ?>), and the application stores it in a web-accessible folder where PHP code runs, they can gain Remote Command Execution (RCE) via a web shell.

This article gives you a complete, hands-on playbook to:

 Note: This incident is not related to SQL Server or xp_cmdshell. Those are Microsoft SQL Server concepts and do not apply to Linux + PHP web servers.

 KIRAN KUMAR K3


 Quick Navigation

  • #environment--symptoms
  • #phase-1-triage--containment
  • #phase-2-evidence-collection
  • #phase-3-threat-hunting-find-web-shells
  • #phase-4-hardening--fix
  • #phase-5-verification--retest
  • #incident-report-template
  • #closure-email-template
  • #faq
  • #best-practices--next-steps

Environment & Symptoms

  • OS: Linux (e.g., Ubuntu/Debian/CentOS/RHEL)
  • Stack: Nginx/Apache + PHP (FPM or Apache module)
  • App: WordPress
  • Indicators:
    • Unexpected modifications inside wp-content/
    • Suspicious files in wp-content/uploads/
    • *.php.jpg, *.png.php, .phtml files
    • Access logs showing cmd=, system(, base64_decode( queries

Phase 1: Triage & Containment

1. Identify web server & PHP mode

# Which web server?
ps -ef | egrep 'apache|httpd|nginx'

# PHP installed and version?
php -v

# Is PHP-FPM running?
ps -ef | grep php

2. Quickly see recently modified items (working directory = web root)

# Shows oldest → newest; last line is most recent
ls -ltr

In WordPress, wp-content being the most recently modified is a strong red flag.

3. Backup any suspicious file (for forensics)

# Example path—adjust to your file
mkdir -p /root/incident-backup
cp /var/www/html/wp-content/uploads/suspicious.jpg /root/incident-backup/

4. Remove the malicious file (after backup)

rm -f /var/www/html/wp-content/uploads/suspicious.jpg
``


Phase 2: Evidence Collection

Keep enough data for RCA and audit trail.

  • Timestamps: When directories/files changed (ls -ltr, stat <file>)
  • Access logs: IPs, User‑Agents, request paths, parameters
  • Server config snapshots: Apache/Nginx vhosts, PHP php.ini
  • File hashes: SHA256 of malicious files

# File timestamps
stat /var/www/html/wp-content/uploads/suspicious.jpg

# Access log examples
tail -n 200 /var/log/nginx/access.log
# or
tail -n 200 /var/log/apache2/access.log

# Hash the file (if still present or from backup)
sha256sum /root/incident-backup/suspicious.jpg


Phase 3: Threat Hunting (Find Web Shells)

A. Most recently modified files

# Modified in last 24 hours
find . -type f -mmin -1440 -ls

# Modified in last 7 days
find . -type f -mtime -7 -ls

# Single most recently modified file
find . -type f -printf '%TY-%Tm-%Td %TH:%TM %p\n' | sort | tail -1

B. Scan for PHP inside uploads (classic web-shell technique)

# PHP tags inside uploads/themes/plugins
grep -R "<?php" wp-content/

# High-signal function hits (not always malicious, but useful pivots)
grep -R "system(" wp-content/
grep -R "exec(" wp-content/
grep -R "shellexec(" wp-content/
grep -R "passthru(" wp-content/
grep -R "procopen(" wp-content/
grep -R "base64_decode(" wp-content/
``

C. Suspicious extensions

find wp-content -type f \( -name ".php.jpg" -o -name ".png.php" -o -name "*.phtml" \) -ls

D. Look for hidden files

find wp-content -type f -name ".*" -ls


Phase 4: Hardening & Fix

1) Fix directory & file permissions (WordPress-safe)

# Directories → 755
find wp-content -type d -exec chmod 755 {} \;

# Files → 644
find wp-content -type f -exec chmod 644 {} \;

❌ Avoid 777 and world-writable perms.\ Ownership: Prefer root:www-data (or root:apache / root:nginx) for content you don’t expect PHP to modify.

# Example (Debian/Ubuntu with Nginx/Apache running as www-data)
chown -R root:www-data wp-content

2) Disable PHP execution in upload directories

Apache (.htaccess inside uploads):

# File: wp-content/uploads/.htaccess
php_flag engine off
<FilesMatch ".php$">
    Deny from all
</FilesMatch>

Restart Apache:

systemctl restart apache2   # Debian/Ubuntu
# or
systemctl restart httpd     # RHEL/CentOS

Nginx (site server block):

# Add inside the appropriate server {} block
location ~ ^/wp-content/uploads/.\.php$ {
    deny all;
    return 403;
}
# Optional: prevent listing / restrict access
location /wp-content/uploads/ {
    autoindex off;
    try_files $uri =404;
}

Restart Nginx:

systemctl restart nginx
``

3) Harden PHP (defense-in-depth)

Locate php.ini:

php -i | grep php.ini

Edit and set:

disablefunctions = exec,system,shellexec,passthru,popen,proc_open
``

Restart PHP services (adjust to your stack):

systemctl restart php-fpm
systemctl restart apache2
systemctl restart nginx
``

4) App-layer upload validation (developer fix)

  • Allowlist extensions only (jpg, png, pdf)
  • Validate MIME type properly
  • Validate magic bytes (file signature)
  • Rename files on server (randomized)
  • Store uploads outside web root and serve via controlled handler
  • Strip metadata and reject double extensions (.php.jpg, .png.php)

5) Web Application Firewall (WAF) (recommended)

  • Enable ModSecurity (OWASP CRS) for Apache/Nginx
  • Add rules to block suspicious params: cmd=, system(, base64_decode(

Phase 5: Verification & Retest

  1. Try to access a PHP file inside uploads (should NOT execute): https://your-site/wp-content/uploads/test.php Expected: 403/404 or file download—not execution.

  2. Re-run searches:

    grep -R "<?php" wp-content/uploads/
    find wp-content/uploads -type f -mtime -1 -ls

  3. Review logs for post-fix attempts:

    tail -n 200 /var/log/nginx/access.log
    # or
    tail -n 200 /var/log/apache2/access.log


Incident Report Template

Title: Malicious File Upload → PHP Web Shell on WordPress (Linux)

Date/Time Detected: YYYY‑MM‑DD HH:MM (TZ)\ Affected Host: hostname / IP\ Application: WordPress\ Summary:\ A malicious image containing embedded PHP code was uploaded to a web-accessible directory. Due to PHP execution being enabled in wp-content/uploads/, the attacker could execute OS commands remotely (RCE) via a web shell.

Impact:

  • Potential server compromise
  • Unauthorized command execution
  • Possible data exposure (depending on shell activity)

Indicators & Evidence:

  • Recent modification in wp-content at <timestamp>
  • Suspicious file(s): <path> (hash: <sha256>)
  • Access logs showing calls to uploaded file(s), parameters like cmd=, system(, base64_decode(

Root Cause:\ Insecure file upload handling + executable PHP in upload directories.

Containment Actions:

  • Backed up and removed malicious file(s)
  • Disabled PHP execution in uploads
  • Hardened permissions (dirs 755, files 644)
  • Restricted dangerous PHP functions

Eradication & Hardening:

  • Searched and removed additional shells
  • Updated .htaccess/Nginx rules
  • Implemented MIME + magic byte validation
  • Considered moving uploads outside web root

Verification:

  • Attempt to execute PHP in uploads → blocked (403/404)
  • No new suspicious modifications observed
  • Logs reviewed with no further exploitation

Status: Closed / Monitoring

Recommendations:

  • Enable WAF (ModSecurity + OWASP CRS)
  • Regular integrity checks (AIDE/Tripwire)
  • Keep WordPress core/plugins/themes updated
  • Enforce least-privilege filesystem permissions

Closure Email Template

Subject: [Closed] Malicious File Upload Incident – WordPress (Linux)

Dear Team,

The incident involving a malicious file upload leading to potential PHP code execution has been investigated and remediated.

Actions Completed:

  • Malicious file(s) removed after preservation for evidence
  • Disabled PHP execution in wp-content/uploads/
  • Hardened directory and file permissions
  • Restricted dangerous PHP functions in php.ini
  • Performed wide search for additional shells (none found)
  • Reviewed web server logs and confirmed no ongoing exploitation

Verification:

  • Attempts to execute PHP within uploads now return 403/404
  • No anomalous modifications post-remediation

Next Steps:

  • Implement/maintain WAF (ModSecurity + OWASP CRS)
  • Validate uploads via MIME + magic bytes and store outside web root
  • Maintain timely updates for WordPress core, plugins, and themes

Regards,\ Kiran Kumar K\ Junior Security Analyst, Information Security Office, IISc


FAQ

Q1: Is this related to xp_cmdshell?\ A: No. xp_cmdshell is a Microsoft SQL Server feature. This incident is Linux + PHP web shell via insecure upload handling.

Q2: Can I just delete the malicious image and be done?\ A: No. You must disable PHP execution in uploads, fix permissions, search for other shells, and verify.

Q3: How do I prevent this permanently?\ A: Enforce strict server-side validation (allowlist + MIME + magic bytes), store uploads outside web root, disable PHP execution in uploads, and deploy a WAF.


Best Practices & Next Steps

  • Keep WordPress core/plugins/themes updated
  • Use least privilege permissions (avoid 777)
  • Implement daily integrity scans
  • Enable WAF with OWASP CRS
  • Monitor logs for anomalous requests
  • Schedule recurring security reviews after major updates

Handy Command Reference (Cheat Sheet)

# Most recently modified (oldest → newest)
ls -ltr

# Files modified in last 24 hours
find . -type f -mmin -1440 -ls

# Most recent single file with timestamp
find . -type f -printf '%TY-%Tm-%Td %TH:%TM %p\n' | sort | tail -1

# Search for PHP signatures in WP content
grep -R "<?php" wp-content/
grep -R "system(" wp-content/
grep -R "exec(" wp-content/
grep -R "shellexec(" wp-content/
grep -R "base64decode(" wp-content/

# Suspicious extensions
find wp-content -type f \( -name ".php.jpg" -o -name ".png.php" -o -name ".phtml" \) -ls

# Permissions hardening
find wp-content -type d -exec chmod 755 {} \;
find wp-content -type f -exec chmod 644 {} \;
chown -R root:www-data wp-content

# Apache: block PHP execution in uploads
# (Create wp-content/uploads/.htaccess)
# phpflag engine off
# <FilesMatch ".php$">
#     Deny from all
# </FilesMatch>

# Nginx: block PHP in uploads (in server block)
# location ~ ^/wp-content/uploads/.*.php$ {
#     deny all;
#     return 403;
# }
# location /wp-content/uploads/ {
#     autoindex off;
#     tryfiles $uri =404;
# }

# Restart services
systemctl restart apache2  # or httpd
systemctl restart nginx
systemctl restart php-fpm



Previous Post
No Comment
Add Comment
comment url