How to Analyze Application Logs and Extract Actionable Insights
By Edgar Mlowe
February 28, 2025
Photo by Timo C. Dinger on Unsplash
Logs often accumulate unnoticed—until something breaks. Then, they suddenly become vital tools for diagnosing issues. By combining just a few command-line techniques, you can quickly spot recurring problems, identify suspicious activity, and strengthen your application’s defenses.
Below is a sample error log we’ll reference in the examples:
[2025-01-01 12:34:56] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:34:56] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:34:57] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:35:00] [client 10.0.0.2:6789] PHP Warning: Division by zero in /var/www/html/script.php on line 23 | referer: http://example.com/script.php
[2025-01-01 13:35:01] [client 10.0.0.2:6789] PHP Warning: Division by zero in /var/www/html/script.php on line 23 | referer: http://example.com/script.php
[2025-01-01 13:36:10] [client 192.168.1.1:12345] PHP Fatal error: Call to undefined function baz() in /var/www/html/lib.php on line 78 | referer: http://example.com/index.php
[2025-01-01 14:36:11] [client 172.16.0.5:54321] PHP Fatal error: Call to undefined function baz() in /var/www/html/lib.php on line 78 | referer: http://example.com/index.php
A Note on Commands
Throughout this blog, you’ll notice repeated use of commands like awk, grep, sed, uniq, and sort. These tools are indispensable for log analysis, allowing you to filter, transform, and summarize data efficiently. Here are some tips for learning these tools:
- Use
man <command>(e.g.,man awk) to access the manual and dive deeper into a specific command - Experiment with small examples to understand how commands like
awkprocess patterns or howgrepextracts data - For a broader look at how these commands enhance productivity, check out my blog post: Practical Linux Command Line Tips for Productivity and Efficiency
These commands are not just for log analysis—they’re powerful for any data manipulation task you encounter. With practice, they can become essential to your workflow.
1. Debugging Application Errors
1.1 Summarize Frequent Issues
sed -E 's/^\[.*\] //' error.log | sort | uniq -c | sort -nr
- Purpose: Remove timestamps/client’s IP, then sort and count duplicates
- Outcome: A quick snapshot of which errors appear most often
3 PHP Notice: Undefined variable $foo ...
2 PHP Warning: Division by zero ...
2 PHP Fatal error: Call to undefined function baz() ...
1.2 Categorize Errors by Type
grep -oE 'PHP [^:]+' error.log | sort | uniq -c | sort -nr
- Purpose: Extract error types (e.g., Notice, Warning, Fatal error)
- Outcome: Know whether notices, warnings, or fatal errors dominate
3 Notice
2 Warning
2 Fatal error
1.3 Find Problematic Scripts
awk -F'in ' '{print $2}' error.log | awk '{print $1}' | sort | uniq -c | sort -nr
- Purpose: Identify which files generate the most errors
- Outcome: Pinpoint error hotspots for focused debugging
3 /var/www/html/index.php
2 /var/www/html/script.php
2 /var/www/html/lib.php
2. Monitoring System Behavior
2.1 Track Problematic IPs
awk -F'\[client ' '{print $2}' error.log | awk -F':' '{print $1}' | sort | uniq -c | sort -nr
- Purpose: Count how many errors each IP triggers
- Outcome: Identify suspicious or high-traffic IPs
4 192.168.1.1
2 10.0.0.2
1 172.16.0.5
2.2 Spot High-Error Time Periods
awk -F'[][]' '{split($2, time, ":"); print $1, time[1]":"time[2]":00"}' error.log | sort | uniq -c | sort -nr
- Purpose: Group errors by minute, revealing spikes or trends
- Outcome: Correlate error surges with deployments or traffic peaks
3 2025-01-01 12:34:00
1 2025-01-01 14:36:00
1 2025-01-01 13:36:00
...
2.3 Analyze IP–Referer Pairs
grep "referer:" error.log | awk -F'\\[client ' '{print $2}' | awk -F':| referer: ' '{print $1, $3}' | sort | uniq -c | sort -nr
- Purpose: Match IP addresses with the URLs they refer from
- Outcome: Detect repeat offenders or malicious traffic patterns
3 192.168.1.1 Undefined variable $foo ...
2 10.0.0.2 Division by zero ...
1 172.16.0.5 Call to undefined function baz() ...
3. Enhancing Security
3.1 Watch Sensitive URLs
grep -E "admin|login" script_error_frequency.txt | sort -nr
- Purpose: Flag frequent hits on admin or login pages
- Outcome: Gauge whether sensitive endpoints are under attack
2 /var/www/html/login.php
2 /var/www/html/admin.php
3.2 Pinpoint Critical Times
grep -E "admin|login" error.log | awk -F'[][]' '{split($2, time, ":"); print $1, time[1]":"time[2]":00"}' | sort | uniq -c | sort -nr
- Purpose: Identify specific time windows for sensitive access
- Outcome: Cross-reference suspicious activity with your security logs
2 2025-01-01 12:34:00
1 2025-01-01 13:35:00
1 2025-01-01 12:35:00
Conclusion
These commands demonstrate how quickly you can glean insights from logs. With a little creativity, you can expand them to track response times, detect performance bottlenecks, and safeguard critical endpoints. Whether you’re tackling PHP errors or any other type of log data, the same principles apply: filter, sort, count, and investigate.
Curious to learn more? Combine these strategies with automation tools, integrate them into CI/CD pipelines, or hook them up to visual dashboards. Your logs will become a gold mine of actionable information.
logging automation monitoring security php shell
Comments