[FR] Generate investigation guides (#4358)

Removed changes from:
- rules/integrations/endpoint/defense_evasion_elastic_memory_threat_detected.toml
- rules/integrations/endpoint/defense_evasion_elastic_memory_threat_prevented.toml
- rules/integrations/endpoint/elastic_endpoint_security_behavior_detected.toml
- rules/integrations/endpoint/elastic_endpoint_security_behavior_prevented.toml
- rules/integrations/endpoint/execution_elastic_malicious_file_detected.toml
- rules/integrations/endpoint/execution_elastic_malicious_file_prevented.toml
- rules/integrations/endpoint/impact_elastic_ransomware_detected.toml
- rules/integrations/endpoint/impact_elastic_ransomware_prevented.toml
- rules/linux/defense_evasion_selinux_configuration_creation_or_renaming.toml
- rules/linux/privilege_escalation_potential_suid_sgid_exploitation.toml
- rules/linux/privilege_escalation_sudo_token_via_process_injection.toml

(selectively cherry picked from commit fe8c81d)

detection_rules/rule_formatter.py

@@ -249,6 +249,14 @@ def _do_write(_data, _contents):
                 # This will ensure that the output file has the correct number of backslashes.
                 v = v.replace("\\", "\\\\")
 
+            if k == 'osquery' and isinstance(v, list):
+                # Specifically handle transform.osquery queries
+                for osquery_item in v:
+                    if 'query' in osquery_item and isinstance(osquery_item['query'], str):
+                        # Transform instances of \ to \\ as calling write will convert \\ to \.
+                        # This will ensure that the output file has the correct number of backslashes.
+                        osquery_item['query'] = osquery_item['query'].replace("\\", "\\\\")
+
             if isinstance(v, dict):
                 bottom[k] = OrderedDict(sorted(v.items()))
             elif isinstance(v, list):

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "detection_rules"
-version = "0.4.2"
+version = "0.4.3"
 description = "Detection Rules is the home for rules used by Elastic Security. This repository is used for the development, maintenance, testing, validation, and release of rules for Elastic Security’s Detection Engine."
 readme = "README.md"
 requires-python = ">=3.12"

rules/apm/apm_403_response_to_a_post.toml

@@ -2,7 +2,7 @@
 creation_date = "2020/02/18"
 integration = ["apm"]
 maturity = "production"
-updated_date = "2024/05/21"
+updated_date = "2025/01/15"
 
 [rule]
 author = ["Elastic"]
@@ -25,11 +25,45 @@ references = ["https://en.wikipedia.org/wiki/HTTP_403"]
 risk_score = 47
 rule_id = "a87a4e42-1d82-44bd-b0bf-d9b7f91fb89e"
 severity = "medium"
-tags = ["Data Source: APM"]
+tags = ["Data Source: APM", "Resources: Investigation Guide"]
 timestamp_override = "event.ingested"
 type = "query"
 
 query = '''
 http.response.status_code:403 and http.request.method:post
 '''
+note = """## Triage and analysis
+
+> **Disclaimer**:
+> This investigation guide was created using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.
+
+### Investigating Web Application Suspicious Activity: POST Request Declined
+
+Web applications often use POST requests to handle data submissions securely. However, adversaries may exploit this by attempting unauthorized actions, triggering a 403 error when access is denied. The detection rule identifies such anomalies by flagging POST requests that receive a 403 response, indicating potential misuse or probing attempts, thus aiding in early threat detection.
+
+### Possible investigation steps
+
+- Review the source IP address and user agent associated with the POST request to identify any patterns or known malicious actors.
+- Examine the URL or endpoint targeted by the POST request to determine if it is a sensitive or restricted resource.
+- Check the timestamp of the request to see if it coincides with other suspicious activities or known attack patterns.
+- Analyze the frequency and volume of similar 403 POST requests from the same source to assess if this is part of a larger probing or attack attempt.
+- Investigate any recent changes or updates to the web application that might have inadvertently triggered legitimate requests to be denied.
+
+### False positive analysis
+
+- Legitimate API interactions may trigger 403 responses if the API endpoint is accessed without proper authentication or authorization. Review API access logs to identify and whitelist known applications or users that frequently interact with the API.
+- Web application firewalls (WAFs) might block certain POST requests due to predefined security rules, resulting in 403 errors. Analyze WAF logs to determine if specific rules are causing false positives and adjust the ruleset accordingly.
+- Automated scripts or bots performing routine tasks might inadvertently trigger 403 responses. Identify these scripts and ensure they are configured with the necessary permissions or exclude their IP addresses from the detection rule.
+- User error, such as incorrect form submissions or missing required fields, can lead to 403 responses. Educate users on proper form usage and consider implementing client-side validation to reduce these occurrences.
+- Maintenance or configuration changes in the web application might temporarily cause 403 errors. Coordinate with the development or operations team to understand scheduled changes and adjust monitoring rules during these periods.
+
+### Response and remediation
+
+- Immediately review the logs associated with the 403 POST requests to identify the source IP addresses and user agents involved. Block any suspicious IP addresses at the firewall or web application firewall (WAF) to prevent further unauthorized attempts.
+- Conduct a thorough review of the web application's access control policies and permissions to ensure that they are correctly configured to prevent unauthorized actions.
+- Check for any recent changes or updates to the web application that might have inadvertently altered access controls or introduced vulnerabilities, and roll back or patch as necessary.
+- Notify the security operations team to monitor for any additional suspicious activity from the identified IP addresses or similar patterns, and escalate to incident response if further malicious activity is detected.
+- Implement additional logging and monitoring for POST requests that result in 403 responses to enhance detection capabilities and gather more context for future incidents.
+- Review and update the web application firewall (WAF) rules to better detect and block unauthorized POST requests, ensuring that legitimate traffic is not affected.
+- If applicable, engage with the development team to conduct a security review of the application code to identify and fix any potential vulnerabilities that could be exploited by attackers."""
 

rules/apm/apm_405_response_method_not_allowed.toml

@@ -2,7 +2,7 @@
 creation_date = "2020/02/18"
 integration = ["apm"]
 maturity = "production"
-updated_date = "2024/05/21"
+updated_date = "2025/01/15"
 
 [rule]
 author = ["Elastic"]
@@ -25,11 +25,45 @@ references = ["https://en.wikipedia.org/wiki/HTTP_405"]
 risk_score = 47
 rule_id = "75ee75d8-c180-481c-ba88-ee50129a6aef"
 severity = "medium"
-tags = ["Data Source: APM"]
+tags = ["Data Source: APM", "Resources: Investigation Guide"]
 timestamp_override = "event.ingested"
 type = "query"
 
 query = '''
 http.response.status_code:405
 '''
+note = """## Triage and analysis
+
+> **Disclaimer**:
+> This investigation guide was created using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.
+
+### Investigating Web Application Suspicious Activity: Unauthorized Method
+
+Web applications often restrict HTTP methods to protect resources, allowing only specific actions like GET or POST. Adversaries may exploit misconfigurations by attempting unauthorized methods, potentially revealing vulnerabilities or sensitive data. The detection rule identifies such attempts by flagging HTTP 405 responses, indicating a method is not permitted, thus highlighting potential misuse or probing activities.
+
+### Possible investigation steps
+
+- Review the web server logs to identify the source IP address associated with the HTTP 405 response to determine if the request originated from a known or suspicious source.
+- Analyze the request headers and payload associated with the 405 response to understand what unauthorized method was attempted and if there are any patterns or anomalies.
+- Check the application configuration to verify which HTTP methods are allowed for the resource in question and assess if there are any misconfigurations that could be exploited.
+- Investigate if there are multiple 405 responses from the same source IP or user agent, which could indicate probing or automated scanning activity.
+- Correlate the 405 response events with other security alerts or logs to identify any related suspicious activities or potential attack vectors.
+
+### False positive analysis
+
+- Routine API calls using unsupported methods may trigger 405 responses. Review API documentation to ensure correct methods are used and adjust monitoring to exclude these known patterns.
+- Automated tools or scripts might inadvertently use incorrect HTTP methods, leading to false positives. Identify and update these tools to use appropriate methods, or whitelist their IP addresses if they are known and trusted.
+- Web crawlers or bots might attempt unsupported methods as part of their scanning process. Configure your monitoring system to recognize and exclude these benign activities based on user-agent strings or IP ranges.
+- Development and testing environments often experiment with various HTTP methods, resulting in 405 responses. Implement rules to exclude these environments from production monitoring to reduce noise.
+- Legacy systems or applications might not support certain HTTP methods, causing frequent 405 errors. Document these systems and create exceptions in your monitoring to prevent unnecessary alerts.
+
+### Response and remediation
+
+- Immediately review the web server and application logs to identify the source IP address and user agent associated with the 405 response. Block the IP address if it is determined to be malicious or part of a known attack pattern.
+- Conduct a security assessment of the web application's configuration to ensure that only necessary HTTP methods are enabled for each resource. Disable any methods that are not required for the application's functionality.
+- Implement or update web application firewall (WAF) rules to block unauthorized HTTP methods and monitor for repeated attempts from the same source.
+- Notify the security operations team to monitor for any additional suspicious activity from the identified source or similar patterns, and escalate to incident response if further malicious activity is detected.
+- Review and update the application's security policies and access controls to ensure they align with best practices and prevent unauthorized method usage.
+- Conduct a vulnerability assessment of the web application to identify and remediate any potential security weaknesses that could be exploited by unauthorized HTTP methods.
+- Document the incident, including the response actions taken, and update the incident response plan to improve future detection and response capabilities for similar threats."""
 

rules/apm/apm_sqlmap_user_agent.toml

@@ -2,7 +2,7 @@
 creation_date = "2020/02/18"
 integration = ["apm"]
 maturity = "production"
-updated_date = "2024/05/21"
+updated_date = "2025/01/15"
 
 [rule]
 author = ["Elastic"]
@@ -25,11 +25,46 @@ references = ["http://sqlmap.org/"]
 risk_score = 47
 rule_id = "d49cc73f-7a16-4def-89ce-9fc7127d7820"
 severity = "medium"
-tags = ["Data Source: APM"]
+tags = ["Data Source: APM", "Resources: Investigation Guide"]
 timestamp_override = "event.ingested"
 type = "query"
 
 query = '''
 user_agent.original:"sqlmap/1.3.11#stable (http://sqlmap.org)"
 '''
+note = """## Triage and analysis
+
+> **Disclaimer**:
+> This investigation guide was created using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.
+
+### Investigating Web Application Suspicious Activity: sqlmap User Agent
+
+Sqlmap is a widely-used open-source tool designed to automate the detection and exploitation of SQL injection vulnerabilities in web applications. Adversaries may exploit sqlmap to extract sensitive data or manipulate databases. The detection rule identifies suspicious activity by monitoring for specific user agent strings associated with sqlmap, flagging potential unauthorized testing or attacks on web applications.
+
+### Possible investigation steps
+
+- Review the logs to identify the source IP address associated with the user agent string "sqlmap/1.3.11#stable (http://sqlmap.org)" to determine the origin of the suspicious activity.
+- Check for any other user agent strings or unusual activity from the same IP address to assess if there are additional signs of probing or attacks.
+- Investigate the targeted web application endpoints to understand which parts of the application were accessed and if any SQL injection attempts were successful.
+- Correlate the timestamp of the detected activity with other security logs or alerts to identify any concurrent suspicious activities or anomalies.
+- Assess the potential impact by reviewing database logs or application logs for any unauthorized data access or modifications during the time of the detected activity.
+- Consider blocking or monitoring the identified IP address to prevent further unauthorized access attempts, if deemed malicious.
+
+### False positive analysis
+
+- Development and testing environments may use sqlmap for legitimate security testing. To handle this, create exceptions for known IP addresses or user agents associated with internal security teams.
+- Automated security scanners or vulnerability assessment tools might mimic sqlmap's user agent for testing purposes. Identify and whitelist these tools to prevent unnecessary alerts.
+- Some web application firewalls or intrusion detection systems may simulate sqlmap activity to test their own detection capabilities. Coordinate with your security infrastructure team to recognize and exclude these activities.
+- Educational institutions or training environments might use sqlmap for teaching purposes. Establish a list of authorized users or networks to exclude from alerts.
+- Ensure that any third-party security service providers are recognized and their activities are documented to avoid misidentification as threats.
+
+### Response and remediation
+
+- Immediately block the IP address associated with the sqlmap user agent activity to prevent further unauthorized access attempts.
+- Review and analyze web server logs to identify any additional suspicious activity or patterns that may indicate further exploitation attempts.
+- Conduct a thorough assessment of the affected web application to identify and patch any SQL injection vulnerabilities that may have been exploited.
+- Reset credentials and enforce strong password policies for any database accounts that may have been compromised.
+- Implement web application firewall (WAF) rules to detect and block SQL injection attempts, including those using sqlmap.
+- Notify the security operations team and relevant stakeholders about the incident for awareness and further investigation.
+- Document the incident details and response actions taken for future reference and to enhance incident response procedures."""