Public Draft 1 (October 7, 2005)
Copyright © 2005 Web Application Security Consortium (www.webappsec.org)
Table of Contents
Web application firewalls (WAF) represent a new breed of information security technology designed to protect web sites from attack. WAF solutions are capable of preventing attacks that network firewalls and intrusion detection systems can't, and they do not require modification of application source code. As today's web application attacks expand and their relative level of sophistication increases, it is vitally important to develop a standardized criteria for product evaluation. How else can we accurately compare or measure the performance of a particular solution?
The goal of this project is to develop a detailed web application firewall evaluation criteria; a testing methodology that can be used by any reasonably skilled technician to independently assess the quality of a WAF solution.
Current categories are as follows:
Deployment Architecture
HTTP Support
Detection Techniques
Protection Techniques
Logging
Reporting
Management
We expect to cover the following categories in the subsequent releases:
XML support
Performance
Compliance, certifications, and interoperability
This document is a result of team effort. The following people have contributed their time and expertise to the project:
Robert Auger (SPI Dynamics)
Ryan C. Barnett (Center for Internet Security)
Anton Chuvakin (Individual)
Matthieu Estrade (Bee Ware)
Jeremiah Grossman (WhiteHat Security)
Achim Hoffmann (Individual)
Dave Jevans (Teros)
Amit Klein (Individual)
Mark Kraynak (Imperva)
Ofer Maor (Hacktics)
Sylvain Maret (e-Xpert Solutions)
Rafael San Miguel (daVinci Consulting)
Ivan Ristic (Thinking Stone) [Project Leader]
Ory Segal (Watchfire)
Ofer Shezaf (Breach)
If you wish to comment on the evaluation criteria please join the
team mailing list by contacting Ivan Ristic
(<ivanr@webkreator.com>).
This section highlights the questions key to determining the feasibility of web application firewall deployment in a given environment.
Describe which of the following modes of operation apply to the WAF:
Passive. Can be configured to passively listen to the traffic.
Bridge. Can be installed as a transparent bridge. Can it be configured to fail open?
Router. Network must be reconfigured to direct traffic through the WAF.
Reverse Proxy. Traffic is re-directed to flow through the WAF by making changes to DNS configuration or by re-directing the traffic on the network level.
Embedded. WAF is installed as a web server plug-in. Which web servers are supported? Explain the level of integration with the web server. Some embedded web application firewalls may only tap into the communication channel and do everything themselves. Others may rely on the web server to do as much of the work as possible. Both approaches have their advantages and disadvantages.
SSL is often used to protect traffic coming from and to web applications. While this type of protection achieves the goal of data protection, it hides the data from the protection systems (e.g. intrusion detection systems, web application firewalls) at the same time. Since SSL is in widespread use--in fact, all really important deployments use it--if a WAF cannot get to the traffic then it will be unable to perform its function.
Describe how the WAF can be deployed to access the protected data:
Terminates SSL. The network needs to be re-configured to move the SSL operations to the WAF itself. WAF decrypts the encrypted traffic to get access to the HTTP data. The communication between the WAF and the web server can be in plain-text, or SSL-encrypted.
Passively decrypts SSL. Configured with a copy of the web server's SSL private key, WAF is capable of decrypting the SSL traffic. The original data stream travels to the web server unaffected, where it is decrypted and processed by the web server.
Not Applicable. Working embedded in the web server, WAF can be positioned to work just after SSL data is decrypted into plain-text.
If WAF is capable of blocking the offending traffic describe the nature of the blocking functionality:
Complete blocking. Attack are prevented from reaching the destination. A WAF that works as a reverse proxy can simply inspect the request before it is forwarded to the web server. A WAF that works on the network level must buffer the data packets until enough of the HTTP request is reconstructed to achieve complete blocking.
Partial blocking. Partial attacks may reach the destination. This may happen, for example, if a request is very large, too large to allow for buffering.
Blocking via a third-party device. Blocking via a third-party device may be a more effective way of blocking, especially for WAFs that work on the HTTP level (embedded or reverse proxy-based). A third-party device may also be in a position to block the traffic more effectively.
Describe the scope of blocking capabilities:
Blocks the HTTP request.
Blocks the connection.
Blocks the IP address.
Blocks the application session.
Blocks the application user.
Describe how WAF is delivered:
Appliance. Software bundled with hardware. Appliances are usually highly optimised. They may contain specialised hardware components to increase performance. Are there other optional hardware components that may further increase performance?
Software only. WAF is delivered as software application that can be installed on a generic computer. Describe the reference hardware configuration. Are there other optional hardware components that can increase performance further?
There are two aspects of the high availability requirement. One is to prevent the web application firewall from becoming a single point of failure. The other, to allow the WAF to scale and remain useful for very busy sites. The questions are as follows:
Does the WAF support multi-node operation? If not, can it be configured to fail open?
In a two-node high availability system, can the second node share the load?
Is the node state shared? If the state is shared then a node can go down without any impact on the system.
Can multiple WAF nodes be configured to work in a cluster (of more than two nodes)?
The following encoding types should be supported:
Supports
application/x-www-form-urlencodedencodingSupports
multipart/form-dataencodingSupports v0 cookies
Supports v1 cookies
Supports chunked encoding in requests
Supports chunked encoding in responses
Supports request compression
Supports response compression
Can restrict methods used
Can restrict protocols and protocol versions used
Strict request validation per-RFC
Validate URL-encoded characters
Validation of the non-standard
%uXXYYencodingValidation of UTF-8 content
Can enforce cookie types used (e.g. only allow v1 cookies)
The following protocol limits should be supported:
Can restrict request method length
Can restrict request line length
Can restrict request URI length
Can restrict query string length
Can restrict protocols length
Can restrict the number of headers
Can restrict header name length
Can restrict header value length
Can restrict request body length
Can restrict parameter name length
Can restrict parameter value length
Can restrict the number of parameters
Can restrict combined parameter length (names and values included)
Support for
POSTfile uploads (multipart/form-dataencoding)Support for
PUTfile uploadsIndividual file size can be restricted
Combined file size can be restricted
The number of files in a request can be restricted
Web Application Firewall must evaluate request data in the character encoding used by the application in order to provide adequate protection.
Web systems are often designed as a combination of many related systems. This allows the attackers to attempt evasion by transforming the attack payload into a form that appears harmless to the web application firewall. The large variety of backend systems used makes the work of WAFs very difficult. In order to make rules and signatures useful WAFs must attempt to detect evasion attempts and transform input data into a normalised form.
Does the WAF support the following normalisation methods:
URL-encoding
UTF-8 encoding
Null byte string termination
Self-referencing paths (i.e. the use of
/./)Path back-references (i.e. the use of
/../)Mixed case
Excessive use of whitespace
Comment removal (e.g. convert
DELETE/**/FROMtoDELETE FROM)Conversion of (Windows-supported) backslash characters into forward slash characters.
Conversion of IIS-specific Unicode encoding (
%uXXYY)
When negative security model is used transactions are allowed by default. Only those transactions that contain attacks are rejected. The success of this model depends on the WAF being able to detect bad requests.
The questions are as follows:
Signature-based. Signature-based products usually detect attacks by performing a string or a regular expression match against traffic.
Rule-based. Rules are similar to signatures but allow for a more complex logic to be formed (e.g. logical
AND, logicalOR). They also allow for specific parts of each transaction to be targeted in a rule.
Positive security model denies all transactions by default, but uses rules to allow only those transactions that are known to be valid and safe. This approach can be more efficient (fewer rules to process per transaction) and more secure, but it requires very good understanding of the applications being protected. The positive security model can also be more difficult to maintain if the web application changes frequently.
The questions in this section are as follows:
Is positive security model supported?
Can positive security model be configured manually?
Can positive security model be configured automatically (training)?
Does the tool support model updates (without re-training)?
Positive security model can be achieved using strict content-validation policies, or using statistical analysis or neural networks in an approach commonly referred to as anomaly-based detection.
Signature/rule database:
Does the product come with a database of signatures (or rules), which are designed to detect known problems and attacks.
How many products does the database support?
How often is the database updated?
Is the database cross-referenced to operating systems, applications, and versions, allowing the administrator to apply only the applicable signatures?
Common web application problems identified so far:
Brute Force Attacks
Insufficient Authentication
Insufficient Authorisation
Weak Password Recovery Validation
Credential/Session Prediction
Insufficient Session Expiration
Session Fixation
Content Spoofing / Cross-Site Scripting
Command Execution / Buffer Overflow / OS Commanding / Format String Attack
LDAP Injection / SQL Injection
SSI Injection
XPath Injection
Directory Indexing
Information Leakage
Path Traversal
Predictable Resource Location
Abuse of Functionality
Denial of Service
Insufficient Anti-Automation
Insufficient Process Validation
NTLM Authentication Connection Sharing
Insecure Indexing
DOM-based XSS
Cross-Site Request Forgery
HTTP Response Splitting
HTTP Request Smuggling
Unique transaction IDs are assigned to every HTTP transaction. A transaction is one request and response pair.
Access logs refer to a record of all transactions that went through the WAF. Access logs are most commonly delivered as files, although other forms are also in use (e.g. databases).
Access logs can be exported as files.
Commonly-used log formats are supported.
Access log format can be customised.
Access logs can be sent to the logging server via Syslog.
Access logs can periodically be uploaded to the logging server (e.g. via FTP or SCP).
Event logs refer to a record of all suspicious transactions. Events logs are commonly delivered as files, or are stored in a database.
We are expecting web application firewalls to support the following notification methods:
Email
Syslog
SNMP Trap
OPSEC Event Logging API (ELA - http://www.opsec.com/intro/sdk_overview.html#ela)
Periodic file upload via FTP or SCP.
Full transaction logs must include complete HTTP requests and responses. The emphasis is on the request body, with the option to record the response body too. The requirements are as follows:
The transaction log format should be well documented.
It should be possible to configure the detail level. For example, to allow users to record request bodies but not record response bodies. A really smart tool can be configured to allow with limited detail on non-suspicious transactions, but to log with full detail of suspicious details.
It should be possible to configure which transactions are logged. For example, a user may want to log only the suspicious transactions, or only the transactions that involve dynamic processing on the server.
The following are requirements for log access:
Logs can be exported (as files) via FTP or SCP, periodically or on demand.
Logs can be accessed directly in the a database (either real or simulated), using an ODBC/JDBC driver. The database schema is published and documented.
Logs can be accessed via an API (pull).
API exists that allows a plugin to be developed to manipulate log entries as they arrive (push).
If the tool supports Syslog then the following requirements need to be evaluated too:
Supports UDP-based Syslog logging.
Supports connection-oriented Syslog logging.
Connection-oriented Syslog logging can be wrapped in SSL to protect the logs in transport.
When SSL is used, client and server certificates can be specified for additional security.
Log retention refers to the ability of the device to preserve the important logs in accordance with the organisational policies, and to prevent the system from overflowing.
Maximum age of the logs can be specified. Logs older than the specified age are deleted.
Maximum size of the log store can be specified. Once the store reaches the maximum size the oldest logs are deleted.
There is support for multiple log retention policies (e.g. transactions involving policy violations can be kept for longer).
Logs can be automatically backed up before they are deleted.
While it is expected the analysts will review the events frequently, reports are needed to track the overall security level:
Can generate comprehensive event reports.
Reports can be generated on demand.
Reports can be generated on schedule (e.g. daily, weekly)
The following report formats are deemed of relevance:
Word
RTF
HTML
PDF
XML (for further, custom, processing)
The goal of customisation is for reports to appear as most other documents produced in the organisation:
Visual customisation (e.g. colour, logo, etc).
Content customisation (target groups, e.g. Executive, Developers, etc)
Management is a key part of any network device. This is especially true for application security devices, since this layer is constantly changing to address specific needs, which are directly related with business requirements. For this reason, a WAF’s management component should meet a set of requirements that will guarantee adequate policy enforcement, refinement, and verification. Also, management should be designed and implemented carefully, so that it doesn't interfere with service components in terms of throughput or availability. The following are topics that must be covered by WAF’s management components. They are classified into several categories in order to better understand what requisite belongs to which component. Also, this approach make easier to evaluate a WAF device from a high abstraction level: components can be scored individually without delving into specific details related to them.
Most of WAF devices can automatically learn application logic and build a policy that addresses security and service requirements in that application. Nevertheless, if this policy generates too many false positives, there should be an easy procedure to selectively relax its enforcement rules, instead of having to rebuild it from scratch with lower security level.
There must be an easy way to include legitimate requests originally considered as attacks by the current security policy. This task shouldn't be harder than clicking on a log entry and pushing changes to the WAF device.
This is very self-descriptive. Newer applications probably need a learning-mode phase that shouldn't apply to the rest of applications for which a stable policy has been already built.
Positive Security Model should clear the need for signatures. Nevertheless, many commercial products combine both methods of detection for improved accuracy. In this case, management components should include facilities to develop custom signatures that identify specific, unique risks associated with protected applications.
There is no standard definition of what can be considered a denial of service in every environment. For this reason, WAFs must let the administrator define or refine parameters used to identify this type of attacks. Even better, the device may be smart enough to build an statistical model of what is considered legitimate use of an application and identify when a DoS is taking place, in terms of deviation from this model.
Policies must offer a high level of granularity. Switching between detection and policy enforcement (prevention) modes cannot be associated with the whole application, but must be granularly associated with every component of an application or every type of attack. This lets an administrator just monitor some potential illegal activity in the application while blocking true intrusion attempts at the same time.
If the new version of one policy fails to correctly protect the website or affects the way services are provided, there must exist one mechanism to easily roll-back to the previous applied policy. Otherwise, it can be too difficult to restore a stable policy before the new one causes serious impact.
Learning mode is based on watching traffic between clients and applications, considering that no attacks will occur over that period of time. This assumption may not be true in many cases; for this reason, there must be management configuration facilities to specify trusted hosts that will let the WAF device learn only legitimate traffic.
There must exist some built-in mechanism that lets the WAF learn application logic without the need for users to manually browse the website. This can be a spider, scanner or crawler that automatically visits every link on every page and behaves as a human user, making use of all the services in the website. The spider must be accurate enough and fully integrated with the policy editor. Also, it may be useful to schedule these scan tasks in order to quickly identify changes made to web applications and adjust the current policy accordingly.
Sometimes wizards and WAF’s intelligence is not enough to create accurate policies. For this reason, there must exist a way to delve into low-level details of policies and adjust them somehow. This can mean making changes to regular expressions or adding new ones.
There must be at least two profiles: administrator and operator. Administrators can make changes to current configuration and perform management tasks, while the operator can just review logs and generate reports. If more than one application is protected, it must be possible to assign different administrators and operators to every application.
When an attack is detected, the WAF returns a default error web page to the user. There must be possible to easily change this page, instead of using low-level components involving command-line tasks.
It’s recommended that IT departments schedule internal or third-party security audit services that identify vulnerabilities in currently deployed applications. A WAF device must comply with these activities and let the administrator specify a list of hosts that will generate no alerts in the log subsystem. Auditors will use these hosts for authorised pen-testing purposes.
If signatures are included as an additional detection method, they must be automatically downloaded and installed in every sensor. It makes no sense to wait until the administrator can perform the upgrade process, since at that time the vulnerability window will probably be too wide. On the other hand, update operations must include not only adding new stuff, but also getting rid of inadequate signatures included in previous sets.
Software patches must be transparently downloaded and applied to WAF devices. This way, bugs can be quickly removed and no human intervention is required. Nevertheless, this feature must be optional, since some organisations may decide to test updates in a lab environment before they are installed in production systems (that is, in WAF devices protecting production systems).
It’s recommended that WAF devices communicate with management systems over a dedicated management network. This enables administrators to securely perform maintenance and configuration tasks. Also, communications between the management systems and WAF devices must be simple enough to traverse firewalls and secure enough (that is, encrypted with strong algorithms) to avoid eavesdropping or other types of attacks based on protocol analysis.
The management components must be able to monitor system status and alert the administrator when an error condition occurs or when performance is somehow impacted. Typical notification mechanisms are e-mail, SNMP, syslog and pager.
Log suppression is a simple mechanism to aggregate logs in a clever way and simplify WAF’s activity reviews: similar entries are presented as being just one, with an additional column showing how many times this entry was registered. Criteria used for joining entries are user-defined.
Statistics can help an administrator check if the WAF device is performing as expected. Also, it provides hints about the protected servers’ activity and performance. The following is basic statistical data that must be available in the ligament tool:
number of requests/connections/sessions per second
percentage of malicious requests
percentage of blocked requests (when combining detection and prevention)
number of errors over time
types of browsers identified (through User-Agent header)
most accessed URLs
Buggy interfaces are painful when dealing with WAF devices. For this reason, special care should be taken to avoid exceptions, management subsystems faults and error messages while using the UI. There must exist a mechanism to assure that if an error occurs while pushing new policies to the sensor, the WAF device will get back to its original state.
On the other side, the UI must be hardened enough to make sure that it is not vulnerable to the same types of attacks it protects of. This includes common techniques like SQL Injection or Parameter Tampering , but also advanced forms of exploitation.
Checkpoint Web Intelligence (which I don’t consider a WAF product) is having a great success because of its integrated management within other Checkpoint technologies (like FW-1). This argument cannot be used with any true WAF products, but obviously it drives decision-making processes in organisations with a high volume of Checkpoint technology.
Management:
Log access user interface (view access/event/transaction logs, search, view user sessions)
Event management user interface (tagging, notes, categorisation, escalation)
Being able to mark certain transactions as containing sensitive information (e.g. credit card numbers)
Logging:
Log hardening (e.g. using hashes).
Sensitive information sanitation (e.g. remove the credit card numbers from the transaction logs).
Log expiration
Log data mining
Reporting:
Web traffic reports
Some interesting documents:
Payment Card Industry Data Security Standard, http://usa.visa.com/download/business/accepting_visa/ops_risk_management/cisp_PCI_Data_Security_Standard.pdf
Threat Classification, http://www.webappsec.org/projects/threat/
A. Licence
This work is licensed under the Creative Commons Attribution License. To view a copy of this license, visit http://creativecommons.org/licenses/by/2.5/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.
