Solutions

Access Log disk usage explained

Solutions ID:    KB3820
Version:    1.0
Status:    Published
Published date:    06/08/2010
 

Problem Description

How does access logging manage disk space?

Resolution

Access Log Disk Usage:

Each log facility has a set of disk objects (or files, if you will) to which it writes log entries.  When an upload is initiated (either as scheduled, "early" due to reaching the configured threshold, or manually started from the console), the facility closes the current file, flags it as READY and starts writing to another file (which is flagged as WRITING).  The upload process sends the READY files to the target server (oldest-first), flagging each as DONE in the process.  Once the upload succeeds, these DONE FILES are deleted from disk.  If uploads fail enough that the maximum allowed disk space is exhausted, logging will be shut off unless configured to "delete oldest log entries" as what to do if max size is reached, in which case the oldest READY file is deleted from disk.

Activities such as caching ignore the configured maximum disk size for access logs - that is, the maximum size isn't "reserved" for use by access log.  So when a local access log file is deleted as described above, that space is available for other SGOS processes to use.

Disk usage can be viewed via the management console under:  Statistics > Resources > Disk Usage. This graph will tell you how much space is being used by system processes, access logging, and cache as well as how much space is available for cache.

Sizing:

The maximum access log file size is restricted to 25% of the smallest disk size or is capped at 20GB for ProxySG appliances with large storage capacity, such as the SG9000. The default early upload threshold is set to 80% of the maximum access log file size.

Early Upload:

Access logs upload according to timed intervals. If access logs grow sufficiently to reach the early global upload threshold an immediate upload of all log files is initiated. In this way early upload acts as a failsafe to prevent the logs from reaching the total maximum size, thereby avoiding potential data loss. If the logs fail to upload for some reason then the SG will continue to store access logs until the global maximum size is reached at which point the setting for "If maximum log size is reached" under global settings will determine the logging behavior (stopped or wrapped). In addition to the global setting early upload can also be configured for each log individually.

Continuous Upload

In the continuous scenario those local files are bypassed and the access log entries are appended to a fixed-size outbound queue in memory.  Access log's streaming process reads blocks of entries from that queue and transfers them directly on the wire to the target server. 

If the outbound queue fills up (due to connection failure or its inability to keep up with log entry generation), the log entries are written to disk as previously described until the queued entries and filed entries are all transferred.

 


Rate this Page

Please take a moment to complete this form to help us better serve you.

Did this document help answer your question?
 
 
If you are finished providing feedback, please click the RATE CONTENT button. Otherwise, please add more detail in the following text box and then click RATE CONTENT.
 
 

Your response will be used to improve our document content.

Ask a Question