Solutions

When I pull a large report from my Reporter server, it runs out of memory.

Solutions ID:    KB3686
Version:    5.0
Status:    Published
Published date:    02/02/2010
Updated:    06/09/2010
 

Problem Description

A  30 day report  works fine, but pulling a report of a longer timespan triggers an out-of-memory alert in the journals, and shuts down all  log sources.

In the journal  files I see this message:

""Memory needed is unavailable for malloc (memory needed: 3224309760, physical memory available: 10734972928, virtual memory available: 1057057587, status: 33""

NOTE:  For a detailed analysis on how Reporter monitors memory for the database ,  see KB3655

Resolution

The above journal entry is not worded well.  Here is what has happened: The user asked for a report to run, which triggered a request for memory to the Operating system.  This request has to be for free memory, which is obviously going to be on top of what is already being consumed by the Reporter service ( the database plus it's own program) , and other applications running on this server's operating system.  For the report to be allowed to run, Reporter needs to see enough memory on both the Phyiscal and the virtual memory.  In this case, physical  memory was insufficient, whereas virtual memory was more than enough.  However, because one of values tested did not have the right memory capacity, Reporter did not allow the report to run.

In above journal message we see a request for 3224309760  bytes of memory to run the report.  The request is less than the physcial memory available, but, because it is larger than remaining virtual memory, the request is denied.  The alerts show that the virtual memory capacity is way beyond physical memory.  In fact, looking at the full journal, located at the end of this article,  the alerts initially show 19.53 GB virtual memory capacity and the final alert shows 22.58 GB – We believe that the operating system, extended the page file during that time.

Summary: This above described memory algorithm was done to prevent  false positives when it asking for Memory.  Reporter makes sure it has enough memory to run the operation if it had to use either  Physical or Virtual memory on it's own, before it proceeeds.  If either one of them reports back a value that is insufficient, we will not run that operation, and in some cases shutdown the database to protect ourselves.   This is because, on some Some Server Operating Systems ( 2008 and LINUX) , physical memory looks full when it is merely using the memory to cach files, and can easliy be relased for applications.

The above analysis was conducted on this excerpt of the a customer's bcreporter.out file : The key message of which  was placed in the Problem Description to make it a searchable document.

Blue Coat Reporter (64-bit), Release Version 9.1.3.7, Build 43962, Windows Server 2003
Copyright (C) 2009 Blue Coat Systems, Inc.  All rights reserved.
2010-01-26 09:23:27 (4b5efadf) ALW.INFO.START
   src/sg_main.cpp,1285,MainThread::Run
      main_00001268(4712),,
   The time zone is 'Eastern Standard Time' (+05:00:00)
2010-01-26 09:23:27 (4b5efadf) ALW.INFO.START
   src/sg_task.cpp,1820,MasterThread::Run_Init
      master_thread_00000f88(3976),,
   Web server initialized CLR port 0.0.0.0:8081
2010-01-26 09:23:27 (4b5efadf) ALW.INFO.START
   src/sg_task.cpp,1901,MasterThread::Run_Init
      master_thread_00000f88(3976),,
   Web server initialized SSL port 0.0.0.0:8082
2010-01-26 09:23:27 (4b5efadf) ALW.INFO.START
   src/sg_task.cpp,2040,MasterThread::OpenSGPPort
      master_thread_00000f88(3976),,
   SGP server initialized port 0.0.0.0:9081
2010-01-26 09:48:28 (4b5f00bc) ALW.WARN.ALERT
   src/sg_alert_event_processor.cpp,3908,HandleAlert_GenericMemory
      worker_thread_000012b0(4784),,
   ALERT raised: Reporter process memory usage has climbed above the 75% (14.64 GB) warning threshold -- currently 80% (15.61 of 19.53 GB)
2010-01-26 09:50:25 (4b5f0131) ALW.ERRO.REPOR
   src/sg_reports.cpp,2679,QueueSummaryMergeTask
      worker_thread_00001264(4708),,
   Memory needed is unavailable for malloc (memory needed: 5329981440, physical memory available: 10734972928, virtual memory available: 1057057587, status: 33)
2010-01-26 09:50:25 (4b5f0131) ALW.ERRO.REPOR
   src/sg_reports.cpp,3534,InitSummaryReportGeneration
      worker_thread_00001264(4708),,
   InitSummaryReportGeneration: Failed to QueueSummaryMergeTask
2010-01-26 09:51:19 (4b5f0167) ALW.ERRO.REPOR
   src/sg_reports.cpp,2679,QueueSummaryMergeTask
      worker_thread_000012e4(4836),,
   Memory needed is unavailable for malloc (memory needed: 3224309760, physical memory available: 10734972928, virtual memory available: 1057057587, status: 33)
2010-01-26 09:51:19 (4b5f0167) ALW.ERRO.REPOR
   src/sg_reports.cpp,3534,InitSummaryReportGeneration
      worker_thread_000012e4(4836),,
   InitSummaryReportGeneration: Failed to QueueSummaryMergeTask
2010-01-26 09:53:28 (4b5f01e8) ALW.INFO.ALERT
   src/sg_alert_event_processor.cpp,3908,HandleAlert_GenericMemory
      worker_thread_000012b0(4784),,
   ALERT cleared: Reporter process memory usage has fallen below the 75% (14.64 GB) warning threshold -- currently 58% (11.25 of 19.53 GB)
2010-01-26 09:58:28 (4b5f0314) ALW.ERRO.ALERT
   src/sg_alert_event_processor.cpp,3908,HandleAlert_GenericMemory
      worker_thread_00001264(4708),,
   ALERT raised: Reporter process memory usage has climbed above the 85% (19.19 GB) critical limit -- currently 86% (19.31 of 22.58 GB)


Rate this Page

Please take a moment to complete this form to help us better serve you.

Did this document help answer your question?
 
 
If you are finished providing feedback, please click the RATE CONTENT button. Otherwise, please add more detail in the following text box and then click RATE CONTENT.
 
 

Your response will be used to improve our document content.

Ask a Question