Best Practice for PacketShapers in IntelligenceCenter who desire to use the URL Detection and Categorization Feature

Solutions ID:    KB4145
Version:    1.0
Status:    Published
Published date:    10/27/2010

Problem Description


This best practice guide applies to IntelligenceCenter installations running IC 2.1.x or 3.x.x versions which provide visibility reporting services to Packet shapers running PacketWise 8.6.0  or later. Prior to this point in time, PacketShapers did not have the feature known as URL Auto Discovery.

Description of Effects of Feature

This feature will automatically provide a URL breakdown of website traffic, similar to the way that auto discovery of classes works for applications. Furthermore it applies categorization of the URLs discovered using the Blue Coat WebFilter service.  This provides increased visibility into network HTTP traffic usage. The URLs discovered count as a class like any other from the Packet Shaper’s perspective and the PacketShaper will continue to collect URLs until such time as its class count maximum is reached. When URL categorization is turned on and the WebPulse Query option is enabled, PacketShaper will send URL queries to the WebPulse cloud service; the service will look up the URL in its extensive database to find the category (or categories) associated with the URL, and send the response back to the PacketShaper for classification.

Web URLs can be grouped into various categories, such as social networking, gambling, pornography, news media, and shopping. PacketShaper is able to analyze the URLs that users are requesting, determine what category the website belongs to, and classify the traffic into the appropriate category class. This gives you granular visibility into the type of web traffic on your network. You can also use Packet Shaper’s control capabilities to assign policies based on the category. For example, you can assign a neverā€admit policy to all the category classes with adult content. This feature was introduced in Packet Wise version 8.6.0. 

Description of Side Effects of Feature and Problem

This feature will work as advertised but just like with Automatic Class Discovery, sometimes the automatic process is prone to inclusion of items that are not of interest to the end user or just simply, in a cluttered manner. This is even true when you apply it to URLs. There can be a performance affect on the PacketShaper by adding these extra classes and even more so, that performance hit can be passed along to IC if one is not careful. Most users will collect a bunch of URLs and then delete the ones they do not wish to retain for permanent classification. The difficulty becomes that with IC in the picture, those temporary URLs will get passed up to the IC Database, taking up valuable table space and ultimately impact performance. Once classes are included in IC, they cannot be removed. If placed under enough stress form “Garbage” auto-discovered URLs, the IC Server will show the typical failure cascade of missing rollup inclusion of data: missing ME-based report data, then missing FDR, increased report generation times, and eventually crashing the DataCollector service as it runs out of memory. The only remedy to get rid of the URLs once present is to roll the database back to a prior state. A prior state BEFORE they existed.

This will not occur every time, not even in most cases, but there’s enough risk that Blue Coat recommends the following best practice if using this new feature on your PacketShapers that may be reported upon in IC.


Best Practices to Avoid Problem
In order to avoid the problem completely, the following methods are specified: 1) Prior to enabling URL Auto-discovery on the PacketShaper, Remove the device form data collection in IntelligenceCenter.

1. Traffic class tree in the monitor page before URL categories feature is enabled.

2. Url categories setup page.  By default, discovery is enabled for individual categories but the global URL categories setting is disabled.

3. Disable category discovery for all categories with "Turn discovery on/off for all categories."  Selectively enable discovery per category by highlighting them then using "Turn discovery on/off for selected categories."  Enable "URL Categories."  Click "apply changes" to commit all changes.

4. HTTP class manage page while enabling discovery within class.  Check the box then click “apply changes”.  For URL categories to be discovered, discovery within class must be enabled for the classes where the traffic is currently being classified (e.g., HTTP and SSL).

5. Enable global discovery on the "Setup" page.

6. Monitor page after url categories classes have been auto discovered.

At this point you edit the discovered URLs and then put  IntelligenceCenter’s data sources back into active status.

Oops, its too late when I found this article?
If you have already made the mistake of not following the above referenced best practices but are not suffering any harmful effects other than the items appearing in your reports when you do not wish them too. Temporarily, you can set the URLs as Inactive but you would have to repeat that each time your login in the IC. Sooner or later you will need to rest the database to an earlier point in time. Yes this can mean a gap in the IC Data in long term storage.


Rate this Page

Please take a moment to complete this form to help us better serve you.

Did this document help answer your question?
If you are finished providing feedback, please click the RATE CONTENT button. Otherwise, please add more detail in the following text box and then click RATE CONTENT.

Your response will be used to improve our document content.

Ask a Question