Category: Configuring Oracle Weblogic Logging

The Oracle Weblogic Watch and Notification system can be used to create automated monitors that observe specific diagnostic state and send notifications based on configured rules. More in detail:

• Watch: this component is used to monitor the MBean attributes.
• Notification: this component includes the events which are triggered based on the values set in the watch. There can be different types of alert like SNMP alerts, JMS Messages, JMX notifications etc.

Create a new Notification

In order to create a Notification, you need at first a Diagnostic Unit, such as the Heap Diagnostic that we set up in the earlier section.

Select your Diagnostic Unit and choose, from the Configuration Menu, the Watches and Notifications submenu. From there, select the “Notification” tab and click on the “New” button as depicted by the following picture:

In the next screen, we will be choosing the type of notification we will use among Mail (SMTP), JMS Messages, JMX Notification, SNMP Trap and Diagnostic Image. In our example, we will pick up JMS Messages from the list, which means that notification messages will be delivered using the JMS system:

Next, specify the notification name and if your notification will be enabled by default:

The following screen will be tailored for your notification type: if you selected to receive a JMS notification, then you have to enter the Queue JNDI name and the Connection Factory JNDI name which will be used when Watch events are triggered.

Click Finish to create your Notification. In the next part, we will show how to associate the Notification with a Watch component.

Create a new Watch

In this section we will show how to create a watch that will monitor the attributes of a specific MBean. Select your diagnostic unit and choose, from the Configuration menu, the Watches and Notifications submenu. From there, select the “Watches” tab and click on the “New” button:

The Watch will be based on a set of “Collected Metrics”, therefore, check that this option is selected and that it is “enabled” as shown by the following picture:

Click Next to continue. The Watch will need a rule expression to be checked, so in the following screen choose to add a new rule expression:

Creating a new Rule Expression is quite easy as it can be done by means of an intuitive wizard and some basic operators (“=”,”>”,”<” etc.). Before doing that, in next step we will need to select the MBean we are going to monitor: in our example, weblogic.management.runtime.JVMRuntimeMBean. Done with the MBean, in the next screen we will select the attribute you want to monitor and specify a value for that rule. As shown by the following picture, we are going to watch the HeapSizeMax attribute, specifying as rule “> 200”:

The last screen will recap your current rule, letting you to add some more expressions, combine with existing rule expressions or simply finish the rule creation.

As last effort, select the JMS Notifier from the list on the left side and click on “>” or “>>” so that it is associated with the watch created. Once done, click Finish.

Target the WLDF to any of the servers where MBeans will be monitored. Now check server logs, where you can verify that your alarms are being triggered.

WLDF (WebLogic Diagnostics Framework) is a group of components that work together to collect, archive, and access diagnostic information in real-time.

The following components are part of the WLDF:

• Diagnostic Image Capture: creates a diagnostic snapshot from the server that can be used for post-failure analysis.
• Archive: captures and persists data events, log records and metrics from server instances and applications.
• Instrumentation: adds diagnostic code to WebLogic Server instances and the applications running on them to execute diagnostic actions at specified locations in the code. The Instrumentation component provides the means for associating a diagnostic context with requests so they can be tracked as they flow through the system.
• Harvester: captures metrics from run-time MBeans, including WebLogic Server MBeans and custom MBeans, which can be archived and later accessed for viewing historical data.
• Watches and Notifications: provides the means for monitoring server and application states and sending notifications based on criteria set in the watches.
• Logging services: manage logs for monitoring server, subsystem, and application events.

 In this section we will introduce you how to use the Administration console to achieve diagnostic information about your servers; if you need further information about WLDF we suggest you having a look through the Oracle documentation: (http://docs.oracle.com/cd/E24329_01/web.1211/e24426/intro.htm)

Creating a new Oracle Weblogic Diagnostic Module

In order to create a new Diagnostic module, perform the following steps:

1.       Expand the “Diagnostics” option from the Domain Structure. From there, select the “Diagnostic modules” and click on “New”.

2.       In the next screen, choose a name and a description for your module: in this example we will show how to create a diagnostic module for collecting information about JVM Heap:

Now click on “OK”. You will return to the Summary page.

Our Diagnostic module has been created; however, it still has no metric associated. Click on the Heap Diagnostic module and select the Configuration tab and Collected Metrics sub tab:

Click on “New” in order to create a new metric. In the next screen, you need to choose between “Server runtime” and “Domain runtime” metrics. (The former means that metrics will be collected at server level; the other option, Domain Runtime, provides access to the Domain Wide metrics and is available only for the Admin Server):

Next, select the MBean you want to collect metrics for. For example, if you are behind JVM metrics, select ‘weblogic.management.runtime.JVMRuntimeMBean’ as shown:

In the next screen, specify which attribute of the MBean you want to monitor. In our case, we will select ‘HeapSizeMax’ and click “Finish”.

Finally, from the summary view of WLDF Modules, select the target instance which you want to monitor (for example WLSNode1) and click Finish.

This quick recipe shows how you can view Oracle Weblogic logs from the Administration Console.

Usually, system or resource loggings are monitored with an operating system tool/shell (like the Unix’s tail -f command). However, the Oracle WLS Administration Console provides a Log Viewer for all log files traced by a domain. To reach the Log Viewer, perform the following steps:

1.       Expand from the left panel the option Diagnostic and, from there, click on the Log Files option.

2.       Select the Service you are interested to view as shown by the following picture:

The Log Viewer can find and display the messages based on a large set of attributes (including date, subsystem, severity, machine, server, thread, user ID, transaction ID etc ). It can also display messages as they are logged or search for past log messages. Here’s a snapshot of the Admin Server’s logs:

Besides the core server logs, there are specialized logs for some built-in Services running in Oracle WLS Server such as:

• HTTP Logs
• Transaction Manager Logs
• Weblogic Auditing provider
• Data source Logging
• JMS Logging

Configuring Oracle Weblogic HTTP Logs

The HTTP subsystem keeps a log of all HTTP transactions in a text file. The default location and rotation policy for HTTP access logs is the same as the Server log. You can set the attributes that define the behavior of HTTP access logs for each server you define.

1.       Navigate to the server you want to configure and pick up the Logging upper tab.

2.       From there, select the lower “HTTP” tab as showed:

HTTP access logs are stored by default on the logs/access.log file and are rotated by size every 500KB. You can alternatively choose to rotate them by time.

Configuring Oracle Weblogic Transaction Logs

Each server has a transaction log which stores information about committed transactions, (coordinated by the server), that may not have been completed. WebLogic Server uses the transaction log when recovering from system crashes or network failures. You cannot directly view the transaction log—the records are in a binary format and are stored in either the default persistent store or a JDBC TLOG store for the server.

You can find the Transaction log configuration by entering into the Server view and, from there, choose the Configuration upper tab and the Services lower tab:

The default transaction storage is on the file system; you can opt for a different storage by selecting the Type combo box which will let you select a data source that can be used as Transaction storage.

If no Directory is specified for the default file store, it defaults to: MW_HOME\user_projects\domains\domain-name\servers\server-name\data\store\default

Configuring Oracle Weblogic Auditing Logs

The Auditing provider records information from a number of security requests, which are determined internally by the WebLogic Security Framework. The WebLogic Auditing provider also records the event data associated with these security requests, and the outcome of the requests. Configuring an Auditing provider is optional. The default security realm (myrealm) does not have an Auditing provider configured.

Configuring Oracle Weblogic Data source Logs

The JDBC subsystem records various events related to JDBC connections, including registering JDBC drivers and SQL exceptions. The events related to JDBC are now written to the server log (e.g. when connections are created or refreshed or when configuration changes are made to JDBC objects).

You can configure the data source logging by navigating to your server and selecting the Logging upper tab and the Data Source lower tab:

Configuring Oracle Weblogic JMS Logs

JMS logging is enabled by default when you create a JMS server; however, you must specifically enable it on message destinations in the JMS modules targeted to the JMS server (or on the JMS template used by destinations).

JMS server log files contain information on basic message life cycle events, such as message production, consumption and removal. When a JMS destination hosting the subject message is configured with message logging enabled, then each of the basic message life cycle events will generate a message log event in the JMS message log file.

The default message log is located in the logs directory, below the server instance root directory: MW_HOME\user_projects\domains\domain-name\servers\servername\logs\jmsServers\SERVER_NAME\JMSServer\jms.messages.log

You can change the default Log file name and Rotation Type by entering into the Services | Messaging left panel; from there, select JMS Server and the server you want to operate with.

From the JMS Server GUI, enter into the “Logging” upper tab menu. The follow picture shows the Logging panel where you can customize your JMS Logging: 

As we said, by default Oracle Weblogic server use the JDK Logging implementation. You can anyway switch to Log4j (http://logging.apache.org/log4j) configuration if you feel more comfortable with it. That will anyway require a few steps to complete it:

1.       At first, from the Advanced Logging screen, select Log4j as the Logging implementation as shown by the following picture:

2.       Now enter your Server’s configuration screen and select Configuration | Server Start. From there, add as properties the log4j configuration file name and set the property weblogic.log.Log4jLoggingEnabled to true as shown by the following snapshot:

Remember that the log4j.properties (or log4j.xml) file needs to be included into Weblogic classpath; thus, supposing that you have placed it into the folder “C:\Weblogic\user_projects\domains\base_domain\config”, then you need adding this directory into the server Classpath text item.

The following log4j.properties defines a RollingFileAppender which will write logs to your domain root directory:(C:\Weblogic\user_projects\domains\base_domain)

log4j.appender.rollingFile=org.apache.log4j.RollingFileAppender  log4j.appender.rollingFile.File=mylog.log  log4j.appender.rollingFile.MaxFileSize=2MB  log4j.appender.rollingFile.MaxBackupIndex=2  log4j.appender.rollingFile.layout = org.apache.log4j.PatternLayout  log4j.appender.rollingFile.layout.ConversionPattern=%p %t %c - %m%n  log4j.rootLogger = INFO, rollingFile

Oracle Weblogic has a feature called Log Filters that allows an administrator to determine what is to be logged, by evaluating a simple expression.

For those in the development trenches, it is also possible to create a custom Java class to do the filtering by code (You can read more about this here: http://docs.oracle.com/cd/E24329_01/web.1211/e24428/config_logs.htm)

Here we will illustrate how to create a Log filter using the Web console:

1.       Inside your “Domain Structure”, click on your domain name. From there select the “Log Filter” Tab.

2.       Next, choose to create a new Log filter. Creating a new Log filter requires defining a Log Filter Expression. In the following example we are filtering out all messages containing the Message Id “320012”:

Once the filter is created, you just need to apply it to the servers from the console. To do this, simply click on each server in the console and go to Logging tab for the server. Open the Advanced section in order to apply the filter to the appropriate loggers.

Learn more about Weblogic Log filter expressions at: http://docs.oracle.com/cd/E24329_01/web.1211/e24426/appendix_query.htm

Oracle WebLogic logging infrastructure supports a logger on each server that collects the log events generated by the application server. By default logs are sent to three destinations:

• Log File: configures messages sent to the Server Log file. 
• Console: configures messages sent to the Console.
• Log Broadcaster: configures messages sent to the Log Domain broadcaster.

The following picture depicts the how logs are sent and collected from the various components of an Oracle WLS Domain:

By default, the Server log file is located in the logs directory below the server instance root; for example, DOMAIN_NAME\servers\SERVER_NAME\logs\SERVER_NAME.log. So, for example, the log file for the Admin Server, part of the base_domain will be:

C:\Weblogic\user_projects\domains\base_domain\servers\AdminServer\logs\AdminServer.log

In addition to writing messages to the server log file, each server instance prints logs messages to its console and broadcasts a subset of its messages to a domain-wide log file. By default, servers forward only messages of severity level NOTICE or higher.

The Domain log file provides a central location from where you can view the overall status of the domain; this file resides in the Administration Server logs directory. The default name and location for the domain log file is DOMAIN_NAME\servers\ADMIN_SERVER_NAME\logs\DOMAIN_NAME.log.

General Server Logging configuration

The server log file records information about events such as the startup and shutdown of servers or the deployment of new applications. The messages include information about the time and date of the event as well as the ID of the user who initiated the event.
You can configure Server Logging for the individual servers as follows:

1.       Navigate into the console and select Environment | Servers option. From the Server view, choose the Logging Tab.

2.       Select the General lower tab to configure the generic logging properties as follows:

At the bottom of the screen, you can further expand Logging configuration by entering in the “Advanced” Tab which is discussed in the following section.

Advanced Weblogic Server Logging configuration

You can configure the Log severity levels in the Advanced tab of the Logging screen. The Combo Box “Minimum severity to log” configures the minimum severity of log messages going to all log destinations, as indicated by the following picture:

By default, all server messages are published. Increase the Minimum severity log in order to reduce the verbosity of your server logs. In addition, in the Advanced Logging screen you can fine tune the Logging configuration for some specific destinations (Log File, Log Console, Log Broadcaster) as shown by the following picture:

How to redirect JVM logs to the Weblogic logging system

The JVM used to start WLS produces standard output logs (think for example of System.out contained in the code) or even standard errors. This can lead to a plethora of information spread in many different points. You can, however, mitigate this issue by directing all these standard out / error logs to the Weblogic logging system; you can achieve this by simply flagging the “Redirect stdout logging enabled” and “Redirect stderr logging enabled” from the Advanced Logging configuration.