The DLB Plus I/O analysis and reporting system is very easy to use. A DCL command is entered which executes the DLBPLUS.COM file. This command file initializes the DLB Plus I/O analysis and reporting system and presents the user with menus from which processing and report selections can be made.

It is recommended that a separate directory be created into which DLB Plus can store collected data. When you run the DLB Plus analysis and reporting system, the directory you are in --- is the directory that will contain the diagnostic data collected by DLB Plus. The collected data is used for the I/O analysis reports and graphs.

When DLB Plus is run and I/O data is collected, the collected (sampled) data represents a snapshot of the images (programs) currently executing on the node from which you are running DLB Plus.

5.1 Activating DLB Plus Master Menu

To start using the DLB Plus I/O analysis and reporting system, log onto the SYSTEM account:

Username: SYSTEM Password: (enter your system password)

To activate the DLB Plus Master Menu, first create the directory in which you want the collected data to be stored and set your default to that directory. For example:

$ CREATE/DIRECTORY $1$DISK1:[MYNAME.DATA] $ SET DEFAULT $1$DISK1:[MYNAME.DATA] <--- enter if no logical created $ SET DEFAULT TTI_DLBP_DATA_LOC <--- enter if logical created

Then, enter the following command:

$ @TTI_DLB:DLBPLUS

When the DLB Plus I/O analysis and reporting system is activated, the DLB Plus package name, version number and copyright information are displayed. A few seconds later, the screen clears and the "DLB Plus" menu displays a message that DLB Plus is being initialized.

After DLB Plus has been initialized, the screen clears and the "Master Menu" is displayed.

Example 5-1 DLB Plus Master Menu

DLB Plus Dynamic Load Balancer PLUS 01-Sep-1995 +-------------------------------Master Menu-------------------------------+ | | | Dynamic Load Balancing Reports and Graphs | | DISPLAY DLB Display Utility SREPORTS Summary Reports... | | STATUS DLB Status Utility DREPORTS Detail Reports... | | GRAPHS Analysis Graphs... | | I/O Sampling SCAN Batch I/O Scan | | SAMPLE Sample I/Os and Locks | | CONS Consolidate Sample Data Batch Procedures | | SELECT Select a Data Sample BATCH Set Up Batch Stream | | LIVE Live Lock Analysis... SUBMIT Submit Batch Stream | | AUTO Automatic Sampling CANCEL Cancel Batch Set Up | | | | Miscellaneous | | SYSTEM VMS System Commands | | EXIT EXIT DLB Plus | +-------------------------------------------------------------------------+ EXIT = Exit \ = Back HELP = Help

Select the SAMPLE item from the menu. The SAMPLE item collects I/O data by taking a snapshot of the images currently running on the node that you are logged onto (see Chapter 9, Sampling, Consolidating, and Selecting).

For each node to be analyzed, perform the following steps:

• Log onto the node to be sampled by DLB Plus
• Set your default directory to the DATA directory where the collected data is to be stored
• Activate the DLB plus Master Menu using the @TTI_DLB:DLBPLUS command
• Select the SAMPLE menu item to collect data for the node

To obtain a complete picture of all I/O activity on all the nodes within a cluster, it is necessary to log onto each node, run DLB Plus from that node and collect I/O data for that node. DLB Plus must be run on each individual node because DLB Plus only samples data for the node on which it is being run.

After all the samples have been taken and the I/O data has been collected:

• Log onto any node that has access to your DATA directory
• Set your default directory to the DATA directory and activate the DLB Plus Master Menu
• Select a node to be reported on and run the I/O activity reports and graphs for the selected node(s)

After the DLB Plus Master Menu is activated (see Section 5.1), various menu items can be selected.

DLB Plus Dynamic Load Balancer PLUS 01-Sep-1995 +-------------------------------Master Menu-------------------------------+ | | | Dynamic Load Balancing Reports and Graphs | | DISPLAY DLB Display Utility SREPORTS Summary Reports... | | STATUS DLB Status Utility DREPORTS Detail Reports... | | GRAPHS Analysis Graphs... | | I/O Sampling SCAN Batch I/O Scan | | SAMPLE Sample I/Os and Locks | | CONS Consolidate Sample Data Batch Procedures | | SELECT Select a Data Sample BATCH Set Up Batch Stream | | LIVE Live Lock Analysis... SUBMIT Submit Batch Stream | | AUTO Automatic Sampling CANCEL Cancel Batch Set Up | | | | Miscellaneous | | SYSTEM VMS System Commands | | EXIT EXIT DLB Plus | +-------------------------------------------------------------------------+ EXIT = Exit \ = Back HELP = Help

The Master Menu contains the items that:

• display system data
• check SYSGEN parameter values
• sample I/O data separately or automatically
• consolidate sampled data
• select nodes or consolidations to analyze
• perform live analysis of lock holders and waiters
• bring up the reports and graphs menus
• set up and scan batch processes

Menu ITEMS are listed on the menu screen. Each item has a short description of what the item does.

To choose a given item, use the "arrow" keys to locate the item to be executed.

An item can also be selected by entering the item name. You need enter only as many characters of the item name as is necessary to distinguish the item from all other items on the menu.

6.3 On-line Help System

In some cases you will be presented with a list of related help topics. If you are interested in a related topic, just enter the topic name when prompted for "Help topic?".

When you have finished getting the help that you need, press [RETURN] or enter EXIT at the "Help topic?" prompt. The help screen is replaced with the screen that was displayed before HELP was entered.

6.4 Exiting and Going Back to Correct Mistakes

Type the word EXIT or [CTRL/Z] to get out of a menu item, go back to a previous menu, or exit out of DLB Plus altogether (if at the Master Menu).

Press the "\" key to back up to the previous prompt or go back to a previous menu.

You can also use the EXIT DLB Plus menu option to exit out of the DLB Plus Master Menu.

6.5 Aborting a Report or Graph

6.6 OpenVMS System Commands Option

When this menu item is selected, the screen clears and you are placed at the system prompt. The screen appears as follows:

Passed to VMS... Press RETURN when done. TESTDISK:[DLB_TESTER] $$ X

The double dollar sign ($$) reminds you that you got to the system prompt from DLB Plus.

When you want to return to the DLB Plus Master Menu, press the [Return] key at the $$ prompt and the menu will be displayed.

DLB Plus DISPLAY provides a view of the operation of an OpenVMS system through a dynamic display. It shows the system load graphically on any VTxxx-compatible terminal. Use DLB Plus DISPLAY to compare before and after effects of changes made to the system. Use it as a data source for system analysis. The following values are displayed:

• Current calculated response index
• Percentage of free memory and the pages of memory on the free list
• Average response index during the period the utility is operating
• Number of processes in computable state and page fault wait state
• Operating system overhead and idle time
• Direct I/O rate and hard and soft page fault rates during the last sampling interval
• Current values of QUANTUM and IOTA
• DECnet Node name
• OpenVMS version
• DLB Plus version
• DLB Plus operating mode
• DLB Plus activity information
• Images activated during last sampling interval

The following section describes DLB Plus DISPLAY and its operation.

7.1 Running DLB Plus DISPLAY

The screen clears and you see the display graph. The graph remains blank for one sample interval while statistics are gathered. Then, the screen is updated at 10 second intervals.

To write the DLB Plus DISPLAY screen to a file, press [CTRL/F] while DLB Plus DISPLAY is running. This creates a file called SYS$SCRATCH:DLBPLUS_DISPLAY.LIS which is a snapshot of the current DLB Plus DISPLAY screen.

Type [CTRL/Z] to terminate DLB Plus DISPLAY.

Example 7-1 DLB Plus DISPLAY

DLB DISPLAY Utility Vx.x Total Samples : 135 Sample Intvl (s) : 10 (1)Node Name : TTI VMS Version : Vx.x DLB Version : DLB Vx.x (2)Response Index Graphic History DLB Op Mode : Ample | Cur RI (%): 95 AV RI (%) : 81 |100| (3)Total Sys Memory: 65536 | 90| X X X X X X Free List Size : 48902 | 80| X X X X X X X X X X X X X % on Free List : 74 | 70| X X X X X X X X X X X X X X X X X X Modified Lst Size : 3108 | 60| X X X X X X X X X X X X X X X X X X X X X X % of MPW_HILIMIT : 31 | 50| X X X X X X X X X X X X X X X X X X X X X X Memory Recovered : 132 | 40| X X X X X X X X X X X X X X X X X X X X X X X X | 30| X X X X X X X X X X X X X X X X X X X X X X X X X X X (4)Hard Fault / Sec: 0 | 20| X X X X X X X X X X X X X X X X X X X X X X X X X X X Soft Fault / Sec : 11 | 10| X X X X X X X X X X X X X X X X X X X X X X X X X X X HF / TotFlts % : 0 + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (5)QUANTUM : 10 (6)# COM PROCS : 1 (7)DLB Cycles Compl: 817 IOTA : 2 # PFW Procs : 0 Total Proc Adj : 4 Total # Procs : 20 Memory Adj Cutoff : 3599 (8)DLB Bias : NONE Ker+Int+MPSyn%: 15 # Inactive Proc : 0 Only mem : No Idle Time % : 30 Adj Last Cycle : 0 Only CPU : No Dir IO / Sec : 1 Eval Last Cycle : 20 Adjust All : No Images Activd : 1

7.2 The DISPLAY Screen

The response index is an indicator of the overall responsiveness of the system to the CPU and I/O needs of processes. The graph represents the response index over the last 45 sampling intervals. By examining this graph, the user can view system trends. Assuming only moderate new process activity and an active DLB Plus process, the graph shows an increasing response index until a steady state has been achieved.

7.2.1 DLB Plus DISPLAY information

1. DLB Plus status information. Included is information on:
   • Node Name
     DECnet node name of the machine this DLB Plus DISPLAY is operating on.
   • OpenVMS Version
     Version of the OpenVMS operating system currently operating on this machine.
   • DLB Plus Version
     Version of the DLB Plus software currently operating on this machine.
   • DLB Plus Op Mode
     The DLB Plus operation modes are Ample, Low and Insufficient. (See the DLB Plus Modes of Operation table in Table 1-1, DLB Plus Modes of Operation for a description of the operation modes.)
   • Total Samples
     Total number of samples taken since this DLB Plus DISPLAY was started.
   • Sample Intervals
     Interval at which DLB Plus DISPLAY samples are being taken.
2. Response Index Graphic History. This shows response index measured in percentages. DLB Plus uses a vertical bar chart with values of 0 to 100%, each bar representing a sample:
   • Cur RI(%)
     This value gives the calculated response index of the entire system. Factors that affect response index include:
     • Free memory
     • Idle time
     • Operating system overhead
     • Page faults
     • Direct I/Os
     • Number of computable processes---Number of processes waiting for page faults to be resolved.
   • Avg RI(%)
     This value is the average of all the response index values since starting DLB Plus DISPLAY.
3. System memory information:
   • Total Sys Memory
     Number of pages of physical memory on this system.
   • Free List Size
     The amount of memory on the free list, expressed in pages.
   • % on Free List
     The % on Free List is the amount of memory available on the Free List expressed as a percentage of the total memory installed on the system. Since OpenVMS uses memory, this value can never be 100%. Low values are a cause for concern.
   • Modified Lst Size
     This is the current size of the modified page list.
   • % of MPW_HILIMIT
     This is the current size of the modified page list expressed as a percentage of the value of the SYSGEN parameter MPW_HILIMIT. MPW_HILIMIT is the threshold value used by OpenVMS to determine when to flush the modified list.
   • Memory Recovered
     This is the amount of memory that has been recovered from processes since the last display update.
4. Information on page faults:
   • Hard Faults/Sec
     The number of page faults per second from the disk devices.
   • Soft Faults/Sec
     The number of soft faults per second.
   • HF / TotFlts %
     This is the number of hard faults expressed as a percentage of total page faults.
5. Checks critical SYSGEN parameters:
   • QUANTUM
     The current system QUANTUM value. As QUANTUM changes, IOTA may also change.
   • IOTA
     The current system setting for IOTA.
6. OpenVMS statistics:
   • #COM Procs
     The number of processes waiting in the scheduler's COM queues.
   • #PFW Procs
     The number of processes waiting for a page fault to be resolved before being allowed to continue.
   • Total # Procs
     The total number of processes in the system.
   • Ker+Int+MPSyn%
     Kernel mode plus Interrupt stack plus Multi-processor synchronization, shown as a percentage of total CPU time.
   • Idle Time %
     This value represents the time the system has spent with no work to perform. This number comes from the scheduler loop waiting to find a process ready to run.
   • DIR IO/Sec
     The number of direct I/O operations per second performed during the last data collection interval.
   • Images Activd
     The number of images activated during the last data collection interval.
7. DLB Plus work cycle information:
   • DLB Plus Cycles Compl
     The number of adjustment cycles DLB Plus has performed since it was started.
   • Total Proc Adj
     The number of process memory adjustments DLB Plus software has done since it was started.
   • Memory Adj Cutoff
     The minimum amount of memory that DLB Plus software has determined should be left available to each process.
   • #Inactive Proc
     The number of processes that have been marked by DLB Plus as inactive.
   • Adj Last Cycle
     The total number of processes adjusted by DLB Plus during its last working set list adjustment cycle. Inactive processes are not available for adjustment after their working set lists have been reduced to the minimum possible.
   • Eval Last Cycle
     The number of processes evaluated by DLB Plus for adjustment during its last working set list adjustment cycle.
8. DLB Plus performance options:
   • DLB Plus Bias
     Bias option selected.
   • Only MEMORY
     Whether the "Only MEMORY" option has been selected.
   • Only CPU
     Whether the "Only CPU" option has been selected.
   • Adjust All
     Whether the option to adjust processes with a system UIC has been selected.

The response index is an indicator of the overall responsiveness of the system to the CPU and I/O needs of processes.

The response index indicates the following about a system:

• When the response index is at 100%, it indicates that any process that transitions to the COM state is able to be scheduled by the CPU with no wait.
• As the response index decreases from 100%, it indicates that the system is experiencing activity which has the net effect of causing delays in a process being scheduled to the CPU.

Use the response index as a relative indicator of system responsiveness. Pay more attention to the changes that span multiple samplings rather than individual sample values. There are several trends that warrant investigation to determine why the system responsiveness is changing; they are:

• The response index is steadily decreasing during a monitoring period.
• The response index is higher or lower for a monitoring period, compared to a previous monitoring period when the same system load was present.

The response index is calculated using the following performance factors:

• Number of processes in COM (computable) state
• Number of processes in PFW (page fault wait) state
• The sum of: the amount of CPU time spent in Interrupt Stack, Kernel Mode, and MP---Multi-Processor synchronization. This sum can loosely be thought of as system overhead.
• Number of hard page faults per second
• Number of direct I/O operations per second
• Amount of CPU idle time
• Amount of memory pages on the free list

The number of processes in COM state is an element of the CPU load on the system. The more processes in the COM state, the longer a given process has to wait to get to use the CPU. In addition, system overhead is incurred due to scheduling activity, quantum end events, etc.

PFW state is a state that a process is in, due to its having to wait to have a page fault resolved by OpenVMS. The more processes in PFW state, the longer a process has to wait to get a page fault resolved. In addition, system overhead is incurred as a result of servicing the page faults.

CPU activity spent in system overhead keeps user processes from executing. This slows down the system.

Hard page faults cause CPU resources to be consumed. In addition, hard page faults cause I/O operations, further slowing down the system.

Direct I/O operations incur system overhead and can cause delays for other processes accessing disks.

Idle time is the reserve capacity the system has to handle processes needing CPU time. The more idle time, the more reserve capacity and thus the greater probability that processes will run quickly.

Free pages are the reserve capacity the system has to handle memory demands. The more free pages, the more reserve capacity. Having more free pages tends to reduce hard page faults.