STAF V3.0.0 Overview


April 22, 2005

Introduction

There are some significant differences between STAF V2 and STAF V3. If you are an existing STAF V2 user, please be sure to read the STAF V3 Migration Guide before upgrading to STAF V3.

New STAF clients are encouraged to start with STAF V3 as it contains many enhancements. Existing STAF V2 clients should wait for a break in their test cycles to migrate to STAF V3, but we encourage our clients to migrate as soon as realistically possible to take advantage of the many new features and capabilities in STAF V3.

Upgrading from STAF V2 to STAF V3 is supported (instead of having to uninstall STAF V2 before installing STAF V3). It is recommended, if possible, that when upgrading to STAF V3, you upgrade all of your STAF machines to V3 at the same time. You can perform basic communication between STAF V2 and STAF V3 machines, but there are many restrictions, so it is best to have an environment that consists of only STAF V3 machines.

If you register any external STAF services with STAF V3, you will have to install a version of the external STAF service that supports STAF V3. Note that STAF services that support V2 don't work with STAF V3 and vice versa because of different service interface levels. For example, to run the STAX service on a STAF V3 machine, you must have STAX V3 installed. An earlier version of STAX such as V1.5.6 will not work with STAF V3.

Installing STAF V3

The installation has been enhanced in STAF V3. We now use a new version of InstallShieldX (10.5) which includes support for more operating systems (including newer versions of Linux). You can upgrade from a previous version of STAF (e.g. another STAF V3.0.0 Beta or STAF V2.x) to STAF V3 without first uninstalling STAF, or you can install STAF V3 to a different directory. Note that for STAF V3.0.0 Beta 5 and earlier, including STAF V2.x, we will actually be doing an uninstall and an install, instead of an actual upgrade due to an InstallShieldX bug.

In addition, you may now install multiple versions of STAF on the same machine. For example, if you already have STAF V2 installed, and you want to try out STAF V3, the installation allows you to install STAF V3 in a different directory without overriding the STAF V2 installation. To run multiple versions of STAF at the same time, you may need to set environment variable STAF_INSTANCE_NAME to differentiate between multiple instances of STAF and you must specify different ports for the tcp interface in the STAF configuration file.

When you install STAF V3 on a Windows system that already has STAF V2 installed (at a different location), only the STAF V3 uninstall is available in Add/Remove Programs. To uninstall STAF V2, you will need to open a command prompt, go to the _uninst directory in the root STAF V2 installation directory, and run "uninstaller.exe".

STAF V2 Migration Utility

To help current STAF V2 customers prepare for migrating from STAF V2 to V3, a new DIAG (Diagnostics) internal service was provided in STAF V2.6.x which allows for the recording of diagnostics data. The DIAG service is being used to track any invocation of a command that will be changing in STAF V3. The DIAG service will record the diagnostic trigger (in this case, the service/request whose syntax and/or results will be changing in STAF V3) along with the source of the trigger (the machine/handle/handle name that originated the request). By running with diagnostics enabled, users are able to identify which test cases, applications, and services they have written will need to be updated to work with STAF V3. See the "STAF V3 Migration Preparation" section in the STAF V2.6.x User's Guide for more information.

New Functions in STAF V3

New functions in STAF V3.0.0 have been rolled out in seven Betas and in the final GA version. Here's a description of some of the major new features in STAF V3.

New Features in STAF V3.0.0 Beta 7

New Features in STAF V3.0.0 Beta 6

New Features in STAF V3.0.0 Beta 5

New Features in STAF V3.0.0 Beta 4

New Features in STAF V3.0.0 Beta 3

New Features in STAF V3.0.0 Beta 2

New Features in STAF V3.0.0 Beta 1

New Format for "Multi-Valued" Results

Changed the format for "multi-valued" results from QUERY/LIST, etc. requests for all the internal services and for the LOG, MONITOR, RESPOOL, ZIP, STAX, EVENT, EVENTMANAGER, CRON, and HTTP external services.

For more information on the external changes made in STAF V3 for these enhancements, please see the following sections in the STAF V3 User's Guide:

Garbage Collected Handle Support

Implemented garbage collection for the Sem and ResPool services so that when handles terminate without freeing semaphores or resource entries, these services will automatically free them.

Note that when you submit a SEM MUTEX REQUEST, etc. request using the STAF command executable from the command line, unless you've set STAF_STATIC_HANDLE to a static handle, when you do a LIST/QUERY of the semaphore, you'll see that the semaphore is not owned by you because garbage collection will have already taken place for the handle that is automatically created and deleted for you by the STAFF command executable.

Similarly, when you submit a RESPOOL REQUEST request using the STAF command executable from the command line, unless you've set STAF_STATIC_HANDLE to a static handle, when you do a LIST/QUERY of the pool, you'll see that the resource entry is no longer owned by you because garbage collection will have already taken place for the handle that is automatically created and deleted for you by the STAF command executable.

Standardized Request Strings for Services

Changed the syntax of following service requests to follow standard guidelines for STAF services:

  1. Updated the HANDLE service's syntax by changing its QUERY request syntax and adding a LIST request as follows: 
    LIST HANDLES [NAME <Handle Name>] [PENDING] [REGISTERED] [INPROCESS] [STATIC]
QUERY HANDLE <Handle>
  2. Updated the PROCESS service's syntax by changing its QUERY request syntax and adding a LIST request as follows: 
    LIST  HANDLES [WORKLOAD <name>] [RUNNING] [COMPLETED]
QUERY HANDLE <Handle>
  3. Updated the SEM service's syntax by changing most its request syntax to use the standard <verb> <noun> syntax instead of a <noun> <verb> syntax. 
    REQUEST MUTEX <Name> [TIMEOUT <Timeout>]
RELEASE MUTEX <Name> [FORCE]

POST    EVENT <Name>
PULSE   EVENT <Name>
RESET   EVENT <Name>
WAIT    EVENT <Name>

DELETE  MUTEX <Name> | EVENT <Name>
QUERY   MUTEX <Name> | EVENT <Name>

For more information on the external changes made in STAF V3 for these enhancements, please see the following sections in the STAF V3 User's Guide:

Added Support for Windows IA64

Added STAF install files for installing a 64-bit version of STAF on Windows IA64.

Providing/Using Data Directory for STAF and its Services

Currently, STAF and its services write data in several different locations (the primary location being {STAF/Config/STAFRoot/data}, and in most cases, this writeable location is not configurable. Some customers want to install STAF in a location that is read-only when STAFProc is running (e.g. in a location that is accessible via a mounted drive or in a directory that STAFProc will not have write permissions), but in STAF 2.x, this is not possible.

Now you can specify the location that STAF and its services can write data by using the new DATADIR operational parameter in the STAF configuration file. STAF V3 and the services that we provide have been updated to use this location to write data. There is a system variable called STAF/DataDir which is assigned the value of the DATADIR operational parameter.

In addition, in the next STAF 3.0.0 Beta, we will allow multiple instances of STAFProc to run on the same machine if a different data directory is specified for each instance of STAFProc via its STAF configuration file.

We have standardized where data within the directory specified by DATADIR should be stored. For example:

If you have any custom services that write data, you should change them to use the writeLocation field made available to services in Java class STAFServiceInterfaceLevel5.InitInfo for Java services and in the STAFServiceInitLevel2 or STAFServiceInfoLevel3 structures defined in STAFServiceInfoLevel3 for C++ services.

Here's a table showing the location that STAF V3 services write data compared to the location that STAF V2 services write data. If you want to use existing data for a STAF V2 service, then you will have to copy the directory or files from the old STAF V2 location to the new STAF V3 location.

Service Directory or Files to Copy STAF 3.0 Location STAF 2.x Location
Log Service Directory {STAF/DataDir}/service/<Service name (lower-case)> {STAF/Config/STAFRoot}/data/log
ResPool Service Directory {STAF/DataDir}/service/<Service name (lower-case)> {STAF/Config/STAFRoot}/data/<Service name (lower-case)>
EventManager Service Directory {STAF/DataDir}/service/<Service name (lower-case)> {STAF/Config/STAFRoot}/data/eventmanagerdata
Cron Service Directory {STAF/DataDir}/service/<Service name (lower-case)> <System.getProperty(user.home)>/crondata
Event Service Files {STAF/DataDir}/service/<Service name (lower-case)>/GenManager.out and RegManager.out <Directory where STAFProc was started>/GenManager.out and RegManager.out
NamedCounter Service File {STAF/DataDir}/service/<Service name (lower-case)>/namedCounter.ser <Directory where STAFProc was started>/.ser
Timer Service Files {STAF/DataDir}/service/<Service name (lower-case)>/tlist.ser and wlist.ser {STAF/Config/STAFRoot}/bin/tlist.ser and wlist.ser

For more information on the external changes made in STAF V3 for these enhancements, please see the following sections in the STAF V3 User's Guide:

VAR Service Syntax Changes

Updated the VAR service syntax to follow standard guidelines for STAF services (as part of a larger initiative to update all of STAF's services).

Here is the new syntax for the VAR service: 

VAR Service Help

SET [SYSTEM | SHARED | HANDLE <Handle>] VAR <Name=Value> [VAR <Name=Value>]...

GET [SYSTEM | SHARED | HANDLE <Handle>] VAR <Name>

DELETE [SYSTEM | SHARED | HANDLE <Handle>] VAR <Name> [VAR <Name>]...

LIST [SYSTEM | SHARED | HANDLE <Handle> | ASHANDLE <Handle> |
      REQUEST [<Request Number>] ]

RESOLVE [SYSTEM | SHARED | HANDLE <Handle> | ASHANDLE <Handle> |
         REQUEST [<Request Number>] ]
        STRING <String> [STRING <String>]...
HELP

Also, the external services we provide have been updated to use the new variable syntax. Note that any custom services that you write will need to be updated to use the new VAR service syntax if the services uses any STAF variables.

For more information on the external changes made in STAF V3 for these enhancements, please see the following sections in the STAF V3 User's Guide:

Added a TRACE Service (and Removed from MISC Service)

In STAF 2.x, the tracing functions were part of the MISC service. In STAF 3.0.0, a new TRACE service was added and the tracing functions were removed from the MISC service.

The syntax also changed for turning on tracing via the STAF configuration file: 

TRACE ALL  [ TRACEPOINTS | SERVICES ]
TRACE TRACEPOINTS <Trace point list> | SERVICES <Service list>
TRACE SET DESTINATION TO < STDOUT | STDERR | FILE <Filename> >

Here is the syntax for the new TRACE service: 

Trace service help

ENABLE ALL  [ TRACEPOINTS | SERVICES ]
ENABLE TRACEPOINTS <Trace point list> | SERVICES <Service list>
ENABLE TRACEPOINT <Trace point> [TRACEPOINT <Trace point>]...
ENABLE SERVICE <Service> [SERVICE <Service>]...

DISABLE ALL  [ TRACEPOINTS | SERVICES ]
DISABLE TRACEPOINTS <Trace point list> | SERVICES <Service list>
DISABLE TRACEPOINT <Trace point> [TRACEPOINT <Trace point>]...
DISABLE SERVICE <Service> [SERVICE <Service>]...

SET DESTINATION TO < STDOUT | STDERR | FILE <Filename> >

LIST [SETTINGS]

PURGE

HELP

For more information on the external changes made in STAF V3 for these enhancements, please see the following sections in the STAF V3 User's Guide:

Added Support for HP-UX IA64, both 32-bit and 64-bit

Added STAF install files for installing a 32-bit version of STAF on HP-UX IA64 and for installing a 64-bit version of STAF on HP-UX IA64.

Note that the HP-UX IA64 64-bit version of STAF requires Java 1.4.1 or later and when registering a Java service, you must specify the -d64 option for its JVM.

For more information on the external changes made in STAF V3 for these enhancements, please see the following sections in the STAF V3 User's Guide:

Communication Interface Enhancements

Currently, STAF 2.x provides one and only one network communication mechanism. That mechanism is TCP/IP. There is no way to configure multiple TCP/IP interfaces, or to enable SSL, or to communicate over any other protocol, such as a Serial Line or older protocols like NetBIOS or SNA, or to specify that traffic should be directed to a specific adapter. This feature will open up the network communication mechanism of STAF to enable users to create network communication mechanisms, called Connection Providers, which can then communicate via any mechanism they choose. In STAF V3.0, the STAF team has provided a Connection Provider that is equivalent to our current implementation, i.e. unencrypted TCP/IP v4 socket traffic, and also supports IPv6 (assuming the operating system supports IPv6).

One advantage to making the communication interfaces pluggable (as opposed to implementing the interface within STAF itself) is that users can create their own Connection Providers without having to redistribute all of STAF (and without having to rebuild STAF on their own). Note that since the Connection Provider interfaces are C/C++ based, new Connection Providers will be platform specific. We will be providing more information in the future on how to write a Connection Provider for STAF.

In conjunction with Feature #626917 (Add notify key to Process service) that was implemented in STAF V2.6.0, these enhancements will resolve many of the problems that people see when using services like STAX, where the use of IP addresses (or mismatching long vs. short names) can cause a job to hang. It will also go a long way to solving DHCP problems as well, although you will still need to be able to resolve the hostname and/or know the IP address of the target system at runtime. Other facilities/services can be added on to STAF to enable "on the fly" registration of DHCP addresses from target systems and is not covered by these enhancements.

STAF's current trust mechanism requires that you grant trust on a single system by single system basis. This is tedious when you want to grant access to all the systems from a given DNS domain, and is particularly problematic when the system to which you want to grant access doesn't actually have a DNS record. These enhancements enable wild cards to be used in trust specifications so that you can grant access to groups of systems at once. Additionally, you are now able to grant access to IP addresses.

Added support to allow multiple copies of STAF to run on the same system. This allows you to use STAF to upgrade STAF, itself, on a given system. A new environment variable called STAF_INSTANCE_NAME has been added to differentiate between multiple instances of STAF. The STAF_INSTANCE_NAME environment variable must be set to the same thing for a given STAFProc daemon and any applications/testcases that want to talk to the instance of STAF. In a future Beta, we will also change the STAF install to allow multiple versions of STAF to be installed on a given system.

Added LIST/QUERY requests to MISC service to show information on which interfaces are enabled

Added support for the DEFAULTINTERFACE operational parameter and provided a new system variable, STAF/Config/DefaultInterface

Removed the USELONGNAMES operational settings and the STAF/Config/EffectiveMachine variable as STAF V3 always uses long names.

Added the STAF/Config/MachineNickname system variable which reflects the nickname for the machine. The nickname defaults to the systems logical network identifier (e.g. long host name for TCP), but can be changed using the MACHINE operational setting in the STAF configuration file. Note that the machine nickname is not used for network communication by STAF. It is used by the LOG and MONITOR services to indicate the machine name for machine logs and monitor messages.

For more information on the external changes made in STAF V3 for these enhancements, please see the following sections in the STAF V3 User's Guide:

User Security (Trust)

Currently, STAF only supports machine level trust, but some users have expressed a need for user level trust as well. For example, you might want to specify trust levels for particular users, in addition (or instead of) particular machines.

Different methods to authenticate a user are supported by the use of authenticators. We are providing a sample authenticator to let you experiment with using user level security. It is not part of the STAF install package. The sample authenticator jar file is available for download here or from the STAF V3 Download page.

Note that how user identifiers and credentials are created and authenticated depends on the authenticator(s) that you provide.

If the user can be authenticated, then the user authentication overrides the machine authentication.

For more information on the external changes made in STAF V3 for these enhancements, please see the following sections in the STAF V3 User's Guide:

Variable Enhancements

Currently, a STAF handle's variable pool is only used in variable resolution if the request is processed locally. If a request is made to a remote system, the STAF handle's variable pool is not delivered to the target system and used in variable resolution. On remote systems, only the remote system's global variable pool is used.

This leads to issues where variables such as a handle's log mask are not propagated to remote systems. However, of possibly larger consequence is that this flaw limits the ability of services to provide configurability on a per handle basis. For example, the Log service could potentially allow for a default logging level to be specified in a STAF variable. But, currently, this would only work for systems doing local logging. With the addition of these enhancements, STAF services will be able to start leveraging STAF variables for defining configurable behavior.

A new "SHARED" option has been added to the VAR service SET, GET, DELETE, LIST, and RESOLVE requests. On SET, GET, and DELETE requests, this option will direct the request to the system's shared variable pool. On a RESOLVE request, the SHARED option will resolve variables in the context of the system's shared variable pool. On a LIST request, this option will list the contents of the system's shared variable pool.

For more information on the external changes made in STAF V3 for these enhancements, please see the following sections in the STAF V3 User's Guide:

STAF Configuration File Changes

Here is the new default STAF configuration file provided with STAF V3.0: 

# Turn on tracing of internal errors and deprecated options
trace enable tracepoints "error deprecated"

# Enable TCP/IP connections
interface tcp library STAFTCP

# Set default local trust
trust machine local://local level 5

# Add default service loader
serviceloader library STAFDSLS

Notes:

  1. The syntax for turning on tracing has changed.

  2. The syntax for the interface line has changed. Also, you can now register multiple network interfaces. For example: 
    interface tcp  library STAFTCP
interface tcp2 library STAFTCP option port=6700

    Note that requests from the local system will now appear as though they came from an interface named "local" and a system identifier of "local". This allows you to specify a trust level for local requests (in previous versions of STAF, local requests were automatically granted a trust level of 5.

  3. Machine trust specifications now can contain IP addresses or long hostnames (short host names are no longer supported). 
    trust machine 9.3.224.82 level 5
trust machine server1.austin.ibm.com level 5

    You can now specify user trust specifications. For example: 

    trust user JohnDoe@us.ibm.com level 4
trust user Jane@company.com level 5
    Note that you can specify wild cards ('*' for zero or more of any character, '?' for any single character) in trust specifications for machines and users. 
    trust machine 9.3.194.* level 3
trust user *@us.ibm.com level 4

New Format for Queued Messages Generated by STAF and its Services

Messages queued by STAF V3.x and it's services have a different format than in STAF V2.x. A new TYPE option was added in STAF V3 to identify the type of message being queued. STAF V3.x and it's services specify a type for all the message that they generate because queued messages can contain various marshalled data structures, such as a map, so the type let's you know the structure of the message.

Here's a table showing the new format of the messages queued by STAF V3 and it's services, as well as showing the format of these messages queued by STAF V2.x clients to a STAF V3 machine.

STAF V3 Queued Messages
Queued Messages from STAF V3 Queued Messages from STAF V2
TYPE: STAF/RequestComplete
MESSAGE A map containing the request completion information as follows: 
{
  'requestNumber': <Request #>,
  'rc'           : <Request Return Code>,
  'result'       : <Request Result Buffer>
}
TYPE: <None>
MESSAGE A string containing request completion information as follows:

STAF/RequestComplete <Request #>;<Request return code>;<Request result buffer>

TYPE: STAF/Process/End
MESSAGE: A map containing the process completion information as follows: 
{
  'handle'      : <Handle #>,
  'endTimestamp': <YYYMMDD-HH:MM:SS>,
  'rc'          : <Process Return Code>,
  'key'         : <Key> | <None>,
  'fileList'    : [
    {
      'rc'  : <Returned File #1 RC>,
      'data': <Returned File #1 Data>
    }
  ]
}
TYPE: <None>
MESSAGE: A string containing process completion information as follows:

STAF/PROCESS/END <Handle>;<Timestamp>;<Return Code>[;<Returned file data>]

TYPE: STAF/Start
MESSAGE: Blank 		TYPE: <None>
MESSAGE: A string containing STAF/START.
TYPE: STAF/Shutdown
MESSAGE: Blank 		TYPE: <None>
MESSAGE: A string containing STAF/SHUTDOWN.
TYPE: STAF/Service/Event
MESSAGE: A map containing the event information as follows: 
{
  'eventID'         : <Event ID>,
  'eventServiceName': <Event Service Name>,
  'handle'          : <Handle #>,
  'handleName'      : <Handle Name>,
  'machine'         : <Machine>,
  'propertyMap'     : {
    <Name>: <Value>,
    ...
  },
  'subtype'         : <Subtype>,
  'timestamp'       : <YYYYDDMM-HH:MM:SS>,
  'type'            : <Type>
}
TYPE: <None>
MESSAGE: A string containing the event information as follows:

STAF/SERVICE/<Event Service Name>/;<Event ID>;<Generating Machine>;<Generating Process>;<Generating Handle>;<TimeStamp>;<Type>;<Subtype>; [<Name>=<Value>;]...

TYPE: STAF/Service/EventManager/End
MESSAGE: Blank 		TYPE: <None>
MESSAGE: A string containing EventManager/End.
TYPE: STAF/Service/Cron/End
MESSAGE: Blank 		TYPE: <None>
MESSAGE: A string containing Cron/End.
TYPE: STAF/Service/STAX/End
MESSAGE: Blank 		TYPE: <None>
MESSAGE: A string containing STAX/End.
TYPE: STAF/Service/STAX/JobWaitComplete/
MESSAGE: Blank 		TYPE: <None>
MESSAGE: A string containing STAX/Job/Wait/Complete/.
TYPE: STAF/STAXMonitor/End
MESSAGE: Blank 		TYPE: <None>
MESSAGE: A string containing STAXMonitorJob/End or STAXJobMonitor/End.
TYPE: STAF/STAFDemo/Stop
MESSAGE: Blank 		TYPE: <None>
MESSAGE: A string containing STAFDemo/End.

Future Enhancements

Some enhancements that have not been provided yet in STAF V3.0.0, but are planned for a later STAF V3 release are: