Fig. 5.1: Examples of Instances
An "Instance" is a single invocation of the FLEX-ES CPU program flexes(1). An Instance provides a z/Architecture (or ESA/390, or System/370, etc.) computer. The computer provided by an Instance may be uniprocessor or multiprocessor (subject to limitations discussed later). More formally, a FLEX-ES Instance is a mainframe uniprocessor or multiprocessor "CPU Complex."
Multiple Instances can run at the same time on the same server. Each simultaneous Instance is a completely separate mainframe computer which runs its own mainframe operating system.
FLEX-ES does not implement PR/SM or provide LPARs. Neither is FLEX-ES the same as VM (although VM can run on FLEX-ES). FLEX-ES provides the ability to run multiple mainframe operating systems simultaneously on the same server through the use of multiple FLEX-ES Instances, each one of which is an independent computer running its own mainframe operating system.
A CPU Complex is not a Sysplex. FSI provides no support for Basic Sysplex (although Basic Sysplex has been achieved by customers in special situations). Parallel Sysplex is not possible because FLEX-ES does not provide certain components necessary for Parallel Sysplex, including the Coupling Facility and the Sysplex Timer.
A server can run, simultaneously, any number of Instances (processor complexes), from zero to the practical performance limitations of the server. (A server running zero Instances would be a server providing only non-CPU resources such as DASD.)
Fig. 5.1: Examples of Instances
Note that these examples omit the relationship between the emulated mainframe CPUs and the underlying server processors. This relationship is important, and will be discussed later.
The sections below discuss FLEX-ES emulated CPUs and their relationships to server processors in some detail. For those already familiar with this but who require a quick reminder of the principles, here they are in brief. If this subject is new to you, skip this section and read on.
Definition:
Rules:
Implications (not rules, but results of rules):
Strong Suggestion:
(It is also possible to specify specifically which server processor is to be used for a particular CPU. This may be done either with the bound statement or with an optional parameter on the dedicated statement. This capability was of importance on the NUMA-Q platform, but generally is not important on uniprocessor and symmetric multiprocessor servers.)
As has been noted earlier, a FLEX-ES "Instance" is an emulated uniprocessor or multiprocessor mainframe CPU complex. There is a one-to-one relationship between emulated CPUs in an Instance and physical server processors. The emulation of a single CPU in an Instance is never divided between multiple server processors, and a single server processor never provides multiple CPUs within a single Instance. Each CPU in an Instance corresponds to exactly one server processor.
The examples below illustrate several valid relationships between emulated mainframe CPUs and physical server processors.
Fig. 5.2: Valid CPU to Processor Relationships
In the first example, a single-CPU emulated mainframe (an Instance) is running on a single-processor server (or perhaps a laptop computer). As will be seen in later sections, since in such a configuration the single server processor must also handle all other server processes the mainframe CPU cannot have the dedicated use of this server processor. If the mainframe CPU is not dedicated, however, this configuration is perfectly valid. (The concept of a "dedicated" CPU will be discussed later. For now the intuitive idea that such a CPU has the exclusive use of its server processor should suffice.)
In the second example, a single-CPU emulated mainframe (an Instance) is running on a two-processor server. Such a configuration is straightforward and needs little comment.
In the third example, a two-CPU emulated mainframe (an Instance) is running on a two-processor server. This is valid so long as the mainframe CPUs are not dedicated, as other server processes need to run on server processors as well.
The fourth and fifth examples each illustrate a three-CPU mainframe Instance running on a four-processor server. Each CPU is, as shown, associated with exactly one server processor. While it is possible to specify explicitly in an Instance's configuration the specific server processors on which each mainframe CPU is to start, such an explicit specification is needed only in unusual circumstances. The default behavior is to leave any specific association unspecified. Thus, Example 4 might represent a configuration as started at one point in time, with CPUs running on server processors 0, 1, and 3. Example 5 might represent the same configuration started at a different time when its CPUs happened to run instead on server processors 0, 2, and 3.
In all five example, it is assumed that all server processors can be used for mainframe CPUs. See the discussion of the concept of server processor "enablement" later in this chapter for a further discussion of this matter.
The examples below illustrate several invalid relationships of mainframe CPUs and server processors. All violate the one-to-one relationship in one way or another.
Fig. 5.3: Invalid CPU to Processor Relationships
In the first example, a two-CPU Instance is shown as if it could run on a single-processor server or laptop. It cannot, because this would require both emulated CPUs to share the single server processor.
In the second example, a two-CPU Instance is shown running on only one processor of a two-processor server. It does not, and will fail if an attempt is made to force it to do so.
(Note: The possible configuration issues which might suggest such an invalid configuration have not yet been discussed. Looking ahead, though, they would include an attempt to start a two-CPU Instance when only one server processor was enabled, an attempt to start a two-CPU Instance when only one server processor was available (due, perhaps, to dedicated use of other server processors), or the misuse of the bound or dedicated configuration statements.)
In the third example, a a single-CPU Instance is shown as if it were running on two server processors. This is not done.
The fourth example compounds the errors shown in the other three.
The examples below illustrate several valid relationships of mainframe CPUs and server processors in situations involving multiple simultaneous Instances. These examples show the simplest conceptual way in which multiple Instances can be arranged to run simultaneously: in none of the examples is a single server processor used by the CPUs of more than one instance.
Fig. 5.4: Valid Relationships, Multiple Instances
Unless the CPUs of an Instance are declared to be dedicated, then the server processors on which they run may also run other server work. This work may include other FLEX-ES Instances (with, of course, non-dedicated CPUs). Therefore, a single server processor may be used by CPUs of different Instances, but will never be used by multiple CPUs within a single Instance. The examples below illustrate several valid possibilities.
Fig. 5.5: More Valid Relationships, Multiple Instances
In the first example, two one-CPU Instances share the single processor of a one-processor server or laptop. Neither CPU can be dedicated. The single server processor handles both CPUs as well as all other server work.
In the second example, two two-CPU Instances share the same two processors on a larger server. Again, no CPU in this configuration can be dedicated. There are legitimate reasons for considering such a configuration. Perhaps, for example, only two processors on the server have been enabled for FLEX-ES CPU use (see below for a discussion of server processor enablement).
In the third example, three server processors are in use, providing a total of four emulated CPUs (two for each two-CPU Instance).
The fourth example is intended to show a rather crowded scene. On a four-processor server, four Instances are running simultaneously. One is a four-CPU Instance, two others are two-CPU Instances, and the fourth is a one-CPU Instance. (The CPU to processor lines for the two-CPU Instances are shown dotted for clarity.)
Certain capabilities of FLEX-ES, such as the ability to run multiple Instances simultaneously, may be used to address needs which may also be addressed by VM or, in hardware mainframe implementations, by PR/SM and LPARs. However, it is important to realize that FLEX-ES is quite different from either of these.
FLEX-ES and VM are conceptually quite distinct. Two major differences are of interest here: processor architecture and the relationship between emulated and physical processors.
While both VM and FLEX-ES present machines created from software (often with hardware/firmware assistance with VM), they differ in the relationship between the architectures which they present and on which they run. VM takes an underlying z/Architecture (or earlier architecture) CPU or CPUs and virtualizes them into an arbitrary number of CPUs of the same architecture. FLEX-ES implements z/Architecture (or earlier architecture) CPUs on top of underlying processors of a different architecture.
There are circumstances where VM can be used to advantage on top of FLEX-ES.
The examples below illustrate this (without mentioning specific processor counts and affiliations).
Fig. 5.6: FLEX-ES Is Not VM
The first example presents, in general terms, VM.
The second example presents FLEX-ES. In this example, two Instances of FLEX-ES are running, and on each a single mainframe OS is running.
The third example illustrates one possible use of VM together with FLEX-ES. In this example, one FLEX-ES Instance is running VM. VM in turn is providing multiple virtual systems each running a guest mainframe OS.
The relationship between virtual machine CPUs and host CPUs in VM is many-to-one or many-to-many. In FLEX-ES, the relationship between emulated processors in Instances and underlying server processors is always one-to-one. (However, the ability to run multiple FLEX-ES Instances simultaneously allows a FLEX-ES installation to provide more emulated CPUs than it has physical server processors.)
The examples below illustrate various ways in which VM and FLEX-ES can (and cannot) address the situation of providing CPUs intended for a uniprocessor VSE system and a dual processor z/OS system.
Fig. 5.7: CPU Relationships, VM and FLEX-ES
In the first example, VM virtualizes the single processor of a uniprocessor z/Architecture mainframe. It provides a total of three virtual processors, on which the two guest operating systems run.
In the second example, VM virtualizes the two processors of a dual processor z/Architecture mainframe to provide the three virtual processors.
The third example illustrates an invalid attempt to use FLEX-ES to provide three emulated CPUs. This attempt fails because the second Instance (the one for the z/OS system) requires two server processors on which to run; one for each of its emulated CPUs.
The fourth example illustrates a valid solution using FLEX-ES. The server has two processors (both, it is assumed, enabled for use for CPU emulation). This allows both instances to run, as the second instance has two server processors available to it.
FLEX-ES and the Logical Partitions (LPARs) provided by PR/SM are conceptually quite distinct. As is the case with VM, the two major differences of interest here are processor architecture and the relationship between emulated and physical processors.
An LPAR is an allocation of CPU and other resources from an underlying z/Architecture (or previous architecture, as appropriate) machine. FLEX-ES, by way of contrast, implements a z/Architecture machine on top of a fundamentally different architecture (IA-32).
Example 1, below, shows a greatly simplified picture of two mainframe OSs each running in an LPAR.
Example 2 presents the same to mainframe OSs each running on a FLEX-ES Instance.
Fig. 5.8: FLEX-ES is Not PR/SM
The relationship between CPU resources provided in an LPAR and underlying CPUs is, in general, many-to-many. In FLEX-ES, the relationship between emulated processors in Instances and underlying server processors is always one-to-one. (However, the ability to run multiple FLEX-ES Instances simultaneously allows a FLEX-ES installation to provide more emulated CPUs than it has physical server processors.)
The examples below illustrate various ways in which LPARs and FLEX-ES can (and cannot) address the situation of providing CPUs intended for a uniprocessor VSE system and a dual processor z/OS system.
Fig. 5.9: CPU Relationships, VM and LPARs
In the first example, two uniprocessor LPARs partition the resources of a uniprocessor z/Architecture mainframe. This allows two mainframe OSs to run, one in each LPAR.
In the second example, two LPARs (one uniprocessor and one two-processor) partition the resources of a two-processor z/Architecture mainframe. The result of this partitioning is a total of three CPUs available to two mainframe OSs, but no LPAR provides more CPUs than are present in the underlying hardware. LPARs partition resources, they do not virtualize them.
The third example illustrates an invalid attempt to use FLEX-ES to provide three emulated CPUs. This attempt fails because the second Instance (indented as a two-CPU Instance for the use of z/OS) requires two server processors on which to run; one for each of its emulated CPUs.
The fourth example illustrates a valid solution using FLEX-ES. The server has two processors (both, it is assumed, enabled for use for CPU emulation). This allows both instances to run, as the second instance has two server processors available to it.
Note that although one of the restrictions in configuring FLEX-ES is the same as one of the restrictions on LPARs (no LPAR can provide more CPUs than there are CPUs in the hardware, no FLEX-ES Instance can provide more CPUs than there are (enabled) processors in the server) a FLEX-ES Instance is not the same as an LPAR. A FLEX-ES Instance is an emulation of one machine on another which requires a one-to-one correspondence between emulated mainframe CPUs and physical server processors. An LPAR is a partitioning of the physical server processors (and other resource) of a single machine.
Within any given FLEX-ES Instance, the number of CPUs in that instance is static. The number of CPUs is fixed in its configuration file and cannot be changed without bringing down the Instance and starting it up again.
Once an emulated mainframe CPU in an Instance has started on a server processor, it remains on that processor. It may share that processor with other work (unless it is dedicated; see below), but it never moves to a different server processor.
This is true for CPUs in both uniprocessor and multiprocessor Instances. When a multiple-CPU Instance starts its CPUs on multiple server processors, each CPU remains associated for the duration of the Instance with the server processor on which it starts. A CPU is never swapped off of one server processor and onto another.
The server operating system does no mainframe driven load balancing between processors. Server OS utilities such as the nice(1) command should not be used to attempt this. The use of the dedicated statement in FLEX-ES CPU declarations provides some coarse-grained control over the use of server processors. Finer-grained control of load distribution between the mainframe CPUs of an Instance is possible at the mainframe OS level through the use of IBM's VM operating system.
In any FLEX-ES server, some number of the server processors are "enabled" for use with FLEX-ES CPU emulation.
The number of enabled processors is fixed in the FLEX-ES license. Once a license is generated, the number of enabled processors cannot be increased without generating a new license. (Decreasing the number of enabled processors involves other issues and is not in general possible.)
For example, a uniprocessor server would have, of course, only one processor enabled.
A 4-way SMP server might have one, two, three, or four processors enabled.
A large server such as a NUMA technology server might have more processors enabled than can be utilized by a single Instance. This is not unreasonable; such a server could easily be running several Instances.
While the license defines the total number of processors which may be used (enabled) for FLEX-ES CPU emulation, it does not define which processors on the server are so enabled. When Instances start, they use processors either as available or as specified by bound and dedicated statements in their configuration. In no case, however, can the number of server processors in use for FLEX-ES CPU emulation exceed the number defined as enabled in the license.
Server processors which are enabled for FLEX-ES CPU emulation may be used in one of three ways:
"bound" and "dedicated" use introduces certain complications, though such use is often highly desirable, especially in production environments. These types of use will be discussed later.
In what for lack of a better term will be called "ordinary" use, server processors up to the number of processors which are enabled for FLEX-ES CPU usage may run processes of all types. These include:
Server processors in excess of the number enabled for FLEX-ES CPU emulation may run processes of all types except FLEX-ES CPU emulation.
For example, consider a four processor server with two processors enabled.
If a single CPU FLEX-ES Instance is started on this server, and if the CPU of this Instance is "ordinary" (not bound or dedicated) then the Instance will start its CPU emulation on any one of the server's four processors. It will not obtain the exclusive use of this processor, which may be running general server processes and/or other FLEX-ES processes.
As discussed earlier, once an Instance starts on a particular server processor or processors, it will not move to different processors. Nor will the server OS do any load balancing between CPUs. This is not relevant, however, to planning the configuration of the Instances on a server.
If another single CPU FLEX-ES Instance is started on this server, it will also start its CPU emulation on any one of the server's four processors. This may, by chance, be the same server processor as the first Instance, or it may be a different one.
Fig. 5.10: "Ordinary" Processor Examples I
If a third single CPU FLEX-ES Instance is started on this server and the other two Instances are running on the same processor, it may also run on that processor or it may run on some different server processor. If the other two Instances are each running on a different processor, then it will run on one or the other of those two processors. (Remember, in this example a maximum of two server processors are enabled.)
Since in this example a maximum of two server processors are enabled for FLEX-ES CPU use, this third Instance cannot start on a third server processor.
Fig. 5.11: "Ordinary" Processor Examples II
If now a fourth Instance, a dual CPU Instance, is started on this server, it will run such that at most two server processors are used for CPU emulation.
It is not possible to start a three CPU Instance on a server such as this one which has only two processors enabled.
Fig. 5.12: "Ordinary" Processor Examples III
Further examples could be constructed. The basic principles, in summary, are:
In "ordinary" use as described above, it is not possible to determine which server processor will be used for a particular mainframe CPU.
It is possible to specify this explicitly using the bound option on the statement defining the CPU.
In an SMP server environment, the bound option is not particularly useful, since all server processors are equal.
In NUMA server environments, the bound option option may be used to ensure that CPUs run on a particular quad. This is useful in keeping all processors in an Instance in as few quads as possible. It is also useful in keeping processors in the same quad as the Contiguous Reserved Memory that they use.
It is possible to make mistakes in the use of the bound option option which would require multiple Instances to use more server processors than are enabled. If this is done, the Instance or Instances which force such a condition will not start.
Fig. 5.13: bound Processor Examples
A bound CPU, as described above, runs on a specified server processor. It does not have the exclusive use of that processor, and other server work (including other Instances) can run on the processor.
A dedicated CPU runs on a specified server processor and has exclusive use of that processor. No other server work of any type can run on the processor.
The dedication of a server processor to a particular CPU emulation permits the most efficient use of that server processor by the CPU emulation. The use of this feature is recommended, when it is possible, if maximum performance is desired. As server processors continue to increase in performance, however, it becomes increasingly reasonable to run even production systems without the use of dedicated CPUs.
It is not necessary to declare all of the mainframe CPUs in a single Instance as dedicated; some may be dedicated, and others not dedicated. However, for performance reasons it is strongly recommended either that all CPUs in a single Instance be declared dedicated or that no CPUs in the Instance be declared dedicated. Declaring only some of the CPUs in an Instance to be dedicated may in some situations result in a performance decrease rather than an increase.
In a single server, at least one processor must be not dedicated. If an attempt is made to start an Instance which includes dedicated processors such that starting this Instance would cause the number of non-dedicated server processors to be less than one, the Instance will not start. This is a natural consequence of the nature of processor dedication. If all of the processors on a server were to be dedicated to mainframe CPU emulation, no processors would remain to run ordinary server work and FLEX-ES I/O emulation.
The example illustrated below shows four possible valid configurations which employ dedicated CPUs on a four processor server with two processors enabled:
Fig. 5.14: dedicated Processor Examples I
The example illustrated below shows invalid configurations:
Fig. 5.15: dedicated Processor Examples II
The example illustrated below shows further invalid configurations.
The first of these fails on several counts. It tries to run more dedicated CPUs than there are enabled processors. It tries to run more dedicated CPUs than there are server processors. It also tries to run two dedicated CPUs on a single server processor.
Fig. 5.16: dedicated Processor Examples III
Just as each FLEX-ES system has a specified maximum number of processors which it will enable for mainframe CPU usage, so also each system knows the clock speed of the server processors on which it is to run. As each Instance is started, it checks the speed of the processors it is running on. If one or more of these speeds differs from the expected value by more than two percent, then the Instance will not start.