Legend has it that Grace Hopper used to carry around a short length of wire to illustrate the equivalence of time and distance. “This is a nanosecond,” she would say, because the wire was about the length that electricity can travel in copper wire in that amount of time. The message is that in computing, distance is not your friend. The greater the distance over which a signal must travel, the greater the processing delay it introduces.
Despite the diligent efforts of many folks, the speed of electricity in a wire (or light in an optical fiber) hasn’t increased much since those days, and the prospects don’t seem promising. The focus has instead been on optimizing protocols by either reducing the number of communications that must occur, or by overlapping/parallelizing the communications with other processing to make them flow asynchronously with respect to other activities. That is the approach taken with z/OS Parallel Sysplex support for Asynchronous Coupling Facility (CF) Duplexing for Lock Structures.
How CF Duplexing Works
A z/OS Parallel Sysplex is a tightly coupled cluster of operating system instances that cooperate to share data and process in parallel against a common client workload. CF lock structures provide a sysplex-wide point of serialization for controlling concurrent access to shared data in the cluster, which is essential to preserve data consistency and integrity across the cluster. For several years, CF lock structures have implemented a high-availability option called CF Duplexing that creates and maintains two copies of the CF lock structure, in two different CF machines. The machines are physically separated within a single data center, by meters, or across multiple data centers, spaced out by kilometers. The problem is that when lock structures are duplexed, they must synchronously exchange signals to coordinate the obtaining and releasing of the locks between the two separate copies of the lock structure in “lock step” with one another. The latency effect of distance weighs heavily here, especially when duplexing between distant data centers.
CF Duplexing Availability
Many Parallel Sysplex clients have wanted to implement CF Duplexing for lock structures for the availability benefit it provides, but haven’t been able to tolerate the speed-of-light delays that the synchronous communication between the CFs introduces. It slows down their application performance too significantly. Instead, we have optimized the duplexing protocol between the CFs to make it asynchronous. Now Parallel Sysplex applications can obtain or release locks in one CF structure, without having to wait for synchronized updates to occur in the other CF structure. The lock updates are communicated asynchronously to the other CF. The net result is much less delay experienced by the application that is obtaining or releasing the locks, and much more efficiency and parallelism between ongoing application processing and the asynchronous propagation of the lock information from one CF structure to the other.
Some of you might be thinking, what good is having a second copy of the CF lock structure if it isn’t up-to-date with the latest lock information? It’s getting updated asynchronously, so at any given moment it is out-of-date relative to the primary copy. Of course, we have lost something by using an asynchronous protocol. But the “magic” behind the solution is in the tight integration between the different layers of the stack. Those layers include the transactional commit behavior of the data-sharing exploiter (DB2) and its lock manager software (IRLM), the operating system (z/OS), the Coupling Facility firmware (CFCC), and a coordinated recovery/failover process that takes place should the primary copy of the lock structure fail. This coordinated recovery brings the secondary copy of the lock structure into sync with the primary copy, and works in conjunction with the transactional commit and backout/recovery processing of DB2 to ensure complete consistency after a failure.
Clients who have been reluctant to duplex their lock structures because of the performance overhead should consider implementing this new solution, especially those clients operating in a multi-site Geographically Dispersed Parallel Sysplex (GDPS) environment over significant distances. Asynchronous CF Duplexing can also help enable GDPS’s so-called continuous availability Disaster Recovery (DR) solution. In the DR solution, critical data and locks associated with a workload are duplexed across two sites, so that the workload can seamlessly continue processing at the surviving site if either site fails.
Delivering an integrated solution
Delivering an integrated solution such as this requires the efforts of a close-knit, collaborative, agile team, and that has most definitely been the case during the development, test, and delivery of this function. Asynchronous CF Duplexing grew out of a shared understanding of the client problem to be solved, and a concerted cross-functional effort to deliver an effective solution. This close teamwork, careful attention to a well-documented design, and use of agile and integrated joint firmware/software function test practices led to an exceptionally smooth delivery of this function both to internal system test areas and to external clients.
The Asynchronous CF Duplexing for Lock Structures solution requires z/OS 2.2 with PTFs (including RMF PTFs for performance monitoring), z/VM 6.4 with PTFs (when running in a z/VM Guest Coupling environment), z13 or z13s with Coupling Facility (CFCC) level 21 with Service Level 2.16 or higher, DB2 V12 with PTFs, and IRLM V2.3 with PTFs.
About the Author
David H. Surman is a Distinguished Engineer specializing in z/OS Core Technology and Parallel Sysplex. He has worked on the design and development of the z/OS operating system for over 30 years. His expertise in Parallel Sysplex includes the z/OS operating system, the design and architecture of the Coupling Facility, and the software and firmware support for clustering, including Coupling Facility resource management, locking and serialization, and sysplex recovery.