Zim 8.00 Release Notes

Lock Management

A. Operating System Independent

With Zim 8, all locks are managed using data structures in shared memory.  The locks are accessible to all Zim instances (no operating system requirement) and are organized using a combination of direct indexing and scatter tables.

All lock management objects are pre-allocated by Zim Server and placed onto "stacks". This allocation and stacking enables a Zim process to acquire or release a lock in approximately 500 machine cycles (200 000 lock operations/second on a single CPU 1GHz machine, 400 000 lock operations/second on a dual CPU 1GHz machine).

B. Automatic and Immediate Deadlock Detection

When a Zim 8 instance detects that a lock is not available, it places the request into a pending lock list.  A special algorithm recursively walks through the lock memory, using the pending list as a reference, searching for processes that are waiting for a lock and are held by the original instance.  If such a lock is located, the new pending lock will introduce a deadlock. The process that attempts to introduce the deadlock is immediately aborted and the transaction ends.

C. Deadlock Reduction

Since Zim 8 instances see the same data all the time and when the serialization locks, it is able to alter the way transactions are internally executed.  This alteration results in a substantial reduction in deadlocks.

D. Serialized Lock Requests

When a lock cannot be acquired, a request is placed on the pending lock list.  Whenever a process removes a lock, either from a commit or abort action, then that process checks the pending list.  The process starts with the oldest request in the list and tries to apply the lock on behalf of the requestor.  If the lock was successfully applied, the request is removed from the list and the requestor is informed that the lock has been granted.  This communication occurs directly between two or more Zim 8 instances with no need to swap out or get Zim Server involved.

E. Timeouts

In Zim 8, a configuration variable exists that regulates the amount of time that a lock request can remain on the pending lock list.  The Lock Manager contains a thread that periodically examines the contents of the pending lock list.  If a request is found that exceeds the allowable time-out value (configurable), the request is removed from the list and the Zim 8 instance is informed that the request is aborted.

F. Abort Retry

During the Abort process, the Zim 8 instances are subject to a delay penalty.  This penalty increases with the number of consecutive aborts (to a preset limit).  By including a random element, even a large number of Zim 8 instances -attempting the same operation - are able to complete the operations in a short time frame.

G. Lock Hierarchy

When a Zim 8 instance tries to get a lock and there are no locks on the ZIM file, it checks the pending lock list.  If there is a pending file lock, read or write for the file then the Zim8 instance will attempt to grant the file lock.  The Zim 8 instance places a lock request at the end of the pending list and waits.  This ensures that older Zim 8 instances are given priority to complete their transactions.  Once the older file locking transaction are completed, the pending locks are applied to the file.

H. Shared Data

When a Zim 8 instance requires a piece of data, it first checks the shared memory (via direct indexing and scatter tables of linked lists) to see if the data is already in memory.  If the data is loaded, the Zim 8 instance registers itself as a user on the data block and the data is addressable by the Zim 8 instance.  No indirect lookup, no address translation is required and it appears as a local memory object.

If the object is not in the memory, then the Zim 8 instance and Zim Server request that the block be brought off the disk and placed in the shared memory.  The Zim Server is the only process that reads the database.  The Zim Server reads the disk, places it into shared memory, places the address of the requested block into the rendezvous area and informs the client the data is available. Once the data is loaded all clients have access to the exact same data, not just a copy per instance. The benefit to this is that the total amount of memory required to support a large number of users is reduced and it allows changes to the database to be immediately available to other processes.

I. Reclamation

When the Zim Server runs out of a given object type (within an object stack), it will attempt to reclaim some unused or un-referenced object of the same type and place it on the stack.  The amount reclaimed each time is configurable but typically it is 1 000 or more.  The administrator has the ability to indicate which zimnnnn file will be the first candidate for data reclamation.  Therefore, the developer can create the mostly read-only entity sets with lower numbers and rapidly changing entity sets with higher numbers.

e.g. If the administrator indicates that the first entity set to be reclaimed is 7, then any loaded data dictionary, entity set definitions, relationships, roles and documents that have been loaded into memory will remain there. If the entity value is set to 100, everything including menus and compiled forms windows, once loaded will remain in memory and never be reclaimed.  If the administrator places an entity set in zim0110 that contains a table of read-mostly data, used by all transactions, and then sets the value to 111, all the data when referenced will be locked into memory.

J. More Memory = Greater Performance

With the improvements made in Zim 8, more physical memory is now available for Zim Server.  If virtual memory is used, the operating system can begin paging and reduce the throughput of the system.  The reduction in memory usage stems from the increase of system CPU time and disk activity.

K. Real Transactions

No Zim 8 instance can access any information unless a transaction has been granted by the Zim Server.  When the Zim 8 instance requests a commit transaction, the Zim Server writes all of the blocks altered by the transaction into a transaction file (common to all Zim8 instances).

Note: A big performance gain is achieved by having the transaction file be the only item in a given spindle.

Once the data is written to the disk, the transaction is considered committed and the client is informed.  Since the information in memory reflects the current state of the database, there is no need to re-read the disk.  

If a Zim 8 instance is issued and aborted, then the immediate checkpoint is issued and the modified blocks are read back from the disk.  In addition, the shared memory state of the Zim8 instance is cleaned.  All other Zim 8 instances references to the data are unaffected.

It is now possible to:

TRANSACTION

ERASE ENT MyEnt

DELETE ALL Fields

QUITTRANSACTION

L. Checkpoint

On start-up, on demand, when a configurable number of transactions have occurred, or when a configurable number of dirty blocks exist in the transaction file, the checkpoint thread of the Data Manager is signalled.  The checkpoint thread prevents the Data Manager from committing or aborting any transaction. It then closes the transaction file, moves it, re-creates an empty transaction file and opens it (the Data Manager and the checkpoint thread use (by reference) the same file handle).

The checkpoint thread then allows the Data Manager to again process the transaction. The checkpoint examines the renamed transaction file and applies the changes to the Zim Database.  The checkpoint thread is the only process that writes to the database. Once all the updates have been applied, the checkpoint places the renamed transaction file into the on-line backup directory (optional) with a name that indicates the first transaction in the file.  The contents of the backup tape and the sum of the online backup files is the current state of the database on the disk.

M. TRANSACTION READ Command

The option READ from the command TRANSACTION is no longer applicable. If it is used, it will be ignored.

N. SET TRANSACTION DEFAULTTYPE Command

The command SET TRANSACTION DEFAULTTYPE is ignored by Zim 8. Its syntax is accepted only for the purpose of compatibility with previous versions and may be removed in the future.

O. SET IMPLICITTRANS Command

The command SET IMPLICITTRANS is ignored by Zim 8. Its syntax is accepted only for the purpose of compatibility with previous versions and may be removed in the future.

Server Side Sequence Numbers

Zim 8 implements a sequence number on the server side that can remain open at all times.  When a Zim 8 instance requests a sequence number for a given file (via the rendezvous area) the server copies the memory sequence number into the rendezvous area and informs the client.  Sequence numbers are maintained in memory and are now part of the header block, Zim Server does not need to continually update the disk.  The sequence numbers on the disk are corrected during a checkpoint or when the first Zim 8 instance requests a record from the given file.  Therefore, there is no contention between Zim 8 instances for a lock on the sequence number file.

Exclusive and Shared Access to ZIM Database

Zim 8 implements a new mechanism that enables a specific user (usually the database administrator) to grant himself/herself exclusive access to a ZIM database. Under normal conditions, database access is shared among users; however, in some situations, the administrator may want exclusive access to the database (i.e. urgent program recompilation or database maintenance).  To obtain exclusive access the database administrator can issue the following command:

SET DATABASE ACCESS EXCLUSIVE

At this point the ZIM Server stops accepting new incoming transactions, waits for the existing transactions to end and then grants exclusive Database access to the specific user.  Any subsequent requests from other users are kept in a suspended state. As soon as all transactions stop, all files are closed. Therefore, it is possible to issue operating system commands to deal with physical database files. When the exclusive access is no longer required, the following command can be issued:

SET DATABASE ACCESS SHARED

From this point, Zim Server activity is resumed and all suspended state users can continue with their normal processing.

ZIM Directories

ZIM 8 provides better handling of ZIM directories (local and foreign) by increasing the number of available directories from 100 to 240.  

Foreign Directories

ZIM 8 allows file belonging to foreign directories to be written just like any other database file. All users belonging to ZIM databases accessing these files can issue normal read and write operations on them.

Object Limitations - ZIM Databases

ZIM 8 enables individual ZIM directories to hold as much as 33 million objects.

Security

• 32 000 users can now access a ZIM Database;

• each user identification (UserName and UserId) must be unique

• The number of organized user groups is increased to 32 000

• To run ZIM, a user name and password are required.  If these are not provided at point of login, the user will be prompted.

Timestamps

Before ZIM 8, a Zim Database could be corrupted when a user inadvertently executed a piece of compiled code that did not match the current definition of the layout within an entity set.  Now all data entity set definitions, roles and fields carry a last modified timestamp.  When a piece of Zim code is compiled, it is give a timestamp equal to the timestamp of the most recently modified object referenced.  When an entity set is created, it is give a timestamp of the most recently modified object definition used to create the entity set.  If the timestamp on the entity set is greater than the timestamp on the compiled code, Zim Server aborts the transaction and indicates that the code must be recompiled.  Simply erasing and creating the entity set will not cause this to occur.  It is the underlying objects used to create the entity set that are examined and not the create date of the entity set.

Per User Objects

Entity Sets, Relationships and Documents can be per user by setting its corresponding attribute in DC. In particular, per user Entity Sets and Relationships with fields can be created on the fly using the following command:

INITIALIZE PERUSER <object>

This command creates a corresponding physical file for the object without the need for storing and copying empty files in the database.  Eventually, the existing file is erased.

Note: Only objects explicitly defined as Per User can be initialized this way.

Upon definition of the object, DC has the ability to deal with Per User files. Per User files no longer need to be indicated in the areas.zim file.

Multi-user Development

With Zim 8's shared memory model and the changes made to field, role and entity set definitions, erase and create operations are immediately visible to all Zim8 instances after a commit transaction (expressed or implied).  If a Zim 8 instance is within a transaction, it will be aborted and immediately started using new definitions.

CPU Performance

During testing, Zim 8 has demonstrated a near linear increase in performance when additional processors are available.  This is primarily due to having only one process read from the database, one process written to the database, transactions on their own spindle and a marked reduction in the interaction between Zim8 and the operating system (OS).

ZIM Executable

ZIM now always run in multi-user mode. All data dictionary operations are also accomplished in multi-user, thus requiring ZIM Data Dictionary.  The increased security measures guarantee that all accesses are safe and protected. ZIM can be invoked from anywhere at operating system level because its parameters identify where the logging process is happening through the parameters:

Zim –u <User Name> -p <password> -n <database name>

If one of these parameters is not provided or if the initial connection is wrong, then the user will be prompted to provide the correct information.

ZIM Utilities

Some existing ZIM utilities have been changed to comply with the new Zim 8 rules (security and reliability) and others have been deleted. In this version of ZIM (8.0), utilities can only be run when Zim Server is not running. Also, they can only be run from within the database.

Existing utilities (the functionality and parameters are the same as previous versions of ZIM except where mentioned):

• ZIMInit

• ZIMDD

• ZIMFiles (parameters +p, +u and +i are deprecated. See comments below for ZIMFiles)

• ZIMFix

• ZIMXtend

The following utilities are no longer supported in ZIM 8:

• ZIMDump

• ZIMUnlok

• ZIMRcvr

ZIMFiles Utility

The options +p and +u are replaced by functions to give administrators better control and security over objects belonging to a database. See functions $UserId, $GroupId and $Permission.

The option +I is no longer required because the ZIM database now accepts as much as 33 million objects.

New Functions

Three new functions are introduced that provide better and safer control over ZIM objects:

• $UserId: returns the UserId that created the given object;

• $GroupId: returns the Group Id of the user that created the given object;

• $Permission: returns the permissions set to the given object.

Examples:

OUT $UserId(MyEnt)

OUT $GroupId(MyEnt)

OUT $Permission(MyEnt.MyField)

ZIMExplore

Zim Explore is a graphical tool that manages all ZIM databases from all machines. The Administrator can control the entire ZIM network from a single location and can perform various tasks manually or automatically.

• Configure Zim Server

• Create and configure any ZIM database

• View error logging

• Activate ZIM products

• Perform utility functions over a specific ZIM file (e.g. zimfix)

• Take corrective measures

• and much more

Note: Even though Zim Explore is a graphical tool (running on Windows only), it controls all the ZIM databases either a Windows or Unix platform.

Configuration Files

ZIM 8.0 has four different configuration files:

1. ZIMDB.ZIM: this is the most important configuration file because it tells Zim Server where all databases are and provides control to access these databases. It’s located in the directory where the ZIM product was installed. Each line of this file contains a reference to one ZIM database in this format:

<database number>; <database name>; <absolute path>

The <database number> is any unique number from 1 to 255; <database name> is an unique name up to 18 characters identifying the database; The <absolute path> addresses where this database is located. This file cannot be changed while Zim Server or ZIM Backup are running.

2. ZIMCONFIG: The ZIMConfig files describe specific configuration options for each ZIM 8 component. Zim Server  reads the zimconfig.srv file located in the directory where the ZIM product was installed, whereas ZIM and all utilities read a zimconfig.zim located where the database is (exactly where <absolute path> is pointing to). There are no other configuration options files.

3. DIRS.ZIM: Same configuration file as previous versions of ZIM. It describes the foreign directories being accessed by this application. If existing, it’s located in the <absolute path> (in other words, each <absolute path> can have one dirs.zim file);

4. AREAS.ZIM:  Same configuration file as previous versions of ZIM. It contains references to files located in different places other than the <absolute path>. Previous options SHARED, NSHARED, per user (”)”, R and RW are now ignored. ZIM only deals with per user files explicitly defined in the Data Dictionary.

The relationship between these files is as following: ZIMDB.ZIM describes where databases are. Each database contains a zimconfig.zim file, plus an optional DIRS.ZIM and an optional AREAS.ZIM. Each entry in DIRS.ZIM point to a foreign directory which in turn might have an AREAS.ZIM.

With Zim Explore, default, maximum, minimum and current values can be captured with one mouse click - guaranteeing that all options are correct.

ZIM Server

The ZIM Server must be running at all times and only one instance per machine.  It provides all the functionalities needed to read and write data for the clients connected to it, and guarantee data integrity through a dedicated lock management, a checkpoint of ongoing transactions appropriate session ending.

ZIM Backup Server

ZIM 8 provides a mechanism to back up databases in real-time and on-line using ZIM Backup Server.

While one instance of Zim Server deals with various ZIM databases and all their associated users making changes in these databases, Zim Backup Server can apply these changes to a selected number of back up databases. The delay between the original change and the back up database is accounted in a few seconds only (if the original database crashes, users can be immediately redirected to the back up database to continue working).

Whenever needed, Zim Backup Server can be stopped to allow hard copies of the databases to be made (to tape, for instance) while the normal database activities go on. Later, backup can be restarted, continuing from the point it was stopped.

For complete details on how to use Zim Backup Server, refer to Zim Documentation.

Zim 8 Help

Zim 8 Help and documentation is comprised by an extensive compiled help documentation (zdl.chm file) and release notes specific to each version of Zim 8, all included with the Zim 8 software.