Data File List

Document path Reference Contents > Data File Repair > Data File List

Data File Repair - Introduction

The Data Fix function provides a method of recovery of data from a data file that has become damaged such that re-indexing using Caliach Vision's Re-Index function fails (See the Re-unite Missing Relatives and Kill Orphans and Re-index Files topic).

The Omnis Studio Data File has a complex internal structure. This structure is optimised to provide fast access and update of data. If this structure is compromised by a terminal, server or network failure full operation of Caliach Vision may not be possible.

To re-create a data file where there is serious index damage you may need to write some program code. You therefore should have a Design edition of Omnis Studio, and use this with the Data Fix Function.

Data File Structure

The Data File is organised into 512 byte blocks. Blocks can contain data, indexes, empty space, control or internal directory information, known as header data. As the data is entered, edited and deleted, so blocks are re-arranged and over time data gets randomly located within the operating system file. There are two methods Omnis Studio uses to locate records:

Failure

When a system breakdown occurs during a file update, insert or delete operation, it is possible that the user's computer may not have completed all of the management update functions necessary to leave the Data File with full integrity. The most likely damage is in the integrity of an index.

Alternatively, the header can get damaged.

If the header is damaged alone, operation of Caliach Vision may continue perfectly normally as indexes are able to point to the data blocks correctly. However, when indexes are damaged several problems may manifest themselves.

Repair

When the data file becomes damaged the best solution is to use a backup taken prior to the failure. However, this is rarely possible, either because there has been too much subsequent processing on the system since the last backup, or the damage has only manifested itself well after the failure and the existing backups contain within them the same damage.

The first action to take when damage is identified is to get everyone off the system. If a backup must be used there is no point making more work for users by allowing them to continue. PLUS any attempt to manipulate a damaged data file will lead to an escalation of damage and a reduced chance of recovery.

Check Free Blocks

Check free blocks is an Omnis utility function that scans the entire data file for empty blocks. In doing so it re-builds the Free Block Chain.

To Check Free Blocks:

  1. Backup your data file.
  2. Launch Caliach Vision.
  3. Operate Data File Browser from the Advanced menu.
  4. Select the data file in the browser, right-click and operate Check Free Blocks.
  5. If the process does not complete successfully, restore the backup, and use the the Data Fix Utility function.
  6. If completed successfully, re-launch Caliach Vision and test whether the problem still exists. If it has not solved the problem, try re-indexing.

Re-Index

See the Re-unite Missing Relatives and Kill Orphans and Re-index Files topic.

Re-Create

When the Data File damage is such that the Omnis Studio utility and Reindex functions cannot recover the data, you can try to Re-Create the Data File using the Data Fix Utility function.

This utility has two functions:

To test that data from each of the files within a damaged Data File can be extracted in full. If not, it may be possible to jump around a damaged area of data records and extract the majority of records. The later requires programming.

Note NOTE: This is NOT a test of the datafile's integrity for use with Caliach Vision. It is simply a test of the ability to extract data from the datafile. Testing here may well produce no errors, even though in Caliach Vision errors are experienced.

To extract all data from each file and save it in a completely new Data File. This will re-create a new Data File containing data identical to your existing damaged Data File.

This program uses a unique index known as the Record Sequence Number to access data records. All files must have a RSN field, to be accessed by this program.

Getting started

Start by making a backup of your Data File.

Warning WARNING: The Data Fix process CAN NOT operate with a datafile which has been re-indexed unsuccessfully. You must restore the backup made before the unsuccessful re-index.

You must ensure that any custom file classes you use in the datafile are correctly copied into you Caliach Vision library. If you do not you may well loose data in this process.

Data Files

When you first open the Data Fix window, the Source data file will be the current Caliach Vision data file. The Source datafile is the one containing the suspect data. Use the Select button to change it, if needed.

The Target datafile is the one that you will copy data to. Typically, you will create a new datafile which will be free of any structural damage. If you have already done this, use the Select button to choose it. Click Create to create a new data file. The system will default the size settings to be equivalent to the Source datafile. For full details see the Create New Data File topic.

Warning WARNING: You should make sure the Target data file is made large enough and with enough segments. You do not want any copying to be incomplete.

With only the Source data file selected, all Test and Copy functions are available to you except the Copy Selected to New Data File function, which needs a Target datafile.

The window has two tab panes.

Test and Copy

Repair Adjustments

Test and Copy

This tab pane lists all datafile classes and examines the Source datafile for its contents with respect to these.

Warning WARNING: The Data Fix Utility does not work with datafiles connected through the Omnis Data Bridge. For details on datafile access see the Data Files and the Omnis Data Bridge topic.

Customisation CUSTOM CAPABILITY: If you have added any custom file classes to Caliach Vision, you MUST operate the Re-build Code function before proceeding.

Check Selected

Select all the files in the list for a full test.

The test looks for duplicate values in an explicitly unique index, which is indicative of a damaged index. If a duplicate index value is spotted a report of such will be made. If not, the number of records scanned, the Count value, should be equal to the apparent number of records in the file, which is information held in the header, the Record value. Missing records, usually at the end, indicate a damaged index.

Each file is scanned in turn. When complete a message is placed in the list. The Count value should be the same as the number of Records shown on the list. This can be a long process if you have large files. The ETA (Expected Time of Arrival) time is given after the first file is scanned and re-estimated after each subsequent file.

When complete, check that all the Count values are the same as the Record values. If they are, all data is recoverable and you are ready to copy the data from the Source to the Target datafile.

Warning WARNING: This is NOT a test of the datafile's integrity for use with Caliach Vision. It is simply a test of the ability to extract data from the datafile. Testing here may well produce no errors, even though in Caliach Vision errors are experienced.

When a Problem is Identified

There are three possible problems that can be identified during the test.

1. A point is reached in a particular file where an Omnis Studio error breaks the operation of the routine and, if you are using a development version of Studio, puts you into the program debugger and the method line in which the error was identified.

In this case you must use the debugger to establish the point in the index where the problem manifests itself. The debugger is likely to trigger on the Next command, if this is the case use the Stack menu to go back to the previous method. Point with the mouse to the field "SEQ" and right-click on it. The field debugger menu will popup and show the value of the index where the damage starts.

2.The list result says "Duplicate RSN identified = 999999". Make a note of the RSN value shown as this will indicate where the index damage starts to have an effect on data access.

3.The list Count value is less than the Record value. Again, make a note of the RSN value shown as this will indicate where the index damage starts to have an effect on data access. If this count is out by a few, it is probably of no consequence.

In all these cases, you need to modify the methods to try and bypass the damaged records so that as few as possible records are lost when the Data File is re-created. See Repair Adjustments below.

Copy Selected to New

Once you are satisfied that you can recover all data (or as much as is practical), you need to perform this Copy function.

Select all files in the list. Click on the Copy Selected to New button. The copy routine first reads all data from the Source datafile into a memory list, in blocks of 5,000 records. When complete the contents of the list is saved into the Target datafile, record by record.

This process may take a long time. When complete you will have a newly constructed datafile, with all new indexes and a new clean header.

Note NOTE: Because the process takes one file at a time, the resultant data file will, in effect, be internally de-fragmented. Improved performance should result. As far as the operating system is concerned, the datafile is likely to be physically fragmented on the disk. User a defragmentation program.

At the end of processing the list will show the results.

Now you can launch Caliach Vision, select the new datafile and check that the data is all there as advertised. Don't destroy any backups until you have run with the new datafile for a while and are confident all is well.

Button

Action

Rebuild Code

Rebuilds the program code to accommodate the latest version of the data file structure. Important if there have been custom file classes added to the program. It re-creates the method code in the cDataFixRepair code class to accommodate all file classes within the Caliach Vision library. You only need to operate this after you have added file classes.

Customisation

CUSTOM CAPABILITY: There are some files in Caliach Vision that are referenced by other files through the Record Sequence Number (RSN). For these it is imperative that the records in them end up having the same RSN value, otherwise the reference in the other file(s) will be lost. The one file in Caliach Vision that needs this special treatment is ENTFILE. If you have custom files which use the same technique of reference you can add them to the method in cDataFixRepair/KeepSeqFileList before you Rebuild Code. E.G.:
Quit method '/ENTFILE/MyFileName/'

Print List

Prints the listing.

Check Selected Files

Checks that all records in the selected files can be extracted and copied to a Target data file.

Copy Selected to New Data File

Copies all available records in the selected files from the Source data file into the Target data file.

Back to top

Repair Adjustments

Making a Program change to jump round duplicate RSN index values.

When you encounter a duplicate RSN, the program can not pass that discontinuity using the normal Next command. You need to alter the code to jump round the discontinuity.

If BOMSEQU=9999998
Find on BOMSEQU {9999999}
Else
Next
End If

Where 9999998 is the HangUp Number and 9999999 is the Recover Number.

Button

Action

Generate New Loop Program Code

Having entered a Hangup Number and a Recover Number, click this function to generate code in the method text.

Edit Program Code

To manually adjust the Loop program code to accommodate special repair circumstances.

Field

Description

File

Name of the currently selected file class in the Test and Copy tab listing.

File description

Selected file class description.

Message

The result of previous testing as shown in the list.

Method text

The current program code for the segment controlling index looping.

HangUp Number

The RSN number after which you want to jump over a damaged index section.

Recover Number

The RSN number you want to jump to to get around a damaged index section.

Log Differences in Selected Files

Compares records in the Source and Target data files for files selected in the list. A log is created.

This function requires both a source and target datafile. First, select the internal files you want to compare in the list on the Test and Copy tab pane, then click on this button. The Trace Log window will open and report. For each file selected the process scans all records in the file in record sequence number order. It identifies the number of any missing or extra records in the target datafile. For each record it compares the contents of each field and will report the first it finds where there is a difference. It uses a binary compare function for list, row and other complex data types.

Back to top

Compiled in Program Version 3.10. Help data last modified 22 MAY 2008 07:42. Class wDataFix last modified 4 MAR 2010 11:06:10.

Document path Reference Contents > Data File Repair > Data File List