What is the difference between a variable-blocked format and a fixed-blocked format?
I have encountered these terms in a data file definition document for some data that I'm getting from a Mainframe system.
I failed to find definitions of these terms in any glossary or Google search - just mentions of them.
Can anyone shed some light on the precise meaning of these terms?
Answer
Forget what you know about data on a harddisk under Windows or Unix/Linux.
Fixed-length records (which can be Record Format (RECFM) F for unblocked, FB for blocked (they can also be FBS, which is Fixed Block Standard, but for a simple single-use dataset (file) it is equivalent to an FB) just contain data, and a known and unchangeable amount of data.
The data they contain can by any of the 256 bit-values possible. There are no record delimiters.
Variable-length records, (RECFM V or VB (VBS is Varible Blocked Spanned, which is something else than a simple record) are prefixed by four bytes (the Record Descriptor Word (RDW)) the first two of which indicate the length of the record. So they contain control information and data. There are no record delimiters (because the length is known instead).
Blocks for a VB dataset also are prefixed by a Block Descriptor Word (BDW), four bytes, the first two of which contain the length of the block.
Data is written and read in blocks only, with deblocking done by io routines. The larger the blocksize, the fewer the ios. However, because the maximum blocksize is less than the size of a track on the disk, the most efficient blocking is "half-track blocking" with a blocksize up to one record less than 27,998 or equal to it.
Fixed-block records have less overhead, because the only information (blocksize (BLKSIZE) and maximum record-length (LRECL) is provided (in the catalog, or the JCL, or by the program, especially if written in Assembler), not stored in the data.
For variable-block records, there are two types of control information (BDW and RDW) embedded with the data, as well as the other information.
The programmer on the Mainframe does not need to know much about this. A program will deal with records, blocking details can be external to the program.
Here's an FB Block:
DATADATADATADATA
How that block breaks down into fixed-length records is held externally to the data. The position of the third record in a block can be calculated. There is a low-level access method which uses this fact.
Here's a VB block:
BDWRDWDATARDWDATARDWDATA
The position of the third record in the block is only known the records prior to it have been processed by the io routines.
Choosing the wrong type, or the wrong record-length, or the wrong blocksize can significantly impact program performance.
When data is transferred off a Mainframe, there are options to remove the RDW (if present), insert delimiters, drop trailing blanks and things like that, so that the data now looks like a native file. Transferring to the Mainframe goes through the reverse process.
If you are going to be receiving data from a Mainframe or sending data to a Mainframe, do so only in character format. No "binary" or "packed" fields, and use explicit signs, explicit decimal places or scaling factors. You'll save yourself so much trouble, and have yourself in auditors' good books.