The Effects of Using a Finite Number of Bits and Approximate Division in a Data Compressor

Abstract

In the design of a data compressor one of the basic problems is error due to the use of a finite number of bits in calculating various parameters. This is error due to truncation (or round off) after a set number of bits to the right of the binary point. Another error that could be introduced to the system is that error caused by the use of an approximate divide instead of a full divide. It is the purpose of this study to find the effect of these two errors so that (1) a judgment may be made as to how many bits to the right of the binary point are necessary, and (2) find if a double shift approximate divide may be used instead of the slower full divide. The study is divided into three basic parts (1) the effect of truncation (or round off), (2) the effect of the double shift approximate divide, an (3) the combined effect of truncation (or round off) and the double shift approximate divide. Each one of these error-causing phenomenon has two variations; i.e., an error caused while compressing data and an additional error caused while reconstructing. The error during compression distorts the tolerance limits, and the error in reconstruction causes distortion of the reconstructed line segment. Each of the errors leads to an overall error in data magnitude over and above the normal allowable error of one tolerance level. For all of the analysis the maximum run length of the compressed data is assumed to be 128. A summary of the study is found in Table 6.