A practical use for old system emulators

I responded to a request for help on data formats used on ICL 1900 systems by the Population Studies Centre, University of Pennsylvania for

The problem was that they had a set of data (14 reels of 1-2" magnetic tape), from the 1977 Malawi Census, that did not apparently match the census data expected and appeared to have binary or some form of compressed data embedded in it. I took a look at a sample of the data supplied and wrote some programs to unscramble it.

The first step was to examine the sample files (using a 1900 tape print utility on a PC) to evaluate what they consisted of, and to ensure that they did resemble 1900 data. The files supplied, in ÔbitstreamÕ format, contained:-

a) a standard 1900 series tape header label b) a standard 1900 series start of subfile followed by variable length data records c) a standard 1900 series end of subfile

I then re-buttembled the files into a 'tape' that could be read on the George 3 system and wrote some PLAN programs under George 3 to look at the data. The data consisted of variable length records in the range 3 - 19 words in length and I was lucky that there was one 19 word record, which corresponded with the original Census Form.

By extracting the records by record length, starting with the longest, I was able to make some sense of them and the 'funny compressed data' at the end of the records. The data at the end of the record consisted of codes that described how the record had been compressed, with the last word of the record containing a count of the number of compressions for that record.

The funny data was counter-modifier words that described how many words were duplicate with the preceding record and they where in the record. The compression turned out to be extremely simple, just drop duplicate words from the record, and not specifically related to the data, but it did reduce the data volume to about 2-3rds of the uncompressed volume. Various programs were written, and modified as a better understanding of the data evolved.

In addition to just de-compressing the records into card images that could be output and then transferred to a *nix system for future use, I also wrote programs to recreate a Census Tape and to process it and produce a Census Report.

Having proved the programs on the sample data, I was asked to complete the processing of the full set of data tapes. A Perl program was needed to convert the bitstream files that had been created from the original tapes to the format required by the G3 system.

As a side note, the Perl programs had to be run under Unix, when run under Windows they appeared to work, but the file sizes were larger than the amount of data converted and random rubbish was interspersed within the valid data, which wasted an afternoon looking for a program error that wasn't there.

After conversion, I found that there were 3 separate multi-reel files, with all of the reels available. Running the decompression program on the first set of tapes and then running a census report, resulted in roughly a 50% population increase.

Decompressing all the tape sets produced 96 sets of data, when we were looking for 24 sets of data (1 per district). Manual analysis of the extracted files showed that we had partial and duplicate sets of data for the districts. In most cases, a single file was an exact match on numbers for a district from published data. In a couple of cases, a group of partial files had to be used.

Once the required files had been identified, I was able to create a new Census Tape and run a Census report from it. The numbers matched, apart from one district where 520 people had been lost, from a total population of 5.5 million.

There was no requirement to create a new tape, the relevant files were transferred to a *nix system for future use, but in my view it rounded off the work, leaving a new complete 'tape' for posterity.

For more information on the programs, see

Was it necessary to use George 3? No, all of the work could have been done in Perl, but it was more fun, and to my way of thinking easier, to work in a 1900 environment, after all I did this work for enjoyment, not for financial reward.