Google Architecture 3747

Phil Smith III

about ten years ago, i had opportunity to spend some time with people at NIH's national library of medicine. at the time, they had a mainframe bdam implementation that dated from the late 60s. two of the people that had worked on the original implementation from the 60s were still around. we were able to exchange some war stories ... because i had an opportunity to be at university that was involved in the original cics product beta test. they had an onr grant to do a library project and i got to shoot cics and bdam bugs.

at the time, somebody commented that there was something like 40k profession nlm librarians world-wide. the process was that they would sit down with a doctor or other medical profession for a couple hrs, take down their requirements and then go off for 2-3 days and do searches ... eventually coming back with some set of results.

nlm had pbutted the search threashold of extremely large number of articles back around 1980 and had a severe bimodel keyword search problem. out to five to eight keywords ... there would still be hundreds of thousands of responses ... and then adding one more keyword (to the search) would result in zero responses. the holy grail of large search infrastructures had been to come up with the number of responses greater than zero and less than a hundred.

in early 80s, nlm got a interface, grateful med. grateful med could ask for the return of the number of responses ... rather than the actual responses. grateful med would keep tracks of searches and count of responses. the person doing the search seemed to involve a slightly, semi-directed random walk ... looking for a query that met the holy grail ... greater than zero responses and less than one hundred.

finding acceptable responses is problem common to most environments after the number of items pbutt billions, regardless of the implementation platform.