ANU Data Mining Group

News:

• 26 November 2008: Slides from several recent presentations and two new publications have been uploaded.

• 18 June 2008: Another new publication is available: A short paper on Febrl to be presented at the KDD 2008 demo session.

• 28 May 2008: Two new publications are now available, one to be presented at KDD 2008 (August) in Las Vegas, the second as presented at PAKDD 2008 in Osaka, Japan in May.

• 11 November 2007: Finally, Febrl Version 0.4 has been released.

Table of Contents:

• Contact and Mailing List
• Research Team
• Project Description
      Sub-Projects (Blindfolded Record Linkage and Geocoding)
• Publications
• Presentations
• Third Party Publications
• Prototype Software
• Web Resources

Contact and Mailing List:

We have created two mailing lists at Sourceforge.Net. For more details see Sourceforge Febrl Mailing Lists. If you are interested in this project please subscribe and we will inform you of software releases, publications and other output of this project.

Research Team:

• Dr Peter Christen (ANU Department of Computer Science)
• Dr Tim Churches (New South Wales Health Department, Centre for Epidemiology and Research)
• Dr Markus Hegland (ANU Mathematical Sciences Institute)
• Dr Lee Taylor (New South Wales Health Department, Centre for Epidemiology and Research)
• Ms Kim Lim (New South Wales Health Department, Centre for Epidemiology and Research)
• Mr Alan Willmore (New South Wales Health Department, Centre for Epidemiology and Research)
• Mr Karl Goiser (ANU Computer Science PhD student)

Project Description:

Record linkage is an important initial step in many research and data mining projects in the biomedical and other sectors, where it is used to improve data quality and to assemble longitudinal or other data sets which would not otherwise be available.

The ANU Data Mining Group is currently working in collaboration with the Centre for Epidemiology and Research at the NSW Department of Health on the improvement of record linkage techniques and software. We are particularly interested in advancing the development of two aspects of record linkage:

• The use of high performance computing (HPC) techniques in order to make linkage of huge data sets faster (or feasible) when using parallel computing platforms, such as clusters of workstations or personal computers, or supercomputers such as the APAC National Facility (a Compaq high-performance platform with 480 processors).
• The use of machine learning and data mining techniques in order to improve upon the accuracy of the linkage and to reduce the very considerable human effort required to link very large data sets.

We have started developing prototype software which undertakes data standardisation, which is an essential pre-processing phase for most record linkage projects, and which implements the "classical" approach to probabilistic record linkage model as described by Fellegi and Sunter (I. Fellegi and A. Sunter, A theory for record linkage. Journal of the American Statistical Association, 1969) and subsequently extended by others. We hope that this prototype software will be of immediate use to biomedical and other researchers.

We plan to use that software as a platform for exploring various parallel computing and machine learning techniques. To our knowledge, no parallel implementation for probabilistic record linkage is currently available. Issues to be explored include data distribution, blocking techniques, parallel preprocessing and load balancing. Although a number of machine learning and other classification techniques have been applied to the record linkage problem over the last few years, no-one has yet focused on using these techniques to reduce or eliminate the time consuming and tedious manual clerical review process which is needed to decide the status of possible or doubtful links between records.

The prototype software is published under a free, open source software license in order to promote collaboration and to encourage others to contribute to the development and maintenance of the software. The tools we are using are also all free, open source software, namely the object-oriented programming language Python and associated extension libraries.

In order to gain an appreciation of the wide range of uses of record linkage in biomedical research, click here to perform a PubMed search for the term "Medical Record Linkage".

Sub-Projects:

• Blindfolded Record Linkage

  The linkage of records which refer to the same entity in separate data collections is a common requirement in public health and biomedical research. Traditionally, record linkage techniques have required that all the identifying data in which links are sought be revealed to at least one party, often a third party (i.e. the organisation performing the linkage). This necessarily invades personal privacy and requires complete trust in the intentions of that party and their ability to maintain security and confidentiality.

  In this project we aim to develop techniques for blindfolded record linkage, based on secure one-way hash transformations and n-gram (e.g. bigram) scores (which permit the calculation of a general similarity measure between strings), without having to reveal the data being compared, albeit at some cost in computation and data communication. These techniques can be combined with public key cryptography and automatic estimation of linkage model parameters to create an overall system for blindfolded record linkage.

  For more details please see our publications.

• Geocoding

  It is estimated that between 80% and 90% of governmental and business data collections contain address information. Geocoding ­ the process of assigning geographic co-ordinates to addresses ­ is becoming increasingly important in many application areas that involve the analysis and mining of such data. In many cases, address records are captured and/or stored in a free-form or inconsistent manner. This fact complicates the task of robustly matching such addresses to a spatially-annotated reference file.

  The aim of this project is to develop geocoding techniques based on our data cleaning, standardisation and linkage matching techniques. The geocoded reference file used is the Australian Geocoded National Address File (G-NAF), a comprehensive high-quality geocoded national address database.

  For more details please see our publications.

Return to TOP

Prototype Software:

The prototype software Febrl (Freely extensible biomedical record linkage) is hosted on Sourceforge.Net and can be downloaded from:

Information on the current (bold font) and past releases are also available locally:

• Febrl Version 0.4 (released 11 November 2007)
• Febrl Version 0.3 (released 7 April 2005)
• Febrl Version 0.2.2 (released 21 November 2003)
• Febrl Version 0.2.1 (released 26 June 2003)
• Febrl Version 0.2 (released 15 April 2003)
• Febrl Version 0.1 (released 6 September 2002)

Febrl is licensed under the ANU Open Source License, please see the release pages for more details.

Please see the Febrl Sourceforge page for more details.

Back to TOP