ANU Data Mining Group

Back to main page

Resource Categories

• Australian privacy and confidentiality resources
• International privacy and confidentiality resources
• Vendors of business-oriented record linkage, merge/purge or list de-duplication software
• Record linkage software of potential interest to biomedical researchers
• Australian health and biomedical research organisations which are known to use record linkage techniques
• Australian resources relevant to record linkage
• International resources relevant to record linkage
• Publications, journals and conferences relevant to record linkage
• Resources for name and address processing
• Python and friends (software and programming resources)
• (One-to-one) Assignment resources
• Geocoding resources
• Cryptography and security resources

Australian Privacy and Confidentiality Resources:

• Office of the Federal Privacy Commissioner - Guidelines on privacy in the health and medical research sectors.
• National Health and Medical Research Council - Health and human research ethics.
• New South Wales Department of Health Information Privacy
• Privacy NSW (Office of the New South Wales Privacy Commissioner)
• ANU Human Ethics
• National Statement on Ethical Conduct in Research Involving Humans
• Legislation relevant to privacy and confidentiality of health and biomedical information in NSW:
  • NSW Freedom of Information Act 1989 - Applies to public sector agencies only.
  • NSW Privacy and Personal Information Protection Act 1998 - Applies to public sector agencies only.
  • Commonwealth Privacy Act 1988 - In particular see Sections 95 and 95A.
  • NSW Health Records and Information Privacy Act 2002
• AAR Privacy in health
• Data Matching (Australian Government Privacy Commissioner)
• ARC Research Network for a Secure Australia (RNSA)

International Privacy and Confidentiality Resources:

• Confidentiality Issues for Medical Data Miners
• UNECE - Work session on statistical data confidentiality 1999
• UNECE - Work session on statistical data confidentiality 2001 (Skopje, 14-16 March 2001)
• Microdata Confidentiality References (W.E.Winkler, Feb. 16, 2002)
• Try searching the WorldLII legislation database.
• Journal of Privacy Technology
• Privacy in Statistical Databases (PSD Conference, July, Barcelona)
• Data Privacy Lab (Latanya Sweeney, Carnegie Mellon University)
• IEEE Security and Privacy

Non-exhaustive List of Vendors of Business-oriented Record Linkage, Merge/Purge or List Deduplication Software:

• Intech Solutions - Highly integrated software solutions to address and resolve information quality issues.
• Ascential Software (incorporating Vality) - Their SuperStan and SuperMatch products and Integrity suite supersede the AutoStan and AutoMatch products formerly sold by MatchWare Technologies.
• Trillium Software - Another large international vendor of data cleaning and matching software products.
• SAS Data Quality Solutions incorporating the DataFlux product.
• Database Consultants Australia - More details on their Nirvana product are available here.
• Datatools - See their Twins III product.
• Direct Marketing Software - Their LISTman product does de-duplication and list washing
• Hopewiser - Their ATLAS product does list de-duplication.
• Language Analysis Systems Inc. Practical applications of computational linguistics expertise.
• LinkageWiz - Probabilistic record linkage software for Windows computers.
• Capscan - Leading supplier of UK domestic and international address software.
• MSI (formerly MasterSoft International) - See their NADIS product suite.
• MarketSoft - Another Australian vendor of list processing software.
• SearchSoftwareAmerica - Vendors of name searching and matching software.
• Other vendors of address processing software licensed under Australia Post's AMAS program.
• Test&Go Quality Control - Control of data from whatever source and in whatever format.
• DataSet V - Personal Information Mining & Refining Suite.
• MATRIX Solutions - Matching software for the mortgage servicing industry.
• E-Scrub - Cleanse Your Data.
• Semaphore Corporation - Free Internet Resources.
• Fuzzy Informatik - German company supplying fuzzy software for data quality.
• WinPure - Data cleaning tool for Windows.
• Accumatch - Real Estate Data Matching Service.
• ChoiceMaker Technologies - Commercial record linkage software which uses a maximum entropy approach.
• The Link King - Record Linkage and Unduplication Software
• Arizcon (NamiX fuzzy name searching)
• Management Information Principles
• Intellexer (EffectiveSoft)

Non-exhaustive List of other Record Linkage Software of Potential Interest to Biomedical Researchers:

• Master Person Index Solutions
• Record Linkage Software, Utilities, and Other Fun Stuff (Rich Pinder, Los Angeles County Cancer Surveillance Program).
• Link_It - Utility for record linkage developed by the Ukrainian National Cancer Registry, Kyiv.
• LinkSolve - Probabilistic record linkage software for Windows computers, used in the Utah Crash Outcome Data Evaluation System (CODES).
• RLESearch A free, open source probabilistic record linking engine written in the Perl programming language.
• Record linkage using error prone strings - A project to develop a free, open source (?deterministic?) record linkage system suitable for use with German names and addresses, written in the Perl programming language.
• GRLS (Generalised Record Linkage System) - A commercial probabilistic record linking system developed and sold by Statistics Canada (e-mail Evelyn Perkins or Martha Fair for details).
• LinkPro - A (?commercial?) suite of SAS System macros for probabilistic record linkage (e-mail Andre Wajda for details).
• Software for Automated Linkage in Italy (SALI) - Free-of-charge (?open source?) deterministic linkage software suited to Italian data, written in the Clipper programming language. Available for scientific purposes from Luigino Dal Maso, Servizio di Epidemiologia, Centro di Rifermento Oncologico, Via Pedemontana Occ., 33081 Aviano (PN), Italy. E-mail: epidemiology@cro.it
• TAILOR: A Record Linkage Toolbox - An environment for testing machine learning approaches to record linkage. Mohamed G. Elfeky, Ahmed K. Elmagarmid and Vassilios S. Verykios. More information is available here.
• AddressParse 1.05 - A Perl module (part of CPAN distribution) for address parsing.
• Health Information Solutions
• LinkPlus from CDC
• Link King

Non-exhaustive List of Australian Health and Biomedical Research Organisations which are known to use Record Linkage Techniques:

• Data Linkage Unit, School of Population Health, Department of Public Health, University of Western Australia.
• Centre for Epidemiology and Research (formerly Epidemiology and Surveillance Branch), Public Health Division, New South Wales Department of Health - See this paper for an example of record linkage used in population health epidemiology.
• Road Injury Information Program - Research Centre for Injury Studies, Flinders University, Adelaide, South Australia
• New South Wales Central Cancer Registry - NSW Cancer Council, Woolloomooloo, Sydney.
• New South Wales Pap Test Register - NSW Cancer Council, Woolloomooloo, Sydney.
• Australian Childhood Immunisation Register (ACIR) - Health Insurance Commission, Canberra.

Australian Resources relevant to Record Linkage:

• National Library of Australia - Various Australian Indexes and Databases.
• Australian Baby Names Database
• Australia on Disk CD-ROM - Electronic White and yellow pages (telephone directories).
• Australian Gazetteer (Geoscience Australia)
• Australia Post Postcodes and Localities
• IP Australia - Patents, Trade marks, Designs.
• Web Service and Data Integration (CSIRO)
• National Data Network
• E-Health research centre Health Data Integration
• National Collaborative Research Infrastructure Strategy (NCRIS)

International Resources relevant to Record Linkage:

• RIDDLE: Repository of Information on Duplicate Detection, Record Linkage, and Identity Uncertainty
• Purdue University Data Quality Project
• WHIRL System - Word-based Information Representation Language.
• SecondString - Open-source Java-based package of approximate string-matching technique.
• Ukrainian Cancer Registry
• CODES 2000 User Forum
• Wisconsin Crash Outcome Data Evaluation System (CODES)
• Record Linkage at Statistics Canada
• K-Base - A knowledge base on statistical data editing.
• Flamingo: Data Cleansing Project, Database Group, UC Irvine.
• IntelliClean - School of Computing, National University of Singapore.
• Data Cleaning and Information Quality - Many links to data cleaning and record linkage resources.
• Data Cleaning and Integration links - Another set of links to commercial data cleaning, data quality and record linkage tools.
• De-Mystifying De-Duplication
• US Patent and Trademark Office - Web Patent Databases.
• Access to Archival Databases (AAD) System
• US Department of Agriculture: Report on Feasibility and Accuracy of Record Linkage To Estimate Multiple Program Participation
• IPUMS (Integrated Public Use Microdata Series)
• MATRIX (Multistate Anti-TeRrorism Information EXchange)
• Open Source Health Records Exchange
• QS Technologies - Deduplication Tools
• List of hard to spell words
• DaQuinCIS (Data Quality in Coorporative Information Systems)
• Information Integration Research Group (University of Southern California).
• QS Technologies Deduplication Comparison Results (LinkPlus and Febrl).
• Record Linkage (Wikipedia)
• Spock Challenge
• NationMaster (massive central data source to graphically compare nations)

Publications, Journals and Conferences relevant to Record Linkage:

• First Workshop on Data Cleaning, Record Linkage, and Object Consolidation - Proceedings.
• International Workshop on Data Quality in Cooperative Information Systems - Italy, January 2003.
• First International Workshop on Data Cleaning and Preprocessing - 9 December 2002.
• Data Quality Workshop, National Institute of Statistical Sciences (NISS), USA, December 2000.
• US Census Bureau Research Reports
• US Census Bureau Record Linkage Workshop
• Proceedings Record Linkage Workshop 1997
• Proceedings Exact Matching Workshop 1985
• Health Informatics Journals
• Statistics in Medicine
• BioMed Central
• Elsevier Journal Artificial Intelligence in Medicine, October 2002 (Data Mining)
• List of data warehousing / integration papers
• Workshop on 'Using Administrative Databases for Health Research: Challenges and Opportunities" (University of Sydney, 25 June 2004)
• IQIS 2005 Workshop (Second International ACM SIGMOD Workshop on Information Quality in Information Systems)
• Indrajit Bhattacharya bibliography of record linkage papers
• Michael D. Larsen (incl. publications and proceedings)

Resources for Name and Address Processing:

• Moby lexicon project
• Links to Postcode Pages
• US Postal Service Official Abbreviations
• US Census Bureau 1990 Name Files
• Behind the Name Etymology and history of first names.
• Family Search
• Cal Poly Pomona Name Pronunciation Guide
• International Phonetic Association
• 2003 Australasian Language Technology Workshop
• Behind the name (the etymology and history of first names)
• Word List Page (Sourceforge.net)
• GNU Aspell
• Implement Phonetic ("Sounds-like") Name Searches with Double Metaphone (on The Code Project)
• Fuzzy Matching and Deduplication (Sourceforge.net)
• US Census 1990 name files

Python and Friends (Software, Programming and Open Source Resources):

• Python - A free, open source, interpreted, interactive, object-oriented programming language.
• Python Spread Module - Python interface to a high performance messaging queuing service for clusters of workstations.
• PyPar - Python bindings to a small but important subset of the Message Passing Interface (MPI) standard - See MPI).
• PyRO - Python Remote Objects.
• Logilab's Python Hidden Markov Model
• GHMM (General Hidden Markov Model library)
• Olivier Cappe - 10 years of Hidden Markov Model publications.
• Software Tools for Natural Language Processing (NLP)
• Object Craft - Fast CSV Python module.
• Lawrence Philips' Metaphone Algorithm
• Pure Python strptime()
• ProbStat - Probability And Statistics Utils for Python
• Python Spelling Construction Set
• PyProximity - A set of modules to provide text proximity search.
• SpamBayes - Bayesian anti-spam classifier written in Python.
• A Logging System for Python
• SciPy - Library of scientific tools for Python.
• Spirit Project - Open Source Software for better health care.
• EGovOS.org - The Center of Open Source and Government.
• Python City - Home Free for Pythonistas.
• Python Bindings for BerkeleyDB 3.1 thru 4.1
• Python Expectation Maximisation implementation
• Dive into Python: Parsing Phone Numbers
• PyTables
• PyML - A Python machine learning package
• NetEpi - Free, open source, network-enabled tools for epidemiology and public health practice
• R - Project for Statistical Computing
• RPy - R from Python
• CompLearn - Compression-based similarity measures.
• DataCompression.info
• Compression algorithms in Python
• Python Toys
• Open source clustering software
• Source Code for Biology and Medicine (BioMed central journal)
• The Python Papers
• IEEE Computing in Science and Engineering (CiSE) on Python: Batteries Included
• PyGTK: GTK+ for Python

(One-to-one) Assignment Resources:

• Linear Assignment Problem (software downloads)
• Assignment problem (a shortest augmenting path algorithm)
• Network flow (operation research notes)
• Dimitri Bertsekas (papers, reports, etc)
• pyLAPJV - Python module for the Jonker-Volgenant algorithm, LAPJV

Geocoding Resources:

• Geocoded National Address File (PSMA)
• Land and Property Information New South Wales
• The Public Health Disparities Geocoding Project Monograph (Harvard School of Public Health)
• Geocode.com
• Geocoding Addresses (Virtual Terrain Project)
• Open GIS Consortium
• Geocoder.US (free downloadable US data and code)
• Map search Switzerland
• GEOnet Names Server (GNS)
• Geographic Names Information System (GNIS)

Cryptography and security resources:

• Cryptography pointers
• Privacy, Security and Data Mining site
• Stanford Portia project
• Privacy and databases (Rajeev Motwani)
• Data Mining and Privacy Group (University of Thessaly, Greece)

Back to main page