Software Development

MPC software engineers specialize in software for the creation and dissemination of population databases. This includes programs for data entry, consistency checking, automated data cleaning and editing, missing data allocation, sampling, disclosure control, database harmonization, metadata creation and parsing, and web-based tools for documentation browsing, extraction, and analysis. The expertise of our developers with techniques for management of large-scale microdata, metadata, and summary files rivals and in some respects exceeds that of the most sophisticated statistical agencies in the world.

In many university-based research environments, programming support is decentralized. Each software project has its own programming staff working in isolation and requiring significant start-up and training resources each time a new project gets off the ground. At MPC, we have chosen a different path. By centralizing programming support for research projects, we encourage our staff to share expertise, and we can leverage innovations developed for one project by applying them across many projects. Our centralized yet flexible model of programming support gives us the freedom to assign individuals from our pool of software developers to particular projects for varying periods. This allows us to maximize the fit between programmer expertise and project needs while maintaining a pool of talented developers.

Our  strategy has allowed us to build powerful platforms that can be leveraged across multiple studies, providing metadata-driven reformatting, missing data allocation, harmonization, metadata creation and parsing, and web-based dissemination at far lower cost than would be required if each project had to start from scratch. For example, the Integrated Health Interview Series (IHIS) data access tools now under development will use the metadata-driven data reformatting and website platform as it exists, saving at least $750,000 in development costs. Without these savings, the IHIS project probably would not have been feasible. This strategy also alleviates the difficulty of ongoing maintenance for our five major microdata websites. Technology is always changing, requiring adaptation to new standards; with a single platform, these changes are much cheaper.

During the past three years, we have streamlined MPC software development by adopting strategies and tools borrowed from the commercial world. These include short development cycles, agile development methods, automated unit tests, trackable work scoping and milestones, and systematic code review. We use open source tools and platforms wherever they do the job without compromise, yet we pay for commercial tools when they offer a compelling advantage over that which is freely and publicly available. Accordingly, we have standardized on Java and GNU C++ as our development languages. We deploy our software in Linux and Solaris environments and use the Apache and Tomcat web servers to make our projects available to the public. We use the MySQL 4.1 relational database engine; the CVS and Subversion source control and version management systems; ANT, make, and CruiseControl for build automation; and RT, a request-tracking and trouble ticket management software. All of these systems are open source and incur no recurring license costs.

We also use the IntelliJ Java development environment, a commercially available tool that has won industry accolades as one of the best and most productive java development environments available. For many internal data development tasks, we use Microsoft Excel, Access, Visual Basic, and SQL Server. These tools allow us to build and deploy software systems for internal use very quickly, where time-to-completion is more important than scalability. For these tasks our non-open source tools provide significant benefit over the open source tools that are currently available. As new tools and technologies become available, we periodically re-evaluate our choices, and change when better alternatives become available.