The high throughput computing ability of a software-system known as HTCondor was critical to the discovery of Einstein’s gravitational waves at LIGO. The one-of-a-kind software is the result of a collaboration between two University of Wisconsin teams that started more than a decade ago.
Albert Einstein said we would never see the gravitational waves he predicted 100 years ago, but little did he know that software like the HTCondor would be developed to help prove him wrong. Since 2004, HTCondor has been at the core of data analysis efforts at the Laser Interferometer Gravitational-Wave Observatory (LIGO). Combined with LIGO, the software has the unique ability to analyze massive streams of data live, and distinguish gravitational information from noise, essentially picking a needle out of a haystack of data. [What Are Gravitational Waves and Why Are They Important?]
The HTCondor team is led by Miron Livny, a University of Wisconsin-Madison professor of computer sciences, and chief technology officer for the Morgridge Institute for Research and the Wisconsin Institute for Discovery. More than a decade ago, a collaboration between his team and LIGO scientists at the University of Wisconsin-Milwaukee just up the road transformed the way LIGO computed data. Ever since then HTCondor has been continuously evolving and getting smarter.
HTCondor takes seriously the core challenges of combing through mountains of data, helping scientists distinguish the good from the bad. The software has been fine tuned over the last decade to block out noise such as wind, temperature, light and seismic activity, all while distinguishing gravitational information with unparalleled precision.
“In the absence of noise, this would have been a very easy search,” said Peter Couvares, a veteran scientist with over 10 years experience working with HTCondor at LIGO. “But the trick is in picking a needle out of a haystack of noise. The biggest trick of all in the data analysis at LIGO is to come up with a better signal-to-noise ratio.”
Another challenge is managing the thousands of individual computers connected to the network from academic institutions around the world. The software has a functionality baked into it to safe guard against local failures. This is an example of how the system has evolved; previously, local failures could have caused the whole experiment to shut down.
Scientists are now extremely excited and confident about the future of the HTCondor/ LIGO collaboration, as they appear to be just hitting their stride. It’s likely that with the initial observation of gravitational waves, a potential tsunami of data awaits from the universe.
“The field of gravitational wave astronomy has just begun. This was a physics and engineering experiment. Now it’s astronomy, where we’re seeing things. For 20 years, LIGO was trying to find a needle in a haystack. Now we’re going to build a needle detection factory,” said Couvares.