Life as a Physicist

Congratulations to everyone at D0 and the Fermilab accelerator division — D0 has been sent 4 fb-1 as of April 30!! That is a lot of data! And years of work!

It will be a while before you see that in an analysis however. First of all, of the 4 fb-1, only about 3.5 fb-1 were written to tape. The rest is lost forever. Where did it go? Well, perhaps our detector was broken for some short amount of time. We do our best to make sure that doesn’t happen of course, but a machine like this does break (my name is no more than a few of those minutes for Level 3/DAQ problems!) We generally run our triggers with a small fraction of dead-time — time where we aren’t accepting new events even if they are coming in - if we were to run with zero dead-time we’d not be able to do nearly the physics program we do [if you want more details, let me know].

The second issue is time. It takes time to understand our data - a while to do something as sophisticated as a Higgs search. Parts of the detector are turned off, which affect efficiencies and systematic errors. The events we trigger on are changed as we try to optimize our data for the higher instantaneous luminosity the Tevatron delivers. On top of that, of course, is the continuous effort to improve our analysis technique’s power. All these changes must be carefully studied to see how they impact each analysis. And once that is done, only then can the data be shown externally. This takes quite some time. The more sensitive the analysis, the more carefully the data must be studied.

So, fantastic for us at D0 (and CDF) for reaching 4 fb-1 delivered. Thanks to the Fermilab accelerator division for doing this despite the trying times. Everyone else: sorry, you’ll have to hold your horses a short while before we show the results of this data!

Amazon has done a lot of work to make GRID computing services accessible to anyone that wants it. Actually, it surprised me that Google or Microsoft didn’t do it first — to run their search engines and other similar things they must have farm computing down to a tee.

In HEP we spend a huge amount of money and cost and time with the GRID. A discussion in a bar some time back generated the question: what would it cost to move HEP into the cloud?

Databases

Yesterday I mentioned databases for storing event data. Amazon has SimpleDB (see this posting to get an idea of how it works). On the surface it looks rather poorly suited to do what we would want to do with our highly structured data. But, ignoring that and some of the overhead it will charge - for the 100 GB of data that Rich had in his database it would cost about 150 bucks a month to store it. Querying is dirt cheap — 14 cents per hour of CPU time used. I have no idea what the performance would be on a database like this, but even if it were x10 slower I doubt it would matter much.

ATLAS’ equivalent database to Rich’s project is thought to be 14 TB/year. That works out to be $21,500/month.

Event Data

Amazon has a simple storage service as well (Amazon S3). Because the data is just a binary blob the cost of storage is much cheaper: 15 cents per GB per month. However, trying to figure out what size ATLAS will actually use if it stored everything in the cloud, and ignored the actual design, is difficult. Making some rough estimates from an old version of the computing model, I’m going to guess about 10 PB per year (that is petabyte!). That is about 1.6 million bucks per month. But we aren’t done with this yet, however - it costs money to move the data in and out. First, just to load the data it will cost about 1 million.

Then we have to use the data - lets say each year we cycle through all the data once — so all 10 PB. That will run about 2.5 million per year (not per month!). But if we use Amazon’s EC2 compute cloud, moving data to it and back is free. In that case, only final datasets will probably be moved. That would be much cheaper.

Computing

This is even harder for me to calculate. This matches up with Amazon’s EC2. One cool thing is data between these computers and S3 is free. Otherwise, for a 32 bit single processor machine that has enough memory to run ATLAS software it looks like it costs about 10 cents per hour of use. Now, in ATLAS an estimate in 2005 was it would take about 3000 kSI2k to reconstruct the average event. So, for an Amazon machine (that is about 1.9 kSI2k) that would take about 26 minutes. So, about 5 cents per event to reconstruct the event. If we expect 2,000,000,000 events per year, then that will cost us $100 million dollars to reconstruct. If someone is familiar with SpecINT2000 and how it works, perhaps they can verify I did this math “ok”. And I’ve not included analysis time which is probably x2 more.

So, there you have it. A lot of money would go into running this in the cloud. Of course, we could never walk up to someone like Amazon and dump this on them. In almost all cases we will do better on our own as we can optimize what we are doing for our uses. Further, the cash that gets spent on this is from all over, and in all different colors. Many nations, for example, buy GRID installations for all scientists in their country. ATLAS just piggybacks on these purchases and uses a portion of them. Still, interesting to see what the cost would be - about 120 million before you even start to analyze the data to produce a physics result!

WARNING: this is very much a back-of-the-envelope calculation!!

Not everyone is satisfied with ROOT as the “tool” to analyze HEP data. Back in D0’s Run I all the data was loaded into a commercial database.

So, before you roll your eyes - you are right. HEP is littered with database train wrecks (can anyone say Objectivity?). However, most of those had to do with trying to store every single last bit of data that came off the data acquisition system in the database. And then also store reconstructed data. And then, in some cases, even the analysis level objects. In fact, ROOT grew out disagreement with this vision (and you can tell who won…).

This project, however, was different. The goal was to store only the high level physics information. For a reconstructed jet, for example, they had the four vector and some other quantities (like electromagnetic fraction of calorimeter energies - 28 values in all). They had separate markers for tight very high quality electrons and loose, lower quality, electrons. Same for muons, jets, etc. To understand the limitations of this — and what you might or might not do with this tool: if you changed your jet energy scale you would have to completely re-load the database. This is not something you do frequently, but you get the idea: this is to do your final selection - the last mile of your analysis. Indeed, the test case was to repeat the Run 1 top discovery analysis. However, if you can do selection quickly imagine the power for scanning over a large SUSY parameter space!

How much data? About 62 million events. As a raw ntuple it was 62.4 GB of ntuples (small by today’s standards, of course!). It took almost 1000 hours to generate these ntuples - applying jet energy scale, etc. After being inserted into the database it was 80 GB of raw data, and another 30 GB of database index data.

They used Microsoft’s SQL Server for this. On a qual 450 MHz Pentium II with 256 MB of memory. Does that tell you how long ago this experiment was done!?

Actually, their DB design was pretty clever. All electrons in one table, all jets in another. Then another table which just listed all tight electrons, and another one that listed all loose electrons, etc.

So, how fast did this thing run? So, looking for a Z boson goes to two electrons took about 7 seconds. It found about 6000 events - the right number. Looking for a W boson decaying to an electron and neutrino took about 18 seconds to find 86,000 events. That is pretty darn good!

Are there plans to do this in ATLAS? Well, perhaps. We have a physics summary database - but it isn’t complete (e.g. doesn’t have all the jets in an event). It its design goal is different: you use it to select a sample of events you actually want to run over.

The project was lead by Rich Partridge at Brown University (with a lot of help from an undergraduate Matt Bowen). For more raw information you can see a talk by Rich at a SLAC meeting the other day (CERN ATLAS agendas, look for meetings on Feb 27, the SLAC ATLAS forum).

At any rate, this was something I’ve been meaning to write about for a while. Unfortunately for an approach like this, about 95% of an analyzer’s time is spent trying to understand what exactly is a tight electron - and its fake rate. However, anything that makes for fast turn around is a boon in my book!

I’m impressed. I’m now sitting in Terminal 4 of Heathrow waiting for my short flight to Paris. And I’ve zipped through this airport — no delays moving about. This is better than the stories I heard from some people arriving last week at Glasgow who’d gone through Heathrow: waits of up to 2 hours to change terminals. Sweet!

Once a year the D0 experiment has a week long workshop away from Fermilab. It is organized around one or two themes. We’ve been all over the place — France, for example. This time, we are in East Lansing. It is hot hot hot! But it was just announced that next year’s workshop will be in Prague. That should be a lot of fun!

Saw this headline on the airplane: “USA might lose good citizens due to immigration process”. Duh. I know quite a few people going through various stages of the immigration process. Most of them are trying to get green cards as possibly a first step towards citizenship. All of them are physicists. They tell stories. I’m not sure I would be willing to put up with what they put up with. The long lines. Getting called on Tuesday and told they need to be 200 miles away to get finger prints the next day — canceling classes or missing presentations they are supposed to do. Apparently the penalty is you go the back of the line when that happens - lines that are often more than a year long. And check this out: there is a certain spot in the process when you can’t leave the country — you are stuck here. If it comes over Christmas or similar important family holiday and you want to go home and visit your family? Tough cookies: better travel under an assumed names.

Buffer overruns are a particularly difficult source of bugs, and in the right places, security holes in software. Wikipedia has a marbled-mouthed description of it:

A buffer overflow is an anomalous condition where a process attempts to store data beyond the boundaries of a fixed-length buffer. The result is that the extra data overwrites adjacent memory locations. The overwritten data may include other buffers, variables and program flow data.

It has some great pictures of what it is talking about, however. Basically, a program stores something where it shouldn’t.

It sounds like the Mars Global Surveyor was a victim of something like this:

…an errant computer command five months earlier had been placed in the wrong location of the computer memory for the spacecraft. That, in effect, implanted a fatal defect in the spacecraft, disabling a safety feature to prevent the solar panels from rotating too far and mangling its ability to communicate with Earth in case of a mishap.

In short — no error checking caught that errant computer command. What a pity. It is very difficult to catch these sorts of programing errors (I see them with some regularity in our experiment’s code base).

The end of the new york times article describes what happened in the last minutes of the Surveyor’s life. The controllers watching from Earth must have been going nuts, helpless and unable to communicate:

In its last 13-minute contact, the Global Surveyor reported numerous alarms to mission controllers but gave no indication that it was in immediate danger.

As the spacecraft tried to recover, it ended up in an orientation such that the Sun was shining directly on a battery, causing it to overheat. The Global Surveyor misinterpreted that signal, sensing that it had overcharged the battery and stopped charging its other battery, as well.

Meanwhile, because the June error caused the craft’s antenna to point in the wrong direction, mission controllers on Earth could not get in touch with the craft again.

I’m going to ask Toby, who is part of the GLAST team, if they have done these sorts of checks. Or if this discovery means a delay while they check over the GLAST software.

A student of mine, Jenne, won a Goldwater scholarship (that is her in the middle, in green). She was a introductory honors student of mine a few years ago. She is one of 317 people to win this award from across all of the USA. The winners get tuition, fees, books, room and board. Sweet! Congratulations!

On a break — like this spring break right now — I often try to do some small “fun” project. Over Christmas I put together a small application for my mobile phone which downloads images from Flickr and caches them (I’ve got a source forge setup for it, but I’ve not actually put the installer up there yet). One thing I’d been curious about doing was having some program that changed my desktop background. Should be easy to write? Thank goodness I searched Flickr for one first: there are several. This one I like a lot. It has a simple interface, you can specify an account, tags, etc., and it will download the pictures one after the other and swap out your background. When I feel I need to procrastinate (often) I just hid all my windows and check out what is on my desktop top. We all need a few more pictures in our lives!

P.S. This is a picture I found on my desktop by closing all my windows…

I’m fascinated by clocks. I think they were the only things I built in my high school wood shop class. I periodically check out the ebay clock listings (though I’ve never purchased anything there). Nick pointed out a very cool clock. This thing moves water around to tell the time — it has 12 tubes, one of reach hour. Would that be neat or what? I need a clock for my office… Of course, this baby will set you back CHF 15,000 (12,263 US bucks).