Life as a Physicist

A very technical post on the performance of TFractionFitter.

My post the other day about TFractionFitter being written up in a arXiv paper generated a few comments - one in particular that got me a little worried:

Looking at the original Barlow and Beeston paper we found that the approximation breaks down in the case of weighted events. If you want to have some fun, change the initial normalization of the templates that you pass to TFF.

As I mentioned, we used this fit method in some of our current work. For my Friday night entertainment I decided to see if I could test it out a bit.

Constructing a few test cases, and then running them 1000’s of times (ensemble tests) isn’t all that hard (well, see below). Recall TFractionFitter takes several Monte Carlo templates and tries to find components of them in a data histogram. I constructed the templates out of two Gaussians for this test. The data histogram I built was 30% of one of the two Gaussians, and 70% of the other.

In a simple test where I create the data histogram once, with 10,000 entries, and then re-create the template histograms over and over (about 150 times) and perform the fit each time, I found that the fitter got the fractions correct - it correctly identified the 30% and the 70%. Further, as is mentioned in the paper, if I looked at how much it got the fraction incorrect each time, it does indeed look like the errors are underestimated (the exact amount depends on how I arrange the two Gaussians).

To test the weighting as described by Mike, I altered each template’s normalization to be 50% of its initial value, but left the data histogram the same. I got the same result as before. I tried altering the number of entries in one or the other and still the same. So, thankfully, I can’t reproduce this (I was using 5.18 to do this testing).

One thing I discovered — my initial choice of Gaussians had the second Gaussian about 20% off the end of the histogram I was using. The result? TFractionFitter got the answer wrong consistently by about 5%. 5% isn’t that big a deal if you get your errors right — but it didn’t. It was almost always more than one sigma low or high (depending on which fraction you looked at) - a clear bias. The distributions we are using in our work look more like falling exponential - so I’ll have to test that out and see if that shows the same thing. This is another buyer-be-ware comment similar to Mike’s.

I used this weekend project as an excuse to put my money where my mouth is… sort of (meaning I spent time, not money!). I’ve long talked about, on this blog and other places, that if we are to really take advantage of multi-core processors we are going to have to change the way we write code. So for this little project I thought was a perfect excuse to use a new programming language — a functional programming language. I used F#, which is based on ML (OCaml is the current standard implementation of ML). If anyone is interested, I can write some (also technical like this post) posts on what it was like. I used F# instead of OCaml, in part because it has access to all the ROOT libraries due to some other work I’ve done.

I was trying to ignore the TV show Big Bang Theory. Mainly because it seemed like a bunch of stereotypes, and perhaps ones I don’t think are right. However, after reading this column by Ouellette in Symmetry magazine, I discover that an old friend of mine, David Saltzberg, is doing the shows science consulting.

People’s opinions of the show weren’t very positive the first time I mentioned it on my blog. Have they changed? I’ll post back when I’ve gotten around to watching it. From Ouellette’s column, it sounds like one has to watch more than just the first one or two episodes (i.e. while I might object to the science stereotypes, women might have complained about the female stereo types and that has been better developed over time).

Persis Drell (SLAC director), along with Steven Chu (LBNL director), and Thom Mason (Oak Ridge Director) had a chance to visit the house speaker the other week.

We explicitly told the Speaker that we three Directors of major research institutions were not asking for anything for our individual laboratories. Our message was that a broad national commitment to scientific research and innovation was necessary—and that this would certainly create an environment in which we, and our partners in the academic community, could compete and thrive.

Exactly. I couldn’t agree more.

I personally doubt there is much more than can be done for this year’s budget mess (but I’m in favor of trying!!), but perhaps we can convince congress to be ready for the ‘09 budget - which probably won’t be settled until well after the election.

An arXiv paper went by the other day that I found interesting for several reasons: “A pitfall in the use of extended likelihood for fitting fractions of pure samples in mixed samples”.

The paper discusses a technique of fitting templates of various sources to data in order to extract constituent fractions. For example, lets say you have a series of particle jets. The jets come from light quarks (up, down, strange), charm quarks, or bottom quarks. You want to know the fraction of each of those types that makes up your series of jets. You are clever - you have a variable that behaves differently for each of the quark types. So, now you can make a template for each jet type in this variable, add them together, vary the fractional size of each template until it fits the variable in your data - and “boom” you have the fractional makeup of your series of jets.

There are a number of tools around that will help you do this so you don’t have to write them yourself. In particular, in ROOT, there is TFractionFitter. This paper calls out an error in TFractionFitter by name. This is the first reason I find this article interesting. This is appearing in a physics paper archive and it specifically targets a tool. It isn’t about physics in particular, but the improper application of a statistical technique. I’m fairly sure (though not 100% positive) that no one has written a paper about TFractionFitter’s operation. Sometimes you’ll see papers written to point out errors in other papers, and then rebuttals. But this is a paper to point out an error in C++ code - ROOT. And, ha, it doesn’t even mention a version number for ROOT - which may be a problem if the error is corrected!

The error is subtle. TFractionFitter will get the fit correct. So the raw numbers that come out will be correct. The probably is when it tries to calculate the errors. Apparently the method it uses to calculate the errors are not valid unless one evaluates things at the central value — which ruins the point because the whole idea of an error is to understand how much things change as you move away from the central value.

Which brings me to the second reason I am interested in this article. A group I am part of in ATLAS is using TFractionFitter! Ops!

As I write this the exchange rate is about $1.54 US to one euro. This will probably come out the middle of next week - no telling what it will be then.

When I moved over here on July 1 the dollar was $1.35 US (see Yahoo! Finance).

Even though France is helping out by paying me a per-diem over here, it is paid in constant dollars. Darn.

Our graduate students and post-docs over here are in the same boat. They are also paid in constant dollars. It is worse for them, of course, as they are paid less than I am and so have less of a cushion to fall back on (I don’t even want to talk about what has happened to my savings account during the stay over here!).

Indeed, we just had to raise the student’s battle pay in order to account for the exchange rate difference. Where does this money come from? Our grant, of course. To supplement the grant so that it can accommodate these increases we are asking for extra cash, of course, but everyone doing physics over in Europe is in the same boat and there just isn’t much extra money in the various funding agencies coffers after the last budget battle.

I don’t see the dollar coming down soon. I keep hoping it will plateau. This latest turn-around is, I think, only temporary. But it is definitely cutting into our ability to send people over to Europe, and we do science with people - less people, less science. It started with a small fraction of bad loans, which is slowly claiming more and more people - and now spreading out from the housing sector to the financial sector and… to the science sector. If there is such a sector!

BTW - when the insurance adjuster dropped by we talked a bit about the financial crisis in the US. He said it was a scandal: “Why haven’t you guys caught the people responsible?”

One one of my lasts posts about computers and HEP, Kevin left a comment.

OK, this post forces me to ask this question: can you estimate the amount of time you spend on “science” questions versus methods development, programming or other things? After reading your blog (and enjoying it of course) for over two years now, I feel you have discussed a lot of issues in programming, ROOT, C++ vs other languages, computers and other things that are tools to do science but which are not themselves science (or at least not physics).

I am fundamentally lazy. Other bloggers — like Tomasso — are not. I’m afraid I do not spend much time on these blog posting (spelling errors!? Me!? Never!). As a result, I tend to write about things I can clack off without having to do any reading or extra research. This usually boils down to physics I’m working on or support work I’m working on, hobbies, or my social/family life. I don’t understand how Tomasso has the time to read and digest all those papers. I suppose once you get going you can do it more and more and more quickly, but I have not been willing to take the time.

The physics, however, suffers. For example, I’m working in the Higgs group at D0 - well, mostly my post-doc and my graduate student are this year - but I can’t really talk about some of the very cool stuff that is happening there. That is a sure way to get hit on the head by the collaboration. In ATLAS I’m also doing some stuff I would love to talk about - but ATLAS has an official blogging policy (i.e. don’t, unless the result or thing is public).

Now, part of the reason I got into HEP is I like that it lives at the corners of physics, software, and hardware. Since D0 has matured, I’ve not been doing very much hardware. The software side of things I’d always done as a hobby. I have always been a big fan of software and have made some fairly flexible and sophisticated frameworks for use in physics analysis. Since that, and watching people who are not me try to understand and read my code, I’ve come to believe C++ just isn’t all that great as a productive programing tool. Now here is the thing: no one cares about software! The byproduct of that is I can talk about software freely on this blog, as I work on it. Take ROOT for example, I recall trying to get a new plot out for single top. With that plot we were able to understand that the way we were doing our b-tagging in our background sample was correct. But in order to make that plot I had to fight through a ROOT bug. Guess what I can write about in the blog?

So, now a direct answer to your question. I’m on sabbatical this year. I would say it was about 50-50 for me. When I’m teaching a course I understand, it is probably 40% physics, 30% software, and 30% teaching. When I’m teaching a class I’ve never taught before it will probably by 10% physics, 10% software, and 80% teaching (ask me in November).

Thanks for the question/comment Kevin. I hope I’ve answered it!

One one of my lasts posts about computers and HEP, Kevin left a comment.

It would appear to me that the LHC has been for many years “methods development,” yet I’m assuming a couple people already have tenure on it and more than a couple Ph Ds have been awarded for developing the technology.

As far as I know, in the USA, no institutions will give a Ph.D. for an experimentalist if they have not touched data. As far as I know, no one has managed to get an experimental LHC Ph.D. in the USA by just running Monte Carlo or working on a bit of the detector. Now, parts of the LHC have taken data — i.e. the test beam.

The point of the test beam is exactly as it sounds - we put portions of the detector in the test beam to test them out. We fire known particles at known energies into bits of the detector and make sure they react (and readout) as we expect them. If they don’t, we adjust the physics models we use to simulate them or perhaps find something wrong with the detectors and fix them. It is not common to get a Ph.D. in the USA on test beam data, but it has happened. For example, D0’s initial startup (Run I) was so delayed I think a few people did this and then remained on D0 as post-docs to get their hands on real data.

Europe is different - there you can get a Ph.D. on Monte Carlo studies or on building a detector. As far as I know, it isn’t viewed as any different than getting a Ph.D. on data.

But, if you are in the USA, what do you do? This is exactly why most HEP groups maintain a foot in more than one pie. For example, I do a lot of work on ATLAS now - but I also do a lot of work on D0. D0 is a running experiment and produces real results. My tenure decision was on D0. I could have started on ATLAS when I arrived at UW 8 years ago - there was plenty of work to do - but it was correctly seen as suicide. Instead I worked on D0. I only just now have graduate students working on the LHC. I bet if you looked at the number of US graduate students on the LHC it was rather small and is now rapidly increasing. And that is because we are finally in the time frame that these students can get a Ph.D. on LHC data.

Finally, I have heard of programs that offer Ph.D. in detector physics and accelerator physics - which is very different from the work I do. I know less about them than I should, however.

However, Kevin correctly points out, once you are past the tenure bar you can just do what you want. Want to put all your marbles in the LHC basket? Go for it - no problem! Directly addressing the implied question in Kevin’s comment - presumably the person on the LHC who is making these criticisms had to go through the tenure process. And hopefully they are applying the same standards that were applied to them. Sometimes it is hard - I went through the qualifying exam as a student. Hated it, and it wasn’t clear that it offered any net gain for me or my fellow students. I passed, and now, about 15 years later, I sometimes catch myself thinking “it wasn’t that bad…” Some people carry that to an extreme. In the tenure case this is exactly why it is necessary to consult with other people in the department to understand if this is something unique criticism held by one person or is generally shared criticism.

Final installment of this 3-part series next.

UPDATE: Changed the tone of MC physics paragraph above.

One one of my lasts posts about computers and HEP, Kevin left a comment.

After reading your blog (and enjoying it of course) for over two years now, I feel you have discussed a lot of issues in programming, ROOT, C++ vs other languages, computers and other things that are tools to do science but which are not themselves science (or at least not physics). I ask because a physics colleague of mine has recently been warned that he is doing too much methods development (and publishing on these developments).

This is a scary situation. My own tenure decision occurred while the Tevatron was struggling to get itself up off the ground. As a result the physics topics I’d been talking about when I got hired at UW were nowhere near being finished. It was a close scrape (at least, that was how I felt). I had physics in my pocket, but much of it was not yet published. UW, which has had some ridiculously good table-top experimenters - holds everyone to a rather high standard. And even under the best of circumstances a HEP person is already at a disadvantages when those standards are applied.

That said, here is my advice. You have to have the physics results. Most big research universities think developing a new method is cool — especially if it will let you do a whole new set of physics results in the future — but the method itself or the sake of the method isn’t all that interesting. The physics results and potential to do physics results is. I have seen people in HEP, for example, get overly involved in the methods development and lose track of the physics side of things. It does hurt them - and this chestnut is true outside of HEP as well.

Now, it would appear that your colleague is not in particle physics but is in table top physics (or similar). One thing that goes into a tenure judgement is expected performance in their field. For example, a theorist is expected to have n-papers per year (I’m not kidding; I’ve heard this said), a table top experimenter isn’t expected to do much their first two years as they build up their lab, but then a good paper every 6 months or so (depending on the challenge they’re facing, of course). In that sense, the tenure decision depends on what subfield you are working in.

Finally, if only one colleague has made this criticism well, it may be that it can be ignored. The first thing to do is ask others in the same sub-field (who have tenure, preferably) if enough work is being done. Often departments will have a formal review process - make sure to have frank discussions during that review process. Make sure to have tenured friends in the faculty that can report on discussions that happen in closed meetings. Finally, look one can look at other people at other institutions in the same field — especially the ones that are perceived as “hot shots.” What are they doing differently? Sometimes it is just a matter of a high wattage bulb burning brightly, other times you can see strategic decisions they made - copy them!! I guess most of this is common sense, but it never hurts to repeat it!

More tomorrow on Kevin’s comment.

Of the various sites in France, one I was hoping not to see was the French medical system. Sadly, over the last two weeks we’ve gotten to know it better than I would have liked.

First, we managed to give Julia something called nursemaid’s elbow. The solution is quick and instant and requires no drugs, but does need someone who knows what they are doing to snap everything back into shape. This required a trip to the local hospital. Quick and fairly efficient. Emergency room - we were in and out in about 2 hours and that included about 20 minutes face-to-face time with a doctor. The amazing thing about it: they never checked our ID’s. They took down all the information, but never verified that we were who we said we were! And we got the bill last week. Unlike the USA - they send only one bill. In the USA for an emergency room visit I remember getting three or more bills. Guess what the total was? About 34 euros! Even with the exchange rate as crappy as it is now I’m pretty sure it costs more than that to register at an emergency room in the USA!

The second one encounter has been worse. Paula’s mom managed to break her ankle while site seeing and then had an adverse reaction to one of the drugs she was given (”we see this twice a year!” - at least, we think that is the proper translation). Course of treatment? 8 days, in the hospital too. It gets worse - she is totally fine. You’d never guess she was sick. That is “worse” (well, not really) because she is totally bored out of her skull. The doctor has been so apologetic (”I know this isn’t why people come to France…”). We are all going over on Sunday to have a picnic at the hospital.

But check this out. Paula’s mom has Canadian insurance with a rider for foreign travel. When they were called and the problem was explained to them, they said - oh, give us the phone number of the hospital. We’ll take care of everything - even if you have to be flown back under medical care. Wow!

I’m not medical expert, obviously, but the care as been great as far as we can tell. But there is one difference. In the US - at least around Seattle - there is some stiff hospital competition. Perhaps one by-product of that is the millions of dollars that are spent on renovations. The interiors of the two hospitals I’ve been in in Seattle are like high end 4-star hotels. The hospitals here are like the insides of 1970 physics buildings: functional cinder block.

On the other hand, Paula’s mom, who likes good food, claims the food in the hospital is good!! So French!

Amazon has done a lot of work to make GRID computing services accessible to anyone that wants it. Actually, it surprised me that Google or Microsoft didn’t do it first — to run their search engines and other similar things they must have farm computing down to a tee.

In HEP we spend a huge amount of money and cost and time with the GRID. A discussion in a bar some time back generated the question: what would it cost to move HEP into the cloud?

Databases

Yesterday I mentioned databases for storing event data. Amazon has SimpleDB (see this posting to get an idea of how it works). On the surface it looks rather poorly suited to do what we would want to do with our highly structured data. But, ignoring that and some of the overhead it will charge - for the 100 GB of data that Rich had in his database it would cost about 150 bucks a month to store it. Querying is dirt cheap — 14 cents per hour of CPU time used. I have no idea what the performance would be on a database like this, but even if it were x10 slower I doubt it would matter much.

ATLAS’ equivalent database to Rich’s project is thought to be 14 TB/year. That works out to be $21,500/month.

Event Data

Amazon has a simple storage service as well (Amazon S3). Because the data is just a binary blob the cost of storage is much cheaper: 15 cents per GB per month. However, trying to figure out what size ATLAS will actually use if it stored everything in the cloud, and ignored the actual design, is difficult. Making some rough estimates from an old version of the computing model, I’m going to guess about 10 PB per year (that is petabyte!). That is about 1.6 million bucks per month. But we aren’t done with this yet, however - it costs money to move the data in and out. First, just to load the data it will cost about 1 million.

Then we have to use the data - lets say each year we cycle through all the data once — so all 10 PB. That will run about 2.5 million per year (not per month!). But if we use Amazon’s EC2 compute cloud, moving data to it and back is free. In that case, only final datasets will probably be moved. That would be much cheaper.

Computing

This is even harder for me to calculate. This matches up with Amazon’s EC2. One cool thing is data between these computers and S3 is free. Otherwise, for a 32 bit single processor machine that has enough memory to run ATLAS software it looks like it costs about 10 cents per hour of use. Now, in ATLAS an estimate in 2005 was it would take about 3000 kSI2k to reconstruct the average event. So, for an Amazon machine (that is about 1.9 kSI2k) that would take about 26 minutes. So, about 5 cents per event to reconstruct the event. If we expect 2,000,000,000 events per year, then that will cost us $100 million dollars to reconstruct. If someone is familiar with SpecINT2000 and how it works, perhaps they can verify I did this math “ok”. And I’ve not included analysis time which is probably x2 more.

So, there you have it. A lot of money would go into running this in the cloud. Of course, we could never walk up to someone like Amazon and dump this on them. In almost all cases we will do better on our own as we can optimize what we are doing for our uses. Further, the cash that gets spent on this is from all over, and in all different colors. Many nations, for example, buy GRID installations for all scientists in their country. ATLAS just piggybacks on these purchases and uses a portion of them. Still, interesting to see what the cost would be - about 120 million before you even start to analyze the data to produce a physics result!

WARNING: this is very much a back-of-the-envelope calculation!!