Life as a Physicist

One of the nice things about being on a new experiment is that I’m not key to its operation. If I don’t do something, or if I sleep late then the experiment won’t be impacted. Not having that responsibility is nice - relaxing. I’ve been operating like this for almost a year now. I and a student and a bunch of others have been working hard to release a note on bottom quark tagging calibration. Fun, and will become key when data arrives. But real data will arrive only if we are lucky later this year. So, not so much pressure.

But, frankly, I’m getting tired of no pressure! There is definitely a part of my personality that likes being in the thick of it.

About 3 weeks ago I started working on a high priority software task - modifying how we write out some of our b-tagging data. The reason this is high priority is we are preparing our software for the first data - so all major changes like this need to go in so we aren’t stuck with the data in an old format. A little bit scary - all of a sudden I’m giving updates in several meetings, I get asked about how things are going in random emails. And I’m deep in the middle of the code muck - something I really don’t understand at a fundamental level.

Have to say, it is fun!! Hopefully I won’t tire of the fun until after I’ve finished the task!

A little bit of fluff. I suspect about half the people who read this blog already know this. I keep forgetting it — though I use the words “core dump” almost everyday (hey - right now I’m working on ATLAS software!).

The term “core dump” relates to an old type of computer memory, in use before the advent of silicon chips, which used grids of tiny magnetized rings to store information. When a computer program in such a system crashed, a record was created about the state of the memory. “Core dump” has come variously to mean any fatal error in a program, as well as slang for getting the long version of a story from someone.

This is from the SLAC today web mailing. The Fermilab version is a bit depressing right now: any day now the layoffs will be announced - and that day is going to be depressing when it finally arrives.

We were out having a pre dinner drink near the Vieux Port this afternoon when the square suddenly filled with people. Then someone blew a whistle. We were sort-of aware there were lots of people, but then after the whistle we looked around and noticed someone pointing a banana, frozen in time. What?? A weird sort of flash mob, perhaps? Plenty of weird stuff going on this afternoon in Marseille. We are going to miss Marseille when we have to return to Seattle.

Noticing something very bad is about to happen when your child sits down on a raised plaque at the base of a tree, and you only have a split second to give warning, your brain comes up with the following sentence: “Honey - Julia shouldn’t sit on that inclined plane because it is covered in poop!”

On the drive from Marseille to Genova I asked the 3 others in the car if they used Instant Messaging (IM). I was the oldest in the car (getting close to 42!) and the rest ranged from mid thirties to late thirties. All are like me in that they have permanent positions. All have students, though I was the only professor - the rest were at labs around France.

How many people used IM for work? None other than me. Who used IM at all? None, other than me. On the way back I was in a car with graduate students - so mid to early twenties - but I totally forgot to ask them. However, since I rode with their advisors on the way out there I can be pretty safe in guessing they don’t use IM for conversations about physics with their advisors.

So, I have a bunch of questions. Do people use it mainly for social interactions? Or am I just in a bubble and it turns out that lots and lots of people use it for work and I just move in the wrong circles?

It was the day, not much more than two decades ago, that HEP lead the world in computing hardware and (to a lesser extent) software. We had to push it forward - the power and flexibility was just not there. I think if you look at the way we operate now - it looks very similar to how it looked back in the ’60’s. Command line. Simple text editors (vi and emacs are BIG). Batch jobs. The older set are only just getting off pine - a text/terminal mail program (moving mostly to the FireFox mail mail reader, and some to web clients).

Is there something holding us back? Or are we as a field voting with our feet: these things don’t help us get new physics results out? Perhaps we usually are in labs where you can run from one office to another and so email is good enough? IM interrupts our workflow and so we don’t want to let it in? What about all those Web 2.0 tools out there? Much of that is based on social networks and so if a lot of physicists don’t get on there, then it doesn’t really help. Are we slowly building critical mass? Or does it just not excite us yet?

Comments?

Just to make sure I stay ahead of the curve, I’m trying out friend feed - think of it as a way to aggregate all of your online activities… Of course, I don’t have many online activities.

Graduate student and advisor bonding over Joss Whedon and Buffy.

Having to go from 8pm to the next morning at 9:30am without a portable. Didn’t get to write a talk I was asked at the last minute to talk about and also got a full 8 hours of sleep. Not done that in months.

A less scary moment was getting lost on the streets of Genova, Italy. This city is packed to the gills with buildings - it is impossible to get a lay-of-the-land unless you are on the side of a hill.

I have pictures, as usual. Along with some from a pretty stunning hike.

I noticed in a Fermilab Today issue (the May 3rd issue) that Fred Ullrich retired. He ran Visual Media Services forever (well, as long as I can remember). I’ve interacted with him on and off for years. Sometimes begging him to prepare a poster for a conference that I’d waited until the last minute to finish off. I wanted to send him a congrats email. But I can’t find his email address!

So, as usual, I’m going to take this one little anecdote and extrapolate it way beyond reasonable limits (this is a blog, after all). The normal place to look for this sort of thing is the Fermilab white pages - an online directory. But the problem is Fermilab has taken computer and employee security to the level that pretty much the day you leave Fermilab all trace of you ever existing is removed from their databases. I realize this is how all of corporate America (and perhaps the rest of the world) works, but the problem is Fermilab feels like a “home” to me - and so this seems like fairly harsh treatment. Not something you’d do to your friend - especially as they are leaving on good terms. Queue the refrain “Ah, it was better in the old days…”

Bummer. At any rate - Fred - thanks for all your help over the years, and have fun starting the second half of your life!

Visual Media Service has a great collection of pictures taken over the years at Fermilab (ha! the page still says to contact Fred — he isn’t gone yet!!). Well worth looking through!

Being an American, I’m a capitalist. Thus, I’m greatly in favor of competition. And there is a big competition coming down the pike: ATLAS vs. CMS. Both are going to be racing to find the not only the big discoveries but also to be the first to see a Z boson, or the first to see a top pair production. In the long run I think this competition will be healthy - it will make sure results come out in a timely fashion and keep the two experiments on their toes. It is most embarrassing, after all, when your competitor points out a big mistake you’ve made!!

A bunch of us were discussing this over lunch the other day at the ATLAS Genova b-tagging workshop. One person was very against the idea of the competition, pointing to the damage it had done at CDF and D0. This surprised me; I’d always thought the competition there was pretty healthy. No, he said, from the outside he sees lots of name-calling and trashing of each other’s results.

Yuck. I hope there are very few people doing that. I can understand where the feeling comes from- you work for months or years only to be beat in the last few weeks. The science is intensity personal. Still…

This lead to us trying to sort out what was good and what was bad about competition. The title of this post was the tag line we came up with.

N.B. Not all American’s are capitalists!

I’ve mentioned before that I think multicore computing is going to hit HEP hard. The basic problem is that we run all of our jobs in an embarrassingly parallel way. If the machine has 8 cores on it, then we run 8 reconstruction jobs. There is nothing wrong with this on the surface, and, indeed, performance tests indicate that so far we observe an almost linear speed increase as a function of the number of cores.

The problem, I fear, is getting data on and off the chip with the many cores. For code to run fast it must be well fed with CPU instructions and the data it is processing. Memory and the CPU have only so much bandwidth. The HEP way of running means that all the various cores are working on very different things - different data, different code - which means on-chip cache hits will be low - requiring more data from outside the chip.

At some point this has to start to impact performance. We’ll see that when we start getting linear increases in performance.

Ars had a nice article a day or so ago on this issue when discussing the various processors and their multi-core capabilities. The study they reference looks specifically at the memory bandwidth. IBM’s cell processor did the best - this is the chip used in the PS3 gaming console. It was specifically designed for high memory bandwidth. Intel’s architectures didn’t do nearly as well, however.

I also found it in interesting that before they were able to test the memory bandwidth they had to deal with some other bottlenecks in the Intel chips. Specifically, the Translation Lookaside Buffer (TLB) (see section 4.1 of the paper for a detailed discussion). Basically, every time the CPU goes to memory it must translate the requested memory address into a physics memory address. To speed up this process, the CPU maintains a cache of these translations. If you hit the cache your memory access proceeds at a very quick speed. If you don’t, then things grind to a halt while the CPU calculates the translation. If you are accessing data spread over a large area of memory you are bound to constantly be overflowing the the TLB cache. This is interesting because it strongly resonates with one of the presentations on optimizing CMS code at CHEP. Specifically, their memory allocation and deallocation (see page 18 of the talk). This means the CPU is constantly jumping around from whatever code it is working on, to the alloc/dealloc, to scanning memory for free blocks - all these operations use up slots in the TLB without actually “getting work done”.

Fortunately, in the Ars article, they mention that the CPU manufactures, like Intel, are aware of this, and the next generation of chips will have much larger TLB’s - which should help with this issue. It is always the next generation, isn’t it?

I wonder if we in HEP will ever hit this memory bandwidth bottleneck? Perhaps when we have 8 cores on a chip (when is that predicted to happen? Next year!? :-)). Getting around the bottleneck will require some major work in how we write our code: it is all inherently single-threaded.

BTW, the point of this study was to optimize a physics simulation — a magnetohydrodynamic system. Physics is everywhere!

Congratulations to everyone at D0 and the Fermilab accelerator division — D0 has been sent 4 fb-1 as of April 30!! That is a lot of data! And years of work!

It will be a while before you see that in an analysis however. First of all, of the 4 fb-1, only about 3.5 fb-1 were written to tape. The rest is lost forever. Where did it go? Well, perhaps our detector was broken for some short amount of time. We do our best to make sure that doesn’t happen of course, but a machine like this does break (my name is no more than a few of those minutes for Level 3/DAQ problems!) We generally run our triggers with a small fraction of dead-time — time where we aren’t accepting new events even if they are coming in - if we were to run with zero dead-time we’d not be able to do nearly the physics program we do [if you want more details, let me know].

The second issue is time. It takes time to understand our data - a while to do something as sophisticated as a Higgs search. Parts of the detector are turned off, which affect efficiencies and systematic errors. The events we trigger on are changed as we try to optimize our data for the higher instantaneous luminosity the Tevatron delivers. On top of that, of course, is the continuous effort to improve our analysis technique’s power. All these changes must be carefully studied to see how they impact each analysis. And once that is done, only then can the data be shown externally. This takes quite some time. The more sensitive the analysis, the more carefully the data must be studied.

So, fantastic for us at D0 (and CDF) for reaching 4 fb-1 delivered. Thanks to the Fermilab accelerator division for doing this despite the trying times. Everyone else: sorry, you’ll have to hold your horses a short while before we show the results of this data!