Life as a Physicist

Nick asked a question on one of my posts:

Okay, humor me for a moment while I learn something: is this laughable because it suggests there are many “types” of higgs particles (which I’ve only ever seen reference to in this article) or just because the article suggests that the search is over and then quietly notes that we’re really still in the same place we were before the article was printed?

The latter. There have been discussions of the possibility of many Higgs for years — for longer than I’ve been in particle physics (>20 years - yikes! That is scary!).

How many Higgs particles depends on which world you live in. Lets say you live in the plain old Standard Model, and only the Standard Model. In that case, there is just one, the ONE Higgs. At the moment the Standard Model predicts pretty much every result we can measure. The Higgs particle was added to the original version of the Standard Model in order to get the W and Z boson masses correct — those are things we can measure today (unlike the Higgs, of course, which remains unseen).

Actually, that paragraph contains a lie — the Standard Model can’t explain everything — dark matter and dark energy, for example. There are other reasons why we think the Standard Model isn’t the whole story as well - so we have to fix it. So, on the one hand, we know it isn’t complete, but on the other hand we also know it can predict the results we measure at current experiments to amazing levels of accuracy. So, if we do fix it, we have to be careful of not breaking it in the process.

So we “extend” it. We develop new theories that “contain” the Standard Model. There are lots and lots of these theories. One of the more popular ones is called SUSY. Another is extra-dimensions. And there are more. Some of these models, like SUSY, actually contain 5 Higgs-like particles. The one that CP proposed in the paper referenced by this article also has 5 Higgs. In these extensions, btw, they must still get the W and Z masses correct - as the Standard Model does — and that is what Marcella (I think!) is complaining about: You might find another Higgs like particle, but the one in this model that does the work of the Higgs that gets the masses of the fermions and bosons like the W and Z right is still going to be just as hard to find.

On the other hand, finding a light Higgs as suggested would be a revelation. And grounds for a Nobel prize if the theory was born out. So the article that CP wrote is just fine. I was complaining about the way the press wrote it up.

Or should I remember that all press is good press?

I’m not sure how this article showed up in my web browser — I must have clicked on some link without being aware of it. But check out the lead paragraph:

Thousands of particle physicists are spending billions to try to spot the elusive Higgs boson, which is key to explaining the origins of mass. But evidence of the Higgs boson–or at least a Higgs boson–may already be lying unnoticed in data from previous experiments, new calculations suggest.

Wow! I’d better read this! And then…

However, this lightweight Higgs is not exactly the Higgs everyone is looking for, says Marcela Carena, a theorist at Fermilab. “The Higgs they are talking about is not the one responsible for giving mass to the W and Z,” she says. It can’t be because it hardly interacts with those particles, Carena says. Indeed, in Yuan’s model, the role of mass-giver falls to one of the heavier Higgses, which is still heavier than the LEP limit, she notes.

Oh. Right. Back to work.

I spent last Thursday, Friday, and Saturday at a really great conference in Grenoble. The topic was top physics, and lessons and techniques at the Tevatron and at ATLAS. I wish I could put my finger on what made it great: it was a small group (less than 50 people, I think), and everyone was willing to talk. The first day we ran about 3 hours over, for example (yes, yes, a good 30 minutes of that was my fault). Both Lorenzo and Harrison gave great talks as well as the others. I took some pictures too.

I have pages of notes from the conference (all stored electronically, of course, despite my busted laptop - which is getting worse). Some tidbits (if you don’t care, skip over to the last paragraph - partly to help me remember):

• At the Tevatron we produced about 8 tops per day, and 4 single tops per day. LHC will be one per second, and a single top 1 ever two seconds. This is a game changer - top becomes a calibration sample, for example (b-tagging, JES…). I can’t help but wonder if we haven’t explored all of its possible uses as a calibration sample yet. Heck, data for 9 hours at low luminosity (but full luminosity) will give you a top mass peak that you can see! And it is going to be exciting to measure that cross section at 14 TeV!! And they really are are going to do everything without b-tagging at the start of the run!
• No one yet talks about doing analysis in such a way to make the eventual combination with CMS easy. I really wish we’d start thinking that way from the beginning.
• I had not fully appreciated the history of the D0 JLIP tagging algorithm — it was nice to see it in action as far back as LEP and also (as I’ve seen) in ATLAS. I’m sure it is in CMS as well.
• During my talk I made the sarcastic comment that one shouldn’t measure things twice - because you are bound to get different answers. I really did mean that sarcastically — but people kept mentioning that I mentioned it thought out the rest of the talks. Sheesh.
• There was a big difference between the ALTAS and D0 talks. In general, the D0 talks talked about one method, and then went into all the details and tricks and the general mess than a proton-proton collider causes. The ATLAS talks were often an overview of techniques. For example, for the Jet Energy Scale ATLAS showed results (but not details) from about 4 different methods. I’m sure this will change when data arrives.
• Harrison made a rather provocative statement fairly early on in the meeting: we spend too much time trying to get the absolute scale of the Jet Energy correct. Instead we should just match the Monte Carlo to the Data and then move on; who cares about the absolute scale. I and others made the obvious point: in order to do the match don’t you end up with the absolute scale. After several goings around (over wine at lunch!), we figured out: he completely agreed that reducing the error on the jet energy was well worth the technique, but he still wonders if we could save time by not developing independent JES for MC and Data.
• I need to start exploring TMVA.
• In general, the D0 analyses are quite sophisticated — using multivariate techniques like decision trees. Most of the ATLAS analyses don’t yet. In fact, there was some scientism from the audience about our use of these techniques. M. Mangano came up to me at one point and asked how we knew that our decision tree analysis had actually seen single top and not something else (MC tests, the Matrix Method also sees it and it isn’t a machine learning, etc.). It sounds to me like this battle isn’t done yet.

There was only one thing I didn’t like about the conference: the agenda page is password protected. Unless you are a member of D0 or ATLAS you aren’t supposed to have the password. The reasoning is as follows: they wanted to have D0 people be able to talk about internal matters — really wanted to know the dirt behind what went wrong and what to try to avoid when working on ATLAS. And the ATLAS folks should talk about anything they wanted to as well - regardless of weather or not it had been approved. It didn’t work out that way: the ATLAS top group reacted saying that they didn’t want preliminary results shown. The result was the page was password protected. This is too bad, and I think, in the end, the password was not required. First of all, ATLAS is only showing Monte Carlo results (though the ATLAS philosophy is different than the D0 one - D0 is quite comfortable with MC results going out without nearly the cross-checks that data results get). Second, as one speaker put it, “new versions of these results will be shown shortly in ATLAS and will improve on these old results” — so the results we were seeing weren’t the preliminary ones in most cases. As for D0, I think all of the D0 speakers got an email from the top group reminding us that only publicly approved plots were allowed to be shown at this conference. Fortunately, it did not limit the discussion and all of us were quite frank in our revealing of faults and other problems. I am in general against external conferences being password protected.

I’m spending this week in Victoria, Canada attending the Computers in High Energy Physics (CHEP) conference. At one time this was my favorite conference (more on that in a later post). These summaries are just things in the talks I found interesting on day 2. [I spent last week there -- I'm just getting around to cleaning up these posts!!!]

Plenary: The Future of GRID Computing - Miron Livny

Most people I talked to really liked this talk. Miron’s group at Wisconsin is the author of Condor, one of the more popular GRID aware free batch systems out there. We’ve made heavy usage of it back at UW to process ATLAS events. But I didn’t like his talk all that much - mostly because I don’t like GRID talks all that much. I got made fun of in a plenary session 2 or 3 CHEPs ago for complaining that we were talking about the GRID too much. I didn’t think it could get worse — but it has - even more of CHEP is devoted to computer science problems (getting jobs to run on the GRID) than it is on physics problems that can be solved with computers.

Miron started on a promising note: GRID computing is dead. What he meant was that you no longer got automatic funding by attaching the word GRID to a proposal. Then he said, “but distributed computing is here to stay. Lets get back to work and away from the hype.”  !!

He also has one slide in his talk where he says that we have to do better in job reliability. I’d like to point you to a NYTimes article that was published today talking about complex systems and how error prone they are. What irks me a bit is that the GRID folks have been stressing reliability for years now and their progress is, well, slow (at least to my eyes). New features and new tool kits seem to appear with alarming frequency — but no one goes back and makes sure that all the layers work well together.

He now helps lead the OSG consortium. This is a loosely affiliated group of labs and universities that have banded together to supply large computing resources to various branches of science (not just HEP). They have some umbrella funding, but its future is not obvious. I wonder if consortiums like OSG will last or will we live in a world of very large facilities like TerraGRID that we buy time on?

He also stated what would be a repeated theme: moving data files around is the single largest weak point of the GRID currently. He is right about that!

Plenary: High Performance Computing at the Petascale and Beyond (Blue Gene!!)- James Sexton (IBM)

James gave another fantastic vendor talk. It was interesting to see the big difference in IBM’s vision vs other company’s visions. Most of the discussion was of Blue Gene and how it fit into the future of computing. IBM positions it as a research machine that they can sell. They expect to develop new technologies here and then slowly feed it down into other product lines.

Blue Gene is a move away from commodity processors: they attempt to put everything on a single chip: memory controller, bus interface, etc. With as many processors as they plan to put in a rack they need absolute reliability. They are using a core based on the PPC (duh) running at only 800 MHz. They put 32 of these cores on a single card along with 10 Gigs of memory. This means each core has about 300 megs to call its own. They then pack these cards into racks — about 4000 cores per rack. They then have high speed interconnects between the cards and racks.

Livermore has 64 racks. Their mean time between failure is 7 days. All I can say is wow! Our L3 system runs about a week before a node fails and we have only about 1400 cores in that (small!) farm. 64 racks is 256,000 cores, or about 180 times bigger than our L3 trigger farm.

Applications that have run on Blue Gene include atomic and material simulations. The comment from the researchers was that these types of problems weren’t each to put into a BG, but it was possible. Protein folding also did some work — but say they need another x3 in compute power. Ouch!!!

James, along with many others, pointed to the fact that the cost of memory is not falling very quickly, so economics is going to force one into the Blue Gene model, or something that looks like it. Further disk storage and getting data in and out of disk storage is also not keeping up, leading one to wonder if it will be cheaper to re-derive results than to store the intermediate results?

His future predictions were pretty guessable: the 2-4 GHz max clock rate was here to stay, but be ready to move to using millions of cores. Memory bandwidth is going to be a problem, and reliability is going to become so much more important in these 1000 core systems.

Plenary: Canadian Cyberinfrastructure - Bill St. Arnaud (CANARIE)

Bill started this by saying that Canada was late to the GRID. This statement surprises me: I remember Canada being one of the first GRID sites to help D0 out with Monte Carlo processing. The CANARIE group is particularly strong in networking. Bill talked about several technologies that allow one to reconfigure networks on the fly. I’m not going to claim I understand this, nor do I understand why this is particularly attractive (except, perhaps, from a security point-of-view), but other talks at the conference made me think this scheme is going to be widely adopted.

One cool project they were working on, Neptune, was a large undersea network of remote sensors. A NYTimes article happened to be published the same day of the talk. Very neat!

Parallel: Booting ROOT with BOOT - Rene Bruin

The idea behind this coming improvement to ROOT is that most ROOT projects use a very small portion of ROOT. So why does all that memory have to be loaded? Why do all those files have to be distributed? Further, there are small bug fixes and it would be nice to distribute them without having to redistribute the whole of ROOT — and allow the user to update on the fly.

In order to reduce ROOT’s memory footprint, the ROOT team is doing what people have been asking it to do for the last 10 years: reduce interdependencies. A rumor has it that the straw that broke the camel’s back was Rene made a minor change to the TClass.h file and discovered that he had to recompile all of ROOT. This is fantastic! Reduced dependencies (and the ROOT team has discovered abstract classes in a big way! Wow!) should make ROOT a more stable product in the long run. Rene showed some data towards the end that had evidence that in the latest versions of ROOT the memory foot print had already been reduced — and, thus, the start up time had also been reduced (they are down from a peak heap size of about 30 megs to 4!). Fantastic!

Parallel: ROOT Graphics: Status and Future - Oliver Couet

Oliver described a bunch of new plots styles (Spider, Parallel, Box, and others). In particular, the Parallel plots looked like they might be interesting in particle physics. The spider and box plots will take me a while to get my head around in how they could be useful.

They have also shifted ROOT over to Open GL now, which has enabled some of the ALICE amazing graphics.

Parallel: ROOT I/O Performance and Functionality - Philippe Canal

Philippe is a very brave man. He spends all of his time down in the I/O sub-layer of ROOT. The ROOT team is usually listed as Fons and Rene — but I think Philippe must be getting close to deserving to be added to that list (if he isn’t already past threshold). The I/O subsystem must be a fairly thankless task, though it does contain some very cool problems that need to be solved without sacrificing speed, but it also has to be some of the most difficult and tedious work!

A lot of work has gone into storing data as compressed values — when you don’t need to save full precision. They are using a technique, it sounds like, that Scott Snyder used for D0’s I/O subsystem when he wrote that many moons ago. They have also improved the asynchronous read-ahead over a network — techniques to keep a data hungry ROOT application well fed despite network latencies. In anticipation of greater PROOF adoption, they have also improved code that concatenates results from several files into a single file. Another very cool thing: they can take an ASCII file and read it into a TTree without a user having to write any code! This will be very helpful for quick-and-dirty TTree building! There was also a cool sounding improvement to the TEventList (use TEntryList now) - a sparse and highly optimized way to store events that pass cuts.

I’m spending this week in Victoria, Canada attending the Computers in High Energy Physics (CHEP) conference. At one time this was my favorite conference (more on that in a later post). These summaries are just things in the talk I found interesting. [I spent last week there -- I'm just getting around to cleaning up these posts!!!]

The first day was both Plenary and parallel sessions. They are using Indigo which means everything can be seen online. I came in on the Clipper in the morning, so I missed the first two talks.

Plenary: LHC Computing - Ian Fisk

Ian talked mostly about HEP’s use of the GRID - that big computing appliance in the sky… err, I mean cloud. He showed a great slide that really tells you how much trouble we’ve gotten ourselves into - all those layers up layers in the GRID: “Not all of them included just for technical reasons.” We keep adding layers and adding layers to hide complexity. It is no wonder some of our systems are so fragile! He points out that everyone is developing their own front-end system interface to the GRID. He made the point (along with others before and I’m sure after)  that the level of complexity means that we have to develop more and better automated testing of the tools.

The ALICE experiment has the ability to make custom datasets based on simple event cuts. A user can define a particular set of event attributes and have a dataset made from that. I can’t help but wonder how that scales if every user requests their own dataset (which is very similar to the next user’s) - that is a lot of I/O. But it is good, none-the-less. I see this in ATLAS too with the TAG - but I’m nervous of the same scalability problems there too.

ALICE also has a clever way of sending jobs out to the GRID (I think this is old hat now): the pull model. They submit a little pilot job. It checks out the machine for basic installation and then calls back to a central scheduler to get the actual job that needs to be run. I’ve known this gives them the ability to avoid black holes, but I didn’t appreciate how much more job scheduling flexibility it gave them: some GRID submission queues can be very long!

In the first reference to multi-core (of many!) he pointed out that with the new machines we are using now we need 8 or 9 copies of the simulation or reconstruction running at once in order to fully utilize the capabilities of a 8 core machine.

In general he painted a rosy picture of the tools. This didn’t match my experience and I asked how things look so good but seem to not work for the end user. He mused that perhaps 90% operation was still pretty bad from an end user point of view. I think it is still worse. I got a bunch of ribbing for that comment later on in the conference.

Plenary: LHC DAQ - Sylvain Chapeland

This was the usual overview of the LHC DAQ. One thing impressed me was that every one is using ROOT to display plots. Java makes an appearance also, but ROOT is the real king here — any time a plot has to be made. Most of the operational displays shown were a few buttons and lots of histograms — I usually think of DAQ operations being a bunch of buttons and some text files. I think this was speaker bias.

He had a comment on the last slide — wondering if we couldn’t have a common DAQ system. Years ago I also wondered why we couldn’t do that. At the very least, a common DAQ system. Indeed, Fermilab has a project that does this for their fixed target experiments (I can’t remember the name). I think this will never happen for the large experiments like ATLAS or CMS. The hardware is so specialized, and the dataflow is so specialized. I can imagine that we might make use of libraries of utilities — but we’ll never see unification of the overall DAQ system or its control.

Plenary: Computer Facilities - Eng Lim Goh (sgi)

Unfortunately, the slides have not been posted. This talk was fascinating (as were all the vendor talks, actually). Goh noted that the data rate out of the ALICE DAQ system, 1.2 GByte/sec, was the same as what the film industry was seeing for their “4D” compression for a movie - so the real world is catching up!

He has seen 10,000 node rack & stack clusters close to production and hears people now talking about 100,000 node rack&stack clusters. Wow!

sgi’s vision of the future are very tightly packed racks. They do away with the pizza box stacking and instead just build CPU’s on a board, stuff the boards into a special high speed back plane, and build modules out of these. Input and output uses Infiniband, which gives performance on par with 10 Gb ethernet.

Supplying power was also a focus — and doing it efficiently. Apparently, one usually gets from line voltage via two steps. sgi has put it in a single step, and they have brought the power supply out to the back of the rack to supply a whole lot of nodes - thus they claim to have one of the most power efficient (watts/flop). Of course, Goh mentioned the backplane that they plug all their processors into now has 1000 amps coursing through it! Yikes!

Despite the efficiency improvements, they still have to cool the rack down - they use water cooling. Goh claims it is neutral — you don’t have to add extra AC to the room - the water chiller will take out at least as much heat as what goes in. Interestingly enough, the coolers are mounted on the back of the rack, after the air has flowed through the processor.

The solution is quite unique — in that it doesn’t play well with other solutions. This question was asked, and Goh said that in units of a single rack it did play well — sgi supplies standard GB interfaces in and out of a rack so that one’s whole data center doesn’t have to be infiniband.

Parallel: ATLAS Analysis Model - Amir Farbin

I managed to miss most of this (late back from getting a Dr. Pepper!), but his talk mentioned several things that looked interesting that I’m not already familiar with:

• SPyROOT - The specific example knows about data sets, relative luminosity, and cross sections and can automatically make plots for multiple datasets at once. Written in Python. The batch version seems to be a simplified version of what we use in D0 (in Python rather than C++). Cute. Not obvious it is robust enough - seems designed only for simple things — no way people are going to end up doing just simple things there!
• We will only simulate 20% of the data we take. I hear this said all the time. A particular analysis only looks at a small subset of the data — will this number matter?

He also talked about his EventView approach to doing analysis — which I think is not going to work out well.

Parallel: A Class To Make Combinations - Nobu Katayama

This was a very cool talk on an algorithm to solve a physics problem - a set of ideas you could walk away from CHEP with an immediately apply. I’d like to see more of this sort of thing!

Nobu was dealing with combinitorics in B-physics. Specifically, a 3 body decay. He wanted to represent it with C++ operator syntax: “D0 = Kaon*Pion*Kaon” — in order to do this you need to loop over all three objects at once, but C++ operator syntax means you will do the Pion*Kaon first, and then the Kaon*<result> second — not evaluate all three. So he wrote a delayed evaluation library: it builds up a syntax tree of all operations but doesn’t actually do them until the “=” sign is hit. At that time the whole parse tree is available and he can see that he needs to do a 3-way combination. Very cool! I’ve seen this technique used to solve all sorts of computational problems (like shipping matrix operations out to a GPU).

I asked why not use a Domain Specific Language: a text file that contains the physics in a more readable form that you could then use a program to automatically turn into C++: easy to read and write, and the speed of C++. He really wanted to complete flexibility of C++ available (for cuts, etc.). Sounds good!

Parallel: The CERN Analysis Facility - J. F. Grosse-Oetringhaus

ALICE is probably the heaviest user of ROOT of any of the ongoing large experiments. They use it for everything - reconstruction, simulation, visualization, etc. You name a feature of ROOT, and they are using it. This includes PROOF, the remote cluster version of ROOT. They currently have about 40 machines setup (500 is the goal). PROOF has, from the sounds of it, received significant debugging on this cluster. Listening to this talk you really get the idea that PROOF is big-iron. Lots of support is required to have a fully functioning system, which is too bad.

I asked how you distribute user code and libraries - you send source files and they are compiled on each PROOF machine that is running. What do users do when they want to run an analysis in PROOF with lots of plots - speaker didn’t know, but he did know that some people were making 100’s of plots in a single event loop (the only way to do it, in my opinion).

Parallel: Visualization with ROOT - Matevz TADEL

Wow. ROOT visualization is pretty impressive! The ALICE detector is a heavy-ion detector. This means when the ions collide there will be a huge mess of tracks in their detector. They are using the visualization code in ROOT as their detector display, and they claim decent performance when they show as many as 1000 tracks. And this is without using acceleration hardware (as far as I can tell). I wonder how fast things would go if they used modern graphics cards and did processing there?

Parallel: The Online Track

There were a lot of good talks on High Level Triggers at this point in the Online Track — but I have seen many of them before. Suffice it to say that good progress is being made and people are already at the point of testing the algorithms to make sure there is enough CPU to do the things that they need. In particular, one referred to a version of the track finding algorithm, called a Kalman Filter, that didn’t require matrix inversion (very slow). I’d not heard of this gain-matrix formalism before, but I found what looks to be a decent reference on the web.

At last Friday’s Fermilab Wine & Cheese CDF showed slides in which they crossed the magical 3 sigma threshold for observation of single top: Evidence! They have two analyses — the Matrix Element and the Likelihood Fit. The Matrix Element is at 3.1 sigma, the Likelihood is at 2.7 sigma. The main change since their last result seems to be a substantial increase in the size of their data set (1.5 fb-1).

That must have been a lot of work! Congrats!

D0 is not updating its results for this round of conferences — so we remain at 3.5 sigma.

The Matrix Element shows a steady increase in sensitivity that is understandable. But I’m curious to know what happened with the likelihood. Back at the end of 2006 it showed no sensitivity in the 1 fb-1 version of the analysis (reported an expected limit, and the observed limit was even tighter than the expected one), but now has almost 3 sigma expected sensitivity. Were they just unlucky before and lucky this time? Or did something fundamental change? Unfortunately, this talk is not the talk to describe this — Mark’s job was to present all the recent new results in 45 minutes; so no time for details!

UPDATE: It was pointed out that we are now at 3.6 sigma, not 3.5 sigma. Ops!

I’m listening to the last day of talks at the ATLAS Overview week. A talk just pointed out that we expect about one top quark to be produced in ATLAS (and CMS) every second. At the Tevatron, on a very good day the rate is closer to 1 every couple of minutes! A 1 fb-1 sample of data at the LHC is going to be worth so much more than the Tevatron when it comes to this high-pT physics. Of course, this is the reason we built the machine!

Not of a person, but of an experiment. HERA has finished. Its last beam was on June 30th. Many many people worked on the experiments (H1 and ZEUS) at this accelerator — I know many of them. When something like this happens it is like a hole opens up in your life. Our experiments now run for so long…

Recently one of the readout crates in D0 started having odd problems. The hardware my group is responsible for started crashing. These were Linux kernel panics: the OS on the board had given up the ghost and was checking out for lunch — instead of taking data for the D0 experiment.

These problems are very hard to track down. And this one was starting to look really tough. All of us were thinking about what software bug had been recently introduced or perhaps the other hardware in the crate had started feeding us bad data and we weren’t protected against it?? The error was happening in the Linux kernel too - which makes debugging rather difficult.

Then Dan, who was in charge of the whole crate, noticed that one of the cooling fans had died. He fixed it, and there have been no crashes since. The problem was that most of us haven’t had to deal with the hardware in so long that it didn’t really occur to us that this could be a hardware problem — less that it could be a cooling problem! The basic problem, is, that the hardware has been too reliable.

This reminds me of a set of computers we purchased to run a farm in this same system. The fans on the CPU’s were all cheap, and started to fail. It took us a while to figure it out: the machines started randomly crashing.

It isn’t all software! How many of us are still working hard on hardware? Probably not enough!

My Dad just pointed out to me that D0’s single top PRL has been marked by PRL as “worth reading.” I wasn’t aware of this designation — but it is generally meant to get readers who are in other sub-fields of physics to read the paper because it is both well written and important. That is cool!

I am also impressed that my Dad still gets the printed version!

N.B. Some of the above links point into the PRL archives. These may require you or your associated university to have a subscription. Sorry!