I’m Not In France Anymore! August 10, 2008
Posted by gordonwatts in D0, physics life, travel.2 comments
That is Pork’s Knee (or Pork’s knuckle as it was labeled on the menu). And it was good. Big enough to search two, and J-mo ate a little too. Somehow I don’t think you would ever find a dish quite like this in France. And the beer along with it – I’ve not tasted beer that good in months! I love French food and wine, but there are some things done better elsewhere…
I’m in Prague this week, along with the family, attending D0’s yearly workshop. I don’t know how much posting will occur (probably not much). And the week after I’m on vacation being a Dad and Paula attends a big conference. I’m pretty sure almost not posting will occur then.
In the mean time I’m going to enjoy as much Czech beer as I can!
How Hard Will The Hunt Be? August 6, 2008
Posted by gordonwatts in D0, Fermilab, Higgs, physics.2 comments
Yesterday I mentioned that the Tevatron experiments had finally started to rule out the Higgs. I thought I’d post another plot that shows exactly how hard it will be – and so gives you an idea of how much hope the Tevatron has of actually catching the Higgs. Click on the plot to get an enlarged version of the jpeg (here for details).
The most important lines in that plot are the black one (1-CLs Observed) and and the 95% CL thick blue line. The thick blue line is the point at which, in our best statistical estimate, we are 95% confident that we have not observed anything. While the blue line is the “goal”, the black line is where we are now – the current observation. A lot goes into that black line – many different physics analysis contribute (from both D0 and CDF), the physics of the Higgs decay, the physics of how the Higgs boson is supposedly made, and how good our detector is at seeing the Higgs. As you can see, we have just peaked above the 95% level near 170. And that is what allows us to say that we’ve excluded the Higgs around 170 GeV.
Now, the future. You’ll note that the curve is pretty flat near where it peaks above 170. That says to me that when we add more data and minor analysis improvements we will be able to quickly broaden the amount of the observed line is above the 95% CL line. Where the black line is steeply falling, however, it require a huge amount of work (even if it is possible at the Tevatron).
Finally, in yesterday’s post the plot started at 114 GeV. This one starts at 155. What about everything from 114 to 155? Yes — we are working on that. For example, at D0 we have individual results already (and if you look at this plot, given the discussion, you can see that how we are doing as far as getting towards ruling things out at low mass – though the plot is a very different type of plot – but you can guess what is going on if you are not familiar with it). I couldn’t find the recent update of the CDF combined results. But the low mass combination between the experiments was not completed in time for ICHEP. I’m hopeful that we will see it soon – but as they say, it ain’t out until it is ready to be out!
A hunting we will go… August 5, 2008
Posted by gordonwatts in D0, Higgs, physics.4 comments
See that little red blob around 170? That is the Tevatron starting to seriously tackle the final big physics problem left on its plate. Where is the Higgs? The question is — will it finish the job before the LHC starts producing real physics?
The numbers on that plot are the mass of Higgs boson, the final bit of the Standard Model we physicists haven’t directly observed. The last experiment to search for the Higgs were the LEP experiments. As you can see, they searched up to 114 GeV. The Tevatron is searching from 114 up as high as it can go — it so happens the first bit it was able to exclude was around 170 GeV in mass.
The Higgs mechanism is what gives most particles mass. If it was absent from our theory then many masses (and other things) we have already measured would be wrong. That does not mean, by the way, that the Higgs has to exist – but something like it does have to exist. The Standard Model Higgs is just the simplest explanation that we came up with fix the masses. If that whole range is searched and nothing is found – that would be huge news. And very puzzling!
Press Release Here. And combined CDF and D0 note describing the analysis here.
ICHEP Should Be Good! July 31, 2008
Posted by gordonwatts in Conference, D0.6 comments
ICHEP is really getting under way this Sunday — it should be a good conference – with a bunch of interesting results. Some I know about, some I’ve heard rumors about — I’m eager to see what is actually going to make it out there. The only bummer for me is they aren’t using CERN’s agenda system so I can’t just DeepZoom the thing!
Oh — an D0 just passed another milestone in data it has collected — over 4 fb-1 now! The results shown at ICHEP are on the 3 fb-1, however.
Bye & Good Luck, Aran July 9, 2008
Posted by gordonwatts in D0, University of Washington.3 comments
Well, it’s happened. Aran has been with the University of Washington for the last 5 years – as a post-doc. And as of July 1st, he has moved on. Without him UW’s presence at D0 would be much less. He has been UW’s public face there – maintaining the Level 3 DAQ system, and running the single top group during our evidence paper. He has also been one of the best student mentors I’ve ever seen.
But fear not — he has definitely not left particle physics — he is now a professor at the University of Rochester!! I find this particularly cool because that is where I went for graduate school myself.
Best of luck!!
Like Red Meat To Dogs June 7, 2008
Posted by gordonwatts in D0.4 comments
So, what is the quickest way to fill up an experimental particle physicist’s InBox? Write an email that looks something like this:
I’d like to propose we change our shift schedule. Instead of shifts from 12pm-8am, 8am-4pm, and 4pm-12pm how about 4am-12am, 12am-8pm, and 8pm-4am? I think it would help everyone get a better night’s sleep.
Whew. That was proposed on one of D0’s mailing lists yesterday and I think I’ve got over 15 massages on that topic alone in an hour and they are still flowing in.
It is a great debate. I personally don’t care one way or the other. It is serious stuff for some — especially when families and teaching schedules are involved. And of course, everyone’s biological clock will likely make them prefer one or the other.
Good luck resolving this, D0. I’ll just continue to do the owl shift (12pm-8am) or the vampire shift (8pm-4am) if we change the schedule.
New Level 3 Lesson May 22, 2008
Posted by gordonwatts in D0, Trigger, computers.add a comment
Ok, here is a dumb lesson I’ve learned the hard way. And thanks to many others who helped resolve it. Lets say you design a distributed system – like your online and trigger and data collection system at D0. This is a medium sized system — perhaps 500 boxes and several 1000 CPU’s at this point. It is key to note that this is a heterogeneous system — many of those boxes are doing different things and have to be custom configured.
Now, since it is heterogeneous, but a distributed system, and all the boxes have to communicate with each other, they have to have a way of finding each other. You definitely can’t use raw DNS and the machine name. Computers change. Sometimes you want to do a hot-swap to an experimental system. Your DNS is managed by a central facility so the turn-around can be a day – and when the accelerator is delivery beam you need less than an hour.
So you have to decide on some sort of name service. Some service that can take a name and reply with a machine. If it is done right, this will disappear into the infrastructure and you’ll not even be aware it is there after a few years.
Ops!
Lets see, we’ve been running since 2001. In about 2003 we started using what the “sanctioned” name server for our Level 3 Trigger and DAQ part of the system. Of course, you have to make sure you know where that name server is for all this to work. We had an alias in DNS for that purpose.
And it turns out that our stuff is one of the few things left using that nameserver. Everyone else loads a python file on the command line. I’d originally designed our system so that you could change the location of a system on the fly without having to reboot one of the components – so the python approach was never considered. And the online system recently cleared out a bunch of machines.
The name server was moved. And that alias? Well, everyone forgot and so it wasn’t established. And then slowly, over time, parts of Level 3 started to fail. Thank goodness it was the monitoring code that failed first. But there were several hours of panic. All of us had forgotten how the system works it has been so long.
Maintaining the same system running for years is so weird. Almost all the code I write I think “Ok — get it running, debugged, and check it in and move on.” Keeping some of it running for years, however, there are other considerations. I bet there are whole books on this. Too bad we HEP people never take the time to read that sort of thing before we do our software development…
HEP in a Database March 19, 2008
Posted by gordonwatts in D0, computers.7 comments
Not everyone is satisfied with ROOT as the “tool” to analyze HEP data. Back in D0’s Run I all the data was loaded into a commercial database.
So, before you roll your eyes – you are right. HEP is littered with database train wrecks (can anyone say Objectivity?). However, most of those had to do with trying to store every single last bit of data that came off the data acquisition system in the database. And then also store reconstructed data. And then, in some cases, even the analysis level objects. In fact, ROOT grew out disagreement with this vision (and you can tell who won…).
This project, however, was different. The goal was to store only the high level physics information. For a reconstructed jet, for example, they had the four vector and some other quantities (like electromagnetic fraction of calorimeter energies – 28 values in all). They had separate markers for tight very high quality electrons and loose, lower quality, electrons. Same for muons, jets, etc. To understand the limitations of this — and what you might or might not do with this tool: if you changed your jet energy scale you would have to completely re-load the database. This is not something you do frequently, but you get the idea: this is to do your final selection – the last mile of your analysis. Indeed, the test case was to repeat the Run 1 top discovery analysis. However, if you can do selection quickly imagine the power for scanning over a large SUSY parameter space!
How much data? About 62 million events. As a raw ntuple it was 62.4 GB of ntuples (small by today’s standards, of course!). It took almost 1000 hours to generate these ntuples – applying jet energy scale, etc. After being inserted into the database it was 80 GB of raw data, and another 30 GB of database index data.
They used Microsoft’s SQL Server for this. On a qual 450 MHz Pentium II with 256 MB of memory. Does that tell you how long ago this experiment was done!?
Actually, their DB design was pretty clever. All electrons in one table, all jets in another. Then another table which just listed all tight electrons, and another one that listed all loose electrons, etc.
So, how fast did this thing run? So, looking for a Z boson goes to two electrons took about 7 seconds. It found about 6000 events – the right number. Looking for a W boson decaying to an electron and neutrino took about 18 seconds to find 86,000 events. That is pretty darn good!
Are there plans to do this in ATLAS? Well, perhaps. We have a physics summary database – but it isn’t complete (e.g. doesn’t have all the jets in an event). It its design goal is different: you use it to select a sample of events you actually want to run over.
The project was lead by Rich Partridge at Brown University (with a lot of help from an undergraduate Matt Bowen). For more raw information you can see a talk by Rich at a SLAC meeting the other day (CERN ATLAS agendas, look for meetings on Feb 27, the SLAC ATLAS forum).
At any rate, this was something I’ve been meaning to write about for a while. Unfortunately for an approach like this, about 95% of an analyzer’s time is spent trying to understand what exactly is a tight electron – and its fake rate. However, anything that makes for fast turn around is a boon in my book!
Understanding an old Level 3 Bug March 17, 2008
Posted by gordonwatts in D0, computers.1 comment so far
About one or two years ago I had to fix a bug in the D0 DAQ Supervisor. The Supervisor is responsible for coordinating the configuration of 400 or 500 farm nodes and about 80 front end crates that generate the data. It is massively multi-threaded. When it is at its busiest it has over 200 threads running. Most are simply me being too lazy to do anything but block while trying to send data to the Internet. Back in the day it ran on a slow dual-core machine under Linux and I did my best to avoid all locks that I could in my multi-threaded code – because locking is expensive, and the Supervisor needed every bit of speed help it could get back then (on a modern machine it is plenty fast enough).
My code was basically some initialization like the following:
global_a = 1.0;
global_b = 5.0;
global_inited = true;
Once global_inited was set to true, then I knew it was safe for the rest of my other threads to look at a and b:
if (global_inited) {
use-global a…
}
Unfortunately this didn’t always work – sometimes the program behaved as if random values had been entered for a and b. I was never able to reproduce this either. It would happen only once in a while, and restarting the supervisor usually fixed it. Eventually, to fix this bug, I re-structured my code so that all the initialization happened before any other thread was started. After that I never saw the bug again. But I never understood why I was seeing the bug!
A guy who works deep in the stack at Microsoft recently started a blog. One of this first posts explains, possibly, what bit me: the compiler and the CPU (both!!) are allowed to reorder the order that global_a, global_b, and global_inited are set!! Since this bug was not reproducible it was probably done by the CPU, though at the time I never tested that (or ever really figured out what caused this).
Superstition in the D0 Control Room March 14, 2008
Posted by gordonwatts in D0, Fermilab.add a comment
There are lots of old superstitions – some of them we still live our lives by. Running a large experiment like D0 is no different. For example, there are a set of ducks along the console – the rumor is if they aren’t there then the whole system will cease to operate. I don’t think anyone has been brave enough to remove them… 
I pulled the following quote from a recent shift report:
Beam was nice for a while. Then while talking to Bill Lee about losing the beam, we lost the beam, thereby illustrating Bill’s spooky powers in the control room.
Bill has long been making our control room run smoothly, and should know the lesson: don’t talk about loosing the beam! You’ll jinx it!! [Technical reason: apparently an important power supply went out of allowed operating range].
