A Very Bad Sign September 17, 2006Posted by gordonwatts in physics.
Look at how many people are there. And 10 minutes before, when I ducked out to get my camera, there were almost twice as many. DZERO needs 5 people in the control room to run. There is significantly more than that there. That can only mean one thing: experts. And what do experts do? They mess with the system. They try to “fix” things. What happens then? Well, things break.
Sure enough, my first owl shift was one of the worst I can ever remember. About 15 minutes before midnight the previous shift noticed that the jet distributions didn’t make sense. Miroslav, the outgoing captain, said with, not without some glee, “Hey. It’s your problem now!” And a problem it was. The calorimeter, which measures energy deposited by particles after a collision, clearly had a hot spot. In real life the collisions spread themselves out randomly, so over time you don’t expect any particular part of the calorimeter to get more energy than any other (this is a simplification). A hot spot is usually an indicatation that your calibration is wrong, or that there has been some sort of an electronics failure. But the hot spot region was huge, and didn’t seem to fit with any single hardware failure. In order to re-create the hot spot multiple electronics boards in the calorimeter would have had to fail at the same time: unlikely. We worked on it until almost 4am, and in the end had to declare defeat. We couldn’t find the source of the non-physical spikes we were seeing. We could only guess there was a new source of noise in the calorimeter. It would need more experts awake, and everyone needed to get some sleep and to think on the idea. We must have made calls to about 20 people (and woken most of them up). It was awful. By the way, the calorimeter was messed up enough that it is unlikely that any of the data I and the rest of the crew took from midnight to 8 am can be used for physics. If that turns out to be the case that is a large loss: we rarely loose more than a few minutes every 8 hours, and it is considered worthy of lengthy discussion in meetings if an hour or two is lost.
Between the end of that shift and my shift tonight (16 hours) the problem had been found and fixed. One of our Muon photo detector tubes was generating a huge amount of noise, which a particular region in the calorimeter was picking up. The process to find it was pretty painful, apparently. So, tonight, to make up for last night, I’m hoping we take some good data. Fingers are crossed!!!