jump to navigation

A Very Bad Sign September 17, 2006

Posted by gordonwatts in physics.

Very Bad SignI flew down to Chicago for the weekend to take some owl shifts. When I arrived in the Control Room on Friday afternoon, this is what I saw.

Look at how many people are there. And 10 minutes before, when I ducked out to get my camera, there were almost twice as many. DZERO needs 5 people in the control room to run. There is significantly more than that there. That can only mean one thing: experts. And what do experts do? They mess with the system. They try to “fix” things. What happens then? Well, things break.

Sure enough, my first owl shift was one of the worst I can ever remember. About 15 minutes before midnight the previous shift noticed that the jet distributions didn’t make sense. Miroslav, the outgoing captain, said with, not without some glee, “Hey. It’s your problem now!” And a problem it was. The calorimeter, which measures energy deposited by particles after a collision, clearly had a hot spot. In real life the collisions spread themselves out randomly, so over time you don’t expect any particular part of the calorimeter to get more energy than any other (this is a simplification). A hot spot is usually an indicatation that your calibration is wrong, or that there has been some sort of an electronics failure. But the hot spot region was huge, and didn’t seem to fit with any single hardware failure. In order to re-create the hot spot multiple electronics boards in the calorimeter would have had to fail at the same time: unlikely. We worked on it until almost 4am, and in the end had to declare defeat. We couldn’t find the source of the non-physical spikes we were seeing. We could only guess there was a new source of noise in the calorimeter. It would need more experts awake, and everyone needed to get some sleep and to think on the idea. We must have made calls to about 20 people (and woken most of them up). It was awful. By the way, the calorimeter was messed up enough that it is unlikely that any of the data I and the rest of the crew took from midnight to 8 am can be used for physics. If that turns out to be the case that is a large loss: we rarely loose more than a few minutes every 8 hours, and it is considered worthy of lengthy discussion in meetings if an hour or two is lost.


Between the end of that shift and my shift tonight (16 hours) the problem had been found and fixed. One of our Muon photo detector tubes was generating a huge amount of noise, which a particular region in the calorimeter was picking up. The process to find it was pretty painful, apparently. So, tonight, to make up for last night, I’m hoping we take some good data. Fingers are crossed!!!


1. Aaron F. - September 17, 2006

Uuugh! What a nightmare! Do the experts mess with the hardware, or just the software?

p.s. Is the normal crew really just 5 people? The computer screens would outnumber them 7 to one!

2. Expert on Shift « Life as a Physicist - September 18, 2006

[…] Hmmm. I’ve just been chastised by another one of the shifters here. I’ve already talked about the ill wind lots of experts in the control room brings. Well. I forgot. I’m an expert of the data acquisition system and trigger. And guess what I’m doing tonight? Trying to improve it. And guess what is happening? I’m causing downtime (I hope only small bits). […]

3. Summer Just About Done « Life as a Physicist - September 20, 2006

[…] Wow. Classes at UW start up on September 27th. I return to Seattle on this Sunday after two months in Ottawa minus two trips. I can’t believe it is almost done. I can’t believe I have a 2 month old baby! […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: