Life as a Physicist

[Written two days ago, but was sitting on my computer unposted...]

Because I boarded the wrong train to Geneva today (I’m now going via Paris… duh!), I have been doing things on my portable that have long required attention — like cleaning out my contact list.

This has been depressing. I still think of myself as young, but I have a number of people in my list that have passed away - many before their time. What a shame!

This is the start of week I-have-no-idea without Internet. The latest news on that front is just too funny — I can’t wait until I have Internet so I can post what is going on! Otherwise, I seem to be down to about one blog posting a week. I miss posting!

When I was in India (I’d provide a link, but I’m writing this offline right now and can’t get a link to a previous post I wrote on this) one of the (many) things that struck me was the use of human labor. When a ditch-witch for several hours would do, they would hire 30 people for a week instead. This was a conscious choice (as it was explained to me): India has a huge population and this is one method they use to address unemployment.

France, and Europe, are by no means the same. However, I do notice that people are much more involved in day-to-day interactions than they are back in the USA. For example, to get a good train schedule it was much more efficient for me to talk to a person and have them book the ticket than it was for me to do it online or at one of the SNCF kiosks. In the USA sites like Expedia and Travelocity have almost made travel agents irrelevant for frequent travelers. But the human touch was nice - even if it took me much longer than it would have taken at one of their kiosks if it had enough power to arrange my schedule (why wouldn’t it route a trip to Geneva through Lyon!?).

Another example: my cell phone broke — it looks like the display cracked somehow. Turns out I can get it replaced under warrantee. Something similar has happened back in the USA — there Cingular over-night shipped me a new phone and then I shipped mine back. They said they would charge me full price if it looked like I’d dropped it. Here, in order to setup the warrantee exchange I had to go to a Orange store (see below for the Catch-22 reason why) and then to actually do the exchange a person will arrive at my apartment on Tuesday morning and do it in person. That is right — a real live person!! At first I was flattered that they would spend the time of a person to do something like that. They are willing to spend that kind of money on this. Neat! But then I realized the flip side: I have to be home between 9am and 1pm. I have no Internet; so I will loose basically half a day of work. I like my work; I don’t particularly want to miss half a day.

Doing business in the USA is definitely more impersonal than here in France. I suppose there are both ups and downs to that. I wonder how much more the French way costs? My cell service would be cheaper if they went the FedEx route, right?

Catch-22: So, your cell phone broken? Then from any Orange mobile dial 700 and you’ll get someone to help. Oh, but wait — their automated phone system tells you to hang up and dial from a fixed line. What if your house doesn’t have a fixed line (it is part of my Internet package… remember? I don’t have Internet). Great, call from work. Oh — that special number only works from fixed FranceTelecom lines. FT owns everything here so that shouldn’t be a problem… except, at least, for the lab where I work! So, after the Orange store told me to go home and dial these numbers, I had to return and tell them to dial them for me. If you want a double catch-22, imagine me, in broken french, trying to explain all of this to the Orange people in the store! And then they got to listen to me, in broken french, explain what was wrong with my phone to the people on the other end of the phone line. On the flip side: one came to my rescue by correcting the way I was using one word - which converted the situation from me having to buy a brand new mobile to getting a warrantee replacement. Thanks, Orange boutique people in Marseille (on Rue Paradis).

UPDATE: Dude showed up about 10 minutes before the end of the 4 hour window. Grubby teeshirt and shorts, with a box in hand. The box had clearly been shipped to him. The whole operation took less than 5 minutes. Why couldn’t they have just shipped the phone directly to me? Well, I have a new phone and it works just fine. Now, if only the Internet would start working too! I am just too demanding!

I’m sorry this blog has been so quiet and that my flickr account has been so quiet. The problem is I still do not have Internet at home! The time I spend at work is pretty much full of doing, uh, work. And work is too far away to come in late at night as well! I have given up trying to predict when we’ll get Internet: I’ve been so wrong. Suffice it to say, a college from UW has moved to Madrid for a year and she had Internet in 2 days!!! France is not doing so well…

I went out with some friends last night — and one of them was bidding on a photo-shoot for an add for VW cars. At the very end of the bid process the agency running the ad called up and asked “We are worried the carbon footprint for the ad shoot. Is there any way you can not use an RV?” The ad, I think, was for the European market. All I can say is: wow! If small things like this are done all across the industry and in everything that it does (like making the ads) — well, that is exactly what we need.

I’m spending this week in Victoria, Canada attending the Computers in High Energy Physics (CHEP) conference. At one time this was my favorite conference (more on that in a later post). These summaries are just things in the talks I found interesting on day 2. [I spent last week there -- I'm just getting around to cleaning up these posts!!!]

Plenary: The Future of GRID Computing - Miron Livny

Most people I talked to really liked this talk. Miron’s group at Wisconsin is the author of Condor, one of the more popular GRID aware free batch systems out there. We’ve made heavy usage of it back at UW to process ATLAS events. But I didn’t like his talk all that much - mostly because I don’t like GRID talks all that much. I got made fun of in a plenary session 2 or 3 CHEPs ago for complaining that we were talking about the GRID too much. I didn’t think it could get worse — but it has - even more of CHEP is devoted to computer science problems (getting jobs to run on the GRID) than it is on physics problems that can be solved with computers.

Miron started on a promising note: GRID computing is dead. What he meant was that you no longer got automatic funding by attaching the word GRID to a proposal. Then he said, “but distributed computing is here to stay. Lets get back to work and away from the hype.”  !!

He also has one slide in his talk where he says that we have to do better in job reliability. I’d like to point you to a NYTimes article that was published today talking about complex systems and how error prone they are. What irks me a bit is that the GRID folks have been stressing reliability for years now and their progress is, well, slow (at least to my eyes). New features and new tool kits seem to appear with alarming frequency — but no one goes back and makes sure that all the layers work well together.

He now helps lead the OSG consortium. This is a loosely affiliated group of labs and universities that have banded together to supply large computing resources to various branches of science (not just HEP). They have some umbrella funding, but its future is not obvious. I wonder if consortiums like OSG will last or will we live in a world of very large facilities like TerraGRID that we buy time on?

He also stated what would be a repeated theme: moving data files around is the single largest weak point of the GRID currently. He is right about that!

Plenary: High Performance Computing at the Petascale and Beyond (Blue Gene!!)- James Sexton (IBM)

James gave another fantastic vendor talk. It was interesting to see the big difference in IBM’s vision vs other company’s visions. Most of the discussion was of Blue Gene and how it fit into the future of computing. IBM positions it as a research machine that they can sell. They expect to develop new technologies here and then slowly feed it down into other product lines.

Blue Gene is a move away from commodity processors: they attempt to put everything on a single chip: memory controller, bus interface, etc. With as many processors as they plan to put in a rack they need absolute reliability. They are using a core based on the PPC (duh) running at only 800 MHz. They put 32 of these cores on a single card along with 10 Gigs of memory. This means each core has about 300 megs to call its own. They then pack these cards into racks — about 4000 cores per rack. They then have high speed interconnects between the cards and racks.

Livermore has 64 racks. Their mean time between failure is 7 days. All I can say is wow! Our L3 system runs about a week before a node fails and we have only about 1400 cores in that (small!) farm. 64 racks is 256,000 cores, or about 180 times bigger than our L3 trigger farm.

Applications that have run on Blue Gene include atomic and material simulations. The comment from the researchers was that these types of problems weren’t each to put into a BG, but it was possible. Protein folding also did some work — but say they need another x3 in compute power. Ouch!!!

James, along with many others, pointed to the fact that the cost of memory is not falling very quickly, so economics is going to force one into the Blue Gene model, or something that looks like it. Further disk storage and getting data in and out of disk storage is also not keeping up, leading one to wonder if it will be cheaper to re-derive results than to store the intermediate results?

His future predictions were pretty guessable: the 2-4 GHz max clock rate was here to stay, but be ready to move to using millions of cores. Memory bandwidth is going to be a problem, and reliability is going to become so much more important in these 1000 core systems.

Plenary: Canadian Cyberinfrastructure - Bill St. Arnaud (CANARIE)

Bill started this by saying that Canada was late to the GRID. This statement surprises me: I remember Canada being one of the first GRID sites to help D0 out with Monte Carlo processing. The CANARIE group is particularly strong in networking. Bill talked about several technologies that allow one to reconfigure networks on the fly. I’m not going to claim I understand this, nor do I understand why this is particularly attractive (except, perhaps, from a security point-of-view), but other talks at the conference made me think this scheme is going to be widely adopted.

One cool project they were working on, Neptune, was a large undersea network of remote sensors. A NYTimes article happened to be published the same day of the talk. Very neat!

Parallel: Booting ROOT with BOOT - Rene Bruin

The idea behind this coming improvement to ROOT is that most ROOT projects use a very small portion of ROOT. So why does all that memory have to be loaded? Why do all those files have to be distributed? Further, there are small bug fixes and it would be nice to distribute them without having to redistribute the whole of ROOT — and allow the user to update on the fly.

In order to reduce ROOT’s memory footprint, the ROOT team is doing what people have been asking it to do for the last 10 years: reduce interdependencies. A rumor has it that the straw that broke the camel’s back was Rene made a minor change to the TClass.h file and discovered that he had to recompile all of ROOT. This is fantastic! Reduced dependencies (and the ROOT team has discovered abstract classes in a big way! Wow!) should make ROOT a more stable product in the long run. Rene showed some data towards the end that had evidence that in the latest versions of ROOT the memory foot print had already been reduced — and, thus, the start up time had also been reduced (they are down from a peak heap size of about 30 megs to 4!). Fantastic!

Parallel: ROOT Graphics: Status and Future - Oliver Couet

Oliver described a bunch of new plots styles (Spider, Parallel, Box, and others). In particular, the Parallel plots looked like they might be interesting in particle physics. The spider and box plots will take me a while to get my head around in how they could be useful.

They have also shifted ROOT over to Open GL now, which has enabled some of the ALICE amazing graphics.

Parallel: ROOT I/O Performance and Functionality - Philippe Canal

Philippe is a very brave man. He spends all of his time down in the I/O sub-layer of ROOT. The ROOT team is usually listed as Fons and Rene — but I think Philippe must be getting close to deserving to be added to that list (if he isn’t already past threshold). The I/O subsystem must be a fairly thankless task, though it does contain some very cool problems that need to be solved without sacrificing speed, but it also has to be some of the most difficult and tedious work!

A lot of work has gone into storing data as compressed values — when you don’t need to save full precision. They are using a technique, it sounds like, that Scott Snyder used for D0’s I/O subsystem when he wrote that many moons ago. They have also improved the asynchronous read-ahead over a network — techniques to keep a data hungry ROOT application well fed despite network latencies. In anticipation of greater PROOF adoption, they have also improved code that concatenates results from several files into a single file. Another very cool thing: they can take an ASCII file and read it into a TTree without a user having to write any code! This will be very helpful for quick-and-dirty TTree building! There was also a cool sounding improvement to the TEventList (use TEntryList now) - a sparse and highly optimized way to store events that pass cuts.

I’m spending this week in Victoria, Canada attending the Computers in High Energy Physics (CHEP) conference. At one time this was my favorite conference (more on that in a later post). These summaries are just things in the talk I found interesting. [I spent last week there -- I'm just getting around to cleaning up these posts!!!]

The first day was both Plenary and parallel sessions. They are using Indigo which means everything can be seen online. I came in on the Clipper in the morning, so I missed the first two talks.

Plenary: LHC Computing - Ian Fisk

Ian talked mostly about HEP’s use of the GRID - that big computing appliance in the sky… err, I mean cloud. He showed a great slide that really tells you how much trouble we’ve gotten ourselves into - all those layers up layers in the GRID: “Not all of them included just for technical reasons.” We keep adding layers and adding layers to hide complexity. It is no wonder some of our systems are so fragile! He points out that everyone is developing their own front-end system interface to the GRID. He made the point (along with others before and I’m sure after)  that the level of complexity means that we have to develop more and better automated testing of the tools.

The ALICE experiment has the ability to make custom datasets based on simple event cuts. A user can define a particular set of event attributes and have a dataset made from that. I can’t help but wonder how that scales if every user requests their own dataset (which is very similar to the next user’s) - that is a lot of I/O. But it is good, none-the-less. I see this in ATLAS too with the TAG - but I’m nervous of the same scalability problems there too.

ALICE also has a clever way of sending jobs out to the GRID (I think this is old hat now): the pull model. They submit a little pilot job. It checks out the machine for basic installation and then calls back to a central scheduler to get the actual job that needs to be run. I’ve known this gives them the ability to avoid black holes, but I didn’t appreciate how much more job scheduling flexibility it gave them: some GRID submission queues can be very long!

In the first reference to multi-core (of many!) he pointed out that with the new machines we are using now we need 8 or 9 copies of the simulation or reconstruction running at once in order to fully utilize the capabilities of a 8 core machine.

In general he painted a rosy picture of the tools. This didn’t match my experience and I asked how things look so good but seem to not work for the end user. He mused that perhaps 90% operation was still pretty bad from an end user point of view. I think it is still worse. I got a bunch of ribbing for that comment later on in the conference.

Plenary: LHC DAQ - Sylvain Chapeland

This was the usual overview of the LHC DAQ. One thing impressed me was that every one is using ROOT to display plots. Java makes an appearance also, but ROOT is the real king here — any time a plot has to be made. Most of the operational displays shown were a few buttons and lots of histograms — I usually think of DAQ operations being a bunch of buttons and some text files. I think this was speaker bias.

He had a comment on the last slide — wondering if we couldn’t have a common DAQ system. Years ago I also wondered why we couldn’t do that. At the very least, a common DAQ system. Indeed, Fermilab has a project that does this for their fixed target experiments (I can’t remember the name). I think this will never happen for the large experiments like ATLAS or CMS. The hardware is so specialized, and the dataflow is so specialized. I can imagine that we might make use of libraries of utilities — but we’ll never see unification of the overall DAQ system or its control.

Plenary: Computer Facilities - Eng Lim Goh (sgi)

Unfortunately, the slides have not been posted. This talk was fascinating (as were all the vendor talks, actually). Goh noted that the data rate out of the ALICE DAQ system, 1.2 GByte/sec, was the same as what the film industry was seeing for their “4D” compression for a movie - so the real world is catching up!

He has seen 10,000 node rack & stack clusters close to production and hears people now talking about 100,000 node rack&stack clusters. Wow!

sgi’s vision of the future are very tightly packed racks. They do away with the pizza box stacking and instead just build CPU’s on a board, stuff the boards into a special high speed back plane, and build modules out of these. Input and output uses Infiniband, which gives performance on par with 10 Gb ethernet.

Supplying power was also a focus — and doing it efficiently. Apparently, one usually gets from line voltage via two steps. sgi has put it in a single step, and they have brought the power supply out to the back of the rack to supply a whole lot of nodes - thus they claim to have one of the most power efficient (watts/flop). Of course, Goh mentioned the backplane that they plug all their processors into now has 1000 amps coursing through it! Yikes!

Despite the efficiency improvements, they still have to cool the rack down - they use water cooling. Goh claims it is neutral — you don’t have to add extra AC to the room - the water chiller will take out at least as much heat as what goes in. Interestingly enough, the coolers are mounted on the back of the rack, after the air has flowed through the processor.

The solution is quite unique — in that it doesn’t play well with other solutions. This question was asked, and Goh said that in units of a single rack it did play well — sgi supplies standard GB interfaces in and out of a rack so that one’s whole data center doesn’t have to be infiniband.

Parallel: ATLAS Analysis Model - Amir Farbin

I managed to miss most of this (late back from getting a Dr. Pepper!), but his talk mentioned several things that looked interesting that I’m not already familiar with:

• SPyROOT - The specific example knows about data sets, relative luminosity, and cross sections and can automatically make plots for multiple datasets at once. Written in Python. The batch version seems to be a simplified version of what we use in D0 (in Python rather than C++). Cute. Not obvious it is robust enough - seems designed only for simple things — no way people are going to end up doing just simple things there!
• We will only simulate 20% of the data we take. I hear this said all the time. A particular analysis only looks at a small subset of the data — will this number matter?

He also talked about his EventView approach to doing analysis — which I think is not going to work out well.

Parallel: A Class To Make Combinations - Nobu Katayama

This was a very cool talk on an algorithm to solve a physics problem - a set of ideas you could walk away from CHEP with an immediately apply. I’d like to see more of this sort of thing!

Nobu was dealing with combinitorics in B-physics. Specifically, a 3 body decay. He wanted to represent it with C++ operator syntax: “D0 = Kaon*Pion*Kaon” — in order to do this you need to loop over all three objects at once, but C++ operator syntax means you will do the Pion*Kaon first, and then the Kaon*<result> second — not evaluate all three. So he wrote a delayed evaluation library: it builds up a syntax tree of all operations but doesn’t actually do them until the “=” sign is hit. At that time the whole parse tree is available and he can see that he needs to do a 3-way combination. Very cool! I’ve seen this technique used to solve all sorts of computational problems (like shipping matrix operations out to a GPU).

I asked why not use a Domain Specific Language: a text file that contains the physics in a more readable form that you could then use a program to automatically turn into C++: easy to read and write, and the speed of C++. He really wanted to complete flexibility of C++ available (for cuts, etc.). Sounds good!

Parallel: The CERN Analysis Facility - J. F. Grosse-Oetringhaus

ALICE is probably the heaviest user of ROOT of any of the ongoing large experiments. They use it for everything - reconstruction, simulation, visualization, etc. You name a feature of ROOT, and they are using it. This includes PROOF, the remote cluster version of ROOT. They currently have about 40 machines setup (500 is the goal). PROOF has, from the sounds of it, received significant debugging on this cluster. Listening to this talk you really get the idea that PROOF is big-iron. Lots of support is required to have a fully functioning system, which is too bad.

I asked how you distribute user code and libraries - you send source files and they are compiled on each PROOF machine that is running. What do users do when they want to run an analysis in PROOF with lots of plots - speaker didn’t know, but he did know that some people were making 100’s of plots in a single event loop (the only way to do it, in my opinion).

Parallel: Visualization with ROOT - Matevz TADEL

Wow. ROOT visualization is pretty impressive! The ALICE detector is a heavy-ion detector. This means when the ions collide there will be a huge mess of tracks in their detector. They are using the visualization code in ROOT as their detector display, and they claim decent performance when they show as many as 1000 tracks. And this is without using acceleration hardware (as far as I can tell). I wonder how fast things would go if they used modern graphics cards and did processing there?

Parallel: The Online Track

There were a lot of good talks on High Level Triggers at this point in the Online Track — but I have seen many of them before. Suffice it to say that good progress is being made and people are already at the point of testing the algorithms to make sure there is enough CPU to do the things that they need. In particular, one referred to a version of the track finding algorithm, called a Kalman Filter, that didn’t require matrix inversion (very slow). I’d not heard of this gain-matrix formalism before, but I found what looks to be a decent reference on the web.

I’ve spent this week in Victoria, Canada attending the Computers in High Energy Physics (CHEP) conference. At one time this was my favorite conference (more on that in a later post). I’ve been writing up day summaries, which I’ll post this weekend, but thought I should get my personal conference summary out before the actual conference summary just to see how it compares.

First, the vendor talks (ibm, sgi, intel) were some of the best talks given. They were given nice long talks in the plenary sessions and all the companies sent excellent speakers. I think these were the best vendor talks I’ve ever seen. It was fascinating to see the different takes each company had on the future of computing: ibm: lots slow cores, little memory, sgi: evolve the current technology to its limits: pack 1000’s of cores into a single rack, intel: 80 cores on a single chip, one or two big cores with all the bells and whistles and lots of little cores that run more slowly.

Computing Hardware

Power, Moore’s law, and Heat dominated the plenary sessions. Every computing facility is feeling the pain. No one has figured out how to solve this. For the near future we will be moving towards tricks — like Sun’s black box computing. Longer term some real change in technology.

Performance & The Multi-Core Future

It seems a given that as we head towards the multi-core future we will need to change the way we write code. The memory bandwidth in and out of a chip will increase more slowly than the number of flops that chip will be capable of — so less data per flop will be a problem! It isn’t obvious we can rewrite our code to run multi-threaded: that is a huge amount of work. Further, I think we don’t have enough data on processor performance to really understand if that would help (though it seems like it would).

To that end there was only one talk (that I saw) that really looked at performance of some of our large reconstruction programs and simulation programs. The result? On a CPU with a 500 GB/sec bus (or was it 50? I can’t remember) the reconstruction program of CMS is using only 40 MB/sec!! If that is true, we will have no trouble scaling up to 80 cores given the current memory bandwidth. Further, the CPU is idle of about 60% of the time (it can process 4 instructions at once, on average it is doing 1.2).

At the start of this conference I was convinced that we were going to have to alter our programming model. But now I think there is a lot of work we can do with our current installed base in the form of optimization. By the next CHEP I expect a lot more studies of this sort. It was sad that so many of us (myself included) talked without really having too much data. We may still have to alter how we approach things — but there is more to be gained in our current frameworks.

I also predict that people working on offline software will be now asked to move away from creating random objects all the time - in some places in the CMS code over 1 million news and deletes were occurring per second! Ack!!

GRID

It is hard for me to tell any difference year to year. But the consensus seems to be funding is “dead” and we need to get on with distributed computing. Oh, and, as in every year, stability is a must (sheesh). I think that with funding drying up organizations like OSG will be disfavored and large installations, centrally and professionally managed, like TerraGRID will become the norm. GRID software will still exist so people can “easily” run on these different large centers.

I also think, give the continuing addition of layers of complexity, that user analysis will not occur on the GRID. ROOT tuples (or something similar) will be produced, downloaded to your local 10 TB cluster, and then run locally.

ROOT

Resistance is no longer an option — we have all been assimilated. It is very nice to see ROOT finally ripping itself apart and putting itself back together in a more modular and separable way. What prompted this? Slow start up times and memory usage. Awesome! Lots of other efficiency improvements and I/O improvements are getting made as well.

I can’t tell how PROOF is coming along. There are now some real installations, but it hasn’t really started to spread. The problem is that at almost every CHEP this has been the case. Unfortunately, from my point of view, PROOF cluster design still is a big iron design. Hopefully it will get simpler as time passes.

Algorithms

There wasn’t enough of this at CHEP. I like CHEP because it straddles Physics and Computer Science. It feels more and more CS like, and less and less physics like. There were some interesting talks — for example integrating advanced separation techniques (like Decision Trees) into ROOT.

There was plenty more going on, but in the few minutes I had to dash this off this is what came to mind. It was a good conference (other than the lousy network access)! Food in Victoria is also really good!

Check that out. That is a real honest-to-goodness paper ticket! That ticket (was) worth real money. I could have gone and cashed it in for, uh, cash! Who knew those things still existed! Not I. So, you might wonder, how did I end up in possession of not one of those, but three!?

That is easy. We flew in Italy. Ok, that isn’t fair. The real reason is we flew on Al Italia. The day after they released a new business plan. A business plan that called for the laying off of some workers in Al Italia Express — the small airline partner to the big Al Italia. Indeed folks: a strike! Had all gone well, we would have been flying for a bit over 2 hours from Trieste to Marseille. Instead it was 15 and a half hours of busses, sitting in airports, waiting, lines, etc., before we finally made it home at 2 in the morning.

The day was full of little annecdotes. We met Marco, a computer programming consultant that was trying to go home for the weekend. After lots of delays and transfers, his last flight from Milan was canceled. Since he was going to have to come back for his job the next day he demanded Al Italia fly him back to where he started and refund the complete cost of the ticket - basically erasing the whole nightmare (I never found out if he got it).

In Trieste, we were put on a bus to Venice in hopes that things would be better. The trip is about 120 km, but in the middle of it he hit a traffic jam that went on forever! It was stop and go the whole way. No one (at least the Italians) seemed to care much. Paula is convinced that the real cause was space aliens, because my proffered explanation was too ridiculous to be true: a very poorly designed toll booth.

And after being put on a bus in Trieste they said “we can get you to Venice, and they can get you to Milan, but because of the strike there may not be a flight from Milan to Marseille). When we got to Venice the ticket agent said, “yes, I heard you had some problems in Trieste, but not here. You will be fine now.” As if the problems were only in Trieste.

And Paula called this beer the best beer she had ever had! That is saying something for Paula.