jump to navigation

The Cost Of Free GRID Access June 13, 2008

Posted by gordonwatts in computers, physics, science, university.

I was giving some thought to the health of our department at the University of Washington the other day. Cheap and readily available computing power means new types of physics simulations can be tackled that have never been done before. Think of it like weather forecasting – the more computer power brought to bear the better the models are at predicting reality. Not only are the old style models better, we can try new weather models and make predictions that were never possible with the previous versions. The same thing is happening in Physics. Techniques and levels of detail we never though possible are now tackled on a regular basis. NSF and DOE both have programs specifically designed to fund these sorts of endeavors.

This means there is a growing need for a physics department to have a strong connection to a large computing resource – in house or otherwise – in order for its faculty members to be able to participate in these cutting edge research topics.

Particle physics is no stranger to these sorts of large-scale computing requirements. In ATLAS, our current reconstruction programs take over 15 seconds per event — we expect to collect 200 events per second – we would need a farm of 200*15=3000 CPUs just to keep pace. And that says nothing about the ability to reprocess and the huge number of Monte Carlo events we must simulate (over 2 minutes per event). And then we have to do this over and over again as we refine our analysis strategy. Oh, and lets not forget analyzing the data either!

However, even though may of us are located at universities, we don’t make heavy use of local clusters. I think there are two reasons. First the small one: the jobs we run are different from most simulation tasks run by other physicist. Their research values high bandwidth communication between CPU’s (i.e. Lattice QCD calculations) and requires little memory per-processor. Ours does not need the communication bandwidth but needs a huge amount of memory per processor (2 GB and growing).

The second reason is more important – we HEP folks get access to a large international GRID for “free”. This GRID is tailor made for our needs – we drove much of the design of it actually. We saw a need for this more than a decade ago, and have been working on getting it built and working smoothly ever since. While we still have a way to go towards smooth operation, it does serve almost all of our needs well. And to a university group like ourselves at the University of Washington, cheaply. By function of being a member of the ATLAS or D0 collaboration, I get a security certificate that allows me to submit large batch jobs to the GRID. An example of the power: it took us weeks to simulate 40,000 events locally. When we submitted it to the GRID we had back 100,000 events in less than a week.

Given that us HEP’rs would rather spend money on a modest size local analysis system – which is quite small compared to what the rest of the physics department needs. And so we don’t really participate in these large systems in our local department. I wonder if there is a hidden cost to that. Could we gain something but moving more of our processing back locally?  Could you more easily convince the NSF to fund a physics compute cluster that was doing Lattice QCD, HEP simulation and analysis, and Astro simulations? Or would they get pissed off because we weren’t using the large centers they are already funding instead? Has anyone tried a proposal like that before?



1. tim head - June 15, 2008

How difficult is it to submit jobs to the grid and what kind of jobs can you submit? Could you in principle submit anything from a bash script to a statically compiled executable?

As you can tell from my question I don’t know nothing about the GRID ;]]

Do you have some good pointers towards some how-tos and guide? Feel like I should find out as I am starting a PhD in particle phys this September.


2. gordonwatts - June 15, 2008

Tim — that is great. Good luck! At what university?

Basically, think of it as a very large batch machine. And on most of them you can run scripts, executables, whatever. You’ll almost always want to do it with a script, however, because you have to setup disk access, etc. Fortunately — most experiments provide a lot of infrastructure so that you as an analyzer don’t have to worry about that.

Every experiment has its own rules and ways to access the GRID. So you’ll have to wait until you are setup to to find out. Do you know what experiment you’ll be working on?

3. tim head - June 16, 2008

I will be at Manchester university in England.

Scripts sounds good, I am an enthusiastic python user and would prefer to stick with it instead of having to deal with c++, root on its own is tedious enough.

Regarding experiment, I applied with atlas in mind but the way Manchester does things means the final decision will only be made in September.

4. MIke M - June 23, 2008

At MIT they are working to morph the cms-T2 center into a full scale, off-campus facility that incorporates computing needs of other departments. It doesn’t include lattice, but it does include quite a few HE(N)P experiments, as well as earth/planetary sciences. My limited understanding is that other departments coming online was critical. However, for cpu intensive simulation-like applications, I continue to wonder if cloud solutions aren’t more efficient and economical. Granted the universities generally provide power/cooling, but I wonder if that will continue indefinitely, not to mention the “cost” of hiring PhD physicists to run grid centers…

5. gordonwatts - June 23, 2008

Mike– for overflow I think it is much cheper to go cloud (Amazon, Google’s current offering isn’t good for what we do). But I suspect it is cheaper to the grant to use local resources when possible. Perhaps the right attitude is a mix. And hopefully the infrastructure is designed do you only have to prep one image or something like that.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: