Life as a Physicist

For those of you out in the real world: you use Word or something similar to write all of your documents. In Physics we use two typesetting packages, TeX. Unlike Word, TeX uses text input files to fully describe the document being produced. This is not WYSIWYG!!

TeX’s inception was in 1977 (the first released version was much after that). Many people in the field of physics will disagree with me when I say that the output that TeX produces is ugly and hard to read (partly because of the font family it uses), and is quite hard to use - especially if you are an infrequent user.

For example, I am editor of a note in ATLAS that needs an internal and external version. The internal version is the same as the external version along with a bunch of extra material. I needed the equivalent of an if statement to turn on and off sections of text depending on what version of the note I was producing. In Word it would take me about 5 minutes to create a new style that would do this - because of the user interface almost all the options are basically “in your face.” When it came to TeX, however, I spent about 2 hours searching the web and trying to create my own commands to do the job before I finally stumbled on a good web page. What a waste of time!

That said, before people in physics think I’ve totally lost it, there are two things I think that TeX does better than word: handles long documents (like a thesis) better than Word, and deals with figure placement better than Word.

TeX is a macro language - built to solve a specific set of typesetting problems. It’s macro language is amazingly flexible and gives you access to almost all parts of the layout engine from its code. But its power is also its weakness for someone like me: I almost never need advanced features - so I can never remember what or how to access them. And, in fact, the TeX macros are so low level there are numerous packages on top of it (LaTeX is probably the most popular). Word exposes a full object model that you can program against in easily in any dynamic language (like the .NET version of python or VB) and with a little more difficulty, C++. However, my impression is for really complex typesetting jobs TeX is a bit easier to deal with. Of course, that is just it: as a physicist I rarely, if ever, need or want that level of control. And that is just it: exposing all of that means the TeX macro language is prohibitively hard to use.

Well, I suffered all day, so you will too (well, only as long as it takes you to read this post). My new portable spent most of the day masquerading as a door stop. It was doing a darn good impression too.

It was me being stupid. I installed a recent OS update despite warnings to the contrary (that this very thing might happen). But I saw several others on the net had done it, and they owned the same portable. No prooooblem!

Er… no. I guess there are different versions of the same portable out there. Whatever, my computer would never get beyond a blank screen in the boot process. Of course, after it happened I started reading about it.

Recovery was pretty painless, considering I didn’t have a backup. It is very cool we’ve gotten to the point I can boot my portable off a USB key - there really is no longer a need for a DVDROM reader in my portable!

At any rate, the lesson is: when the vendor says “don’t install, might cause bad problems,” listen — especially the night before you will be depending on it! Yes, I have gone to school. Perhaps I’ll learn sometime soon!

I’ve mentioned several times on this blog that the lack of new features in C++ makes me nervous. I’ve not seen Java grow much either, but that may have more to do with the blogs I look at. I saw this on Gosling’s blog, the creator of Java (and many other things — like emacs):

Java feels alive, not stuck in some chisel marks on stone tablets. Closures were left out of Java initially more because of time pressures than anything else. Closures, as a concept, are tried and true - well past the days of being PhD topics. The arguments are in the details, not the broad concepts.

I am very happy to see this sort of thing. Programming languages should be living things.

Closures are a very cool concept, btw. They come straight from functional programming, and they make it very easy to design call-backs in complex systems. Basically, I can write the following (in no particular syntax of any language that exists today - that I am aware of):

int x = 0;

vector<int> junk;

… fill the vector of int’s junk with some values…

junk.foreach (doit (int val) { x = x + val; });

I define this inner function “doit”, which is passed as an argument to the foreach method of the vector junk. The foreach method calls “doit” for each value. And do it adds to x the value generating the sum. Anyone that has used STL and the std::for_each algorithm function knows the kind of pain it causes: if you have to maintain any state between each call you end up having to create an object. In C++ that is a lot of syntatic overhead. Closures almost totally eliminate that. I’ve used the C# language, which has closures, and writing this sort of code just flows from your finger tips - in C++ you have to fight to get the code to come out, your source files become huge, and you generate all these objects you never reuse.

Maybe someday soon C++ will have this as well. That would be sweet and would make STL a whole lot more powerful — and most importantly — easy to program.

A few days ago I had a post about extending objects - the desire to add little utility methods to objects that others had defined (and control). A few of the comments triggered something way back in the back of my head.

We (particle physics) have solved this sort of problem already. Anyone remember FLECS or RATFOR (holy cow, look at the original authors of RATFOR)? (there are others). I personally used FLECS when I was on the AMY experiment. We wanted some C-like structures in our FORTRAN so we used FLECS to add them — curly brackets, etc. The FLECS preprocessor was an extra step in the compilation process - it produced F77 code that was fed to the real compiler.

I’ve seen almost nothing like that for C++. Certainly, it is a much more complex language, so it is much harder to extend in this way, but it must be possible. Are there such languages? Any extension methods there?

P. Canal, a ROOT developer, pointed out to me that CINT supports extension methods in much the same way that python does - you can just add methods to your objects. Of course, in the case of CINT, this requires you to have the class loaded into CINT and you have to call it through CINT.

Sometimes, when coding up a particle selection, you end up with code that looks like the following:

if (abs(jet.Eta()) < 2.0 && jet.Pt() > 10.0 && jet.EMF()<0.90) {
  good_jets.push_back(jet);
}

That is pseudo code, of course. The jet object is some collaboration defined object — not something you can really alter. But that selection can get pretty messy and long (especially for things like electrons) and you want to share it with others, so you write a small subroutine so you can write

if (ismykindofjet(jet)) {
  good_jets.push_back(jet);
}

Promotes reuse, factorization — all sorts of good things. I would like to see C++ add a bit of extra syntactic sugar (which is so _not_ C++ like):

if (jet.ismykindofjet()) {
  good_jets.push_back(jet);
}

It just looks so much cleaner. In other languages these are called extension methods. They don’t give you access to the private variables of the class - but they do make the code look cleaner. At least, they would make my code look a lot cleaner!

I suppose it is too late to get something like this into the next round of C++.

My new Lenovo X61t is doing great. One thing that is so much better than my old m200 is the CPU. My Skype calls actually work now. I can do video. And I can video conference at full bandwidth - and still use my computer for other things. My previous portable barely had the CPU to keep up. And you know what is even better? When on battery and power saver mode, it still has muscle left over - my old portable would just roll over and didn’t even have the spare cycles to laugh at me.

One thing I did this time, btw, was choose a slow processor — the 1.6 GHz dual core CPU (my m200 was a 1.96 GHz Pentium M). This plus modern electronics means my battery will last for almost 4 hours under full usage. That is great for the size and weight and power of this portable. Two years from now I’ll probably buy a new one - wonder what the battery life will be then? And how many cores? And what OS will I be running?

It isn’t without its problems, however.

Climbing on my soap box…

My mother has used computers for 30 years - or something close to that. Windows almost exclusively. At work she used it mostly for word processing and writing books - she was an English professor at Rutgers. She can’t spell either - the difference between her and me? She made sure she learned how to spell.

At any rate — she is comfortable with computers, but not a power user. Now that she has retired she has taken up birding and photography in a big way. There are amazing pictures all around our house - she enters contests as well. My sister and I and Dad are all working on her to get a flickr (or similar) account. You’ll be impressed (if they are ever posted online).

She just got the new D300 last week. She takes pictures in RAW format and the program she uses - Adobe’s Photoshop Elements — needed an update in order to read new new D300 RAW format. Adobe already has this update available. Cool!

Except… First thing she did was download the file. She then looked at the web site for installation instructions and saw the link to a PDF file. Only, when she clicked on it it brought up the warning “Are you sure you want to run acrobat reader? This program has known incompatibilities with Vista!” I have no idea what program installed that old version of Acrobat reader on her brand spanking new Dell Vista portable, but Mom decided that no, she didn’t want to run something that might break things. So, she proceeded without installation instructions. As it turns out, the installation instructions are both in the PDF file and also down at the bottom of the HTML page (why do this?? Why have a special PDF link when HTML will work??).

The downloaded ZIP file contained only a “Cammera Raw.8bi” file. No readme file. Why not include the readme file in that!?? So my mother tried double click it. Guess what — it opens the photo shop elements picture editor! And in doing so it doesn’t install the new Camera Raw file! What is the point of that other than to really confuse people!? Why not include a double-click to install file? That is 15 lines of open source XML code to do that!

Finally, she tried copying it into the Photoshop Elements directory (she didn’t realize how close she was!). Of course, that didn’t work. At that point she called me. We eventually got the file into the Plug-ins\File Formats directory. I really don’t know why Adobe made her do it - it was definitely not something she was comfortable doing, and Adobe’s Photoshop Elements product is aimed at the casual computer user, not the power user. Adobe is a big company - they produce some excellent products. They can do better when it comes to things like this.

Ok, getting off my soapbox.

How many of you use the 1960’s email reader, pine? This trusty old goat of a IMAP mail reader is everywhere. It is written by my home institution, University of Washington. I think (!?) the history is it was written as a test client to the original IMAP server - IMAP was built at UW. But somehow it became the default mail reader. If someone is using a text mail reader, they are using pine.

I hate it. The reason is that I can’t send embeded plots, hard to format things properly… Modern mail readers are much nicer from that point of view. People used to complain bitterly if I accidentally sent around a HTML formatted email message to a mailing list - “turn your Microsoft crap off!”. Now, I don’t get complaints any longer if I do it - what with everyone using modern Linux mail programs and also the mass injection of Mac’s. At any rate — I’m very happy to see this change.

I’m continually amazed by how many people use pine. Really. Modern GUI’s are so much more powerful for managing the 100’s of messages we get a day. But pine survives and, as far as I can tell, thrives. I don’t get it.

Of course, my opinion hasn’t stopped UW from moving forward. UW got a grant to rewrite it. The next version will be called Alpine. Actually, it is more an evolution of pine, but still.

I complain about ROOT a lot. However, one thing I have to say is that when you report a bug it is only a matter of days before it gets fixed in their source code repository. Philippe Canal just did a whole bunch of work after I found a few bugs in the MakeProxy code that means it will now be possible to use weird variables like vector<vector<double> > without having to think (these are produced by the EventView tool in ATLAS). Nice!

BTW, MakeProxy is a step back in time. Remember PAW (ok, I know, some of you are still using it). One of the nice things about the FORTRAN macros back then was the auto-loading of ntuple columns. Only what you referenced in your FORTRAN code was loaded. That isn’t the case in ROOT — it has no good way to scan compiled C++ code to determine what is used. Philippe wrote an automatic proxy-generator that replaces each ntuple branch with a bit of code that first makes sure the data is read in before accessing it (I’m pretty sure Philippe was the progenitor of the idea). In the end this has the effect of doing the same thing — only what is needed is read. I’ve seen x10 speed-ups caused by this. The only it can’t do is look inside a custom compiled object — so this doesn’t buy you as much if you are running in that case.

Otherwise, I definitely recommend checking out TTree::MakeProxy.

[Note: I wrote this post a long time ago; I'm not sure why it stayed in my drafts folder. But comments in response to this post on the java VM and Microsoft CLR made me realize I'd not posted it yet - though I thought I had!!]

I’ve been curious for a long time how much of a CPU hit do you take if you run your physics software in a Virtual Machine as opposed to the bare machine. I’m sure anyone who has used VM’s knows you take a hit — they always feel more sluggish. My impression is that for anything disk related they can be as much as 50% slower, but if it is just CPU it is just 5% or 10% slower. So, where does a real physics program stand in that?

To test this I and Todd, who works with our group back at UW, ran a few quick tests. We simulated about 20 ATLAS events. All of this was done on the same physical CPU. In all cases the ATLAS software was running on Scientific Linux 4 (basically a Red Hat clone). In no cases of the VM’s was any hardware vitalization going on.

** We also ran tests with VMWare, but I can’t find the email with the results now. However, I remember them being almost exactly the same as Virtual PC. The same hardware was used for all the tests.

So… running a VM will force you to buy 25% more machines to get the same work done. When you are running on a farm of 1000 CPUs that can be significant. On the other hand, using the approach that CoLinux does means you don’t loose nearly as much!

One reason we looked into this is that UW has a very large pool of Windows machines that sit in student Lab’s. Most of the time they are idle — so why not configure them to run simulation jobs during off hours? Of course, we can’t install Linux on them - so we thought about using VM’s instead. We did a pilot run with about 30 machines running the Virtual PC version, and produced about 40,000 events fairly quickly. It was very useful in allowing us to work with the machines we had.

Unfortunately, we had several BSODs installing CoLinux - but never one installing the VMWare or Virtual PC. And if one is running on Lab machines that are used by students one can not afford to have a BSOD: the lab manager will kick you off those machines so fast your head will spin! The other thing about colinux was that it was significantly more difficult to configure than any of the VM’s. However, just like the VM’s, once you got a disk up and running you could distribute that everywhere.