Life as a Physicist

The Ultimate Logbook January 8, 2011

I couldn’t leave this alone. I mentioned the ultimate logbook in my last posting. This is the logbook that would record everything you did and archive it.

It isn’t difficult. The web already has a perfect data format for this – Atom (or RSS). Just imagine. Each source code repository you commit to would publish a feed of all of your changes (with a time stamp, of course!) in the Atom format. Heck, your computer could keep track of what files you edited and publish a list of those too (many cloud storage services already do do this). Make a plot in ROOT? Sure! A feed could be published. Ran a batch job? The command you used for submission could be polished.

Then you need something central that is polling those RSS feeds with some frequency, gathering the data, and archiving it. Oh, and perhaps even making it available for easy use.

Actually, there is a service that does this already. Facebook. Sure! Just tell it about every RSS feed and it will suck that data in. Some of you are probably reading this on Facebook – and this posting got there because I told Facebook about this blog’s Atom feed and it sucked the data in.

Of course, having a write-only repository of everything you did is a little less than useful. You need a powerful search engine to bring the data you are interested in back out. Especially because a lot of that data is just a random command which contains no obvious indication of what you were working on (i.e. no meta-data).

And finally, at least for me, I don’t really want something that is static. Rarely is there a project that I’m finished with and I can neatly wrap it up and move on. Heck, there are projects I put down and pick up again many months later. This ultimate logbook doesn’t really support that.

Perhaps it is best to split the functions. Call this a ultimate logbook a daily log instead, and then keep separate bits of paper where you do your thinking… Awww heck, right back to where we started!

BTW, if you think Facebook might be interesting as a solution here, remember several things. First, as far as I can tell, there is no way to search your comments or posts. Second, you might get ‘Zuckenberged’ – that is, the privacy settings might get changed and your logbook might become totally public.

Log Book Follow-up January 5, 2011

Starting back in March I wrote a bunch of posts on logbooks: where do you keep your log book?, what do you keep in it? (and more of what you put in it). I can’t help it. The logbook is near and dear to my heart. I promised a follow-up posting. Finally… In summary (nothing in any particular order):

• What goes into a log book: pictures, code, text, screenscrapes, files, plots, handwriting, paper
• What do you use: Evernote, old style (bound notebook), loose paper, wiki/twiki, yojimbo, google wave, email (as in email a plot to yourself), tiddywiki, blogging software, text file, DEVON Think Personal, Journler (now defunct).

One thing I didn’t ask about but all of you contributed anyway was how the logbook got used (there is no right way – the logbook has to work for you, of course):

• Gave up – nothing but an inbox
• Just keep track of thinking
• Exploded: link services to track papers, paper for jotting down notes, email, etc. – a bit of everything
• Every last thing goes into the logbook, including bathroom breaks.

No one mentioned using a kindle/nook to read their logbook, btw. For software that gets used most like a logbook it looks to me like Evernote wins.

For me the most surprising method was email. And by surprising, I  mean smacking myself on the forehead because I’d not already thought of it. Here is the idea: just email your log book entries – with files and attachments, etc., to your logbook email account. Then use the power of search to recover whatever you want. And since you can stick it on Gmail or Hotmail or Yahoo mail, you have almost no size restrictions – and it is available wherever you happen to have a internet connection. Further, since it is just email, it is trivial to write scripts to capture data and ship it off to the logbook.

Now, I’ll ramble a bit in way of conclusion…

Do you remember MIcrosoft’s failed phone, the Kin? It was basically a smart phone w/out the apps. But one of the cool things it did was called Kin Studio. The point was this – everything you did on the phone was uploaded to the cloud. All the text messages you sent or received, all the pictures you took, etc. Then on the web you could look back at any time at what you did and have a complete record. Now, that is a logbook.

Of course, there are some problems with this. Who wants to look at lots of messages that say “ok!” or “ttl” or similar? And the same problem would occur if we were able to develop the equivalent of the Kin studio for logbooks. It would be a disaster. Which I think gets to the crux of what many of you were wrestling with in the comments of those posts (and something I wrestle with all the time): what do you put in a logbook!? There is a part of me that would like to capture everything – the ultimate logbook. Given todays software and technology this wouldn’t be very hard to write!

In thinking about this I came up with a few observations of my own behavior over the last few years:

One way to look at this is: what do you look up in a logbook? I have to say – what I look up in my logbook has undergone some dramatic changes since I was a graduate student. Back then we didn’t have the web (really) or search engines. As a result writing down exactly what I needed to do to get some bit of code working was very important. Now it is almost certain I can find a code sample on the web in one or two searches. So that doesn’t need to go into the logbook anymore. Plots still go in – but 90% of them are wrong. You know – you make the plot, think you are done, move on to the next step and in the process discover a mistake – so you go back and have to remake everything. And put the updated version of the plot into your logbook. Soon it becomes a waste of time – so you just auto-generate a directory with all the plots. So it always has the latest-and-greatest version. Hopefully you remember to put some of those into your logbook when you are done… but often not (at least me).

What is the oldest logbook entry you’ve ever gone back to? For me it was the top discovery – but that was nostalgia, not because I needed some bit of data. I rarely go back more than a few months. And, frankly, in this day and age, if you do an analysis that is published in January, by July someone (perhaps you) have redone it with more data and a better technique in July. You need those January numbers to compare – but you get them from an analysis note, not from your logbook! In short, the analysis note has become the “official” logbook of the experiment.

I have to say that my logbook current serves two functions: meeting notes and thinking. Meeting minutes are often not recorded – so keep a record. Especially since I’m using an electronic notebook I can mark things with an “action” flag and go back later to find out exactly what I need to do as a result of that meeting. The second heaviest use for me is brainstorming. Normally one might scribble ideas on some loose paper, perhaps leave them around for a day or two, come back refine them, etc. I use my logbook for that rather than loose paper.

Now a days I definitely do not keep a log book in the traditional way. Certainly not in the way I was taught to use a logbook in my undergraduate physics classes! Here is a quote from an ex-student of mine (in the comments of one of the previous posts – and I can copy this because he already has a job!!):

I have a rather haphazard attitude toward these things–I have a logbook, but I use it to remember things and occasionally to sort out and prioritize my thoughts. So it’s fairly sparse, and it certainly would be of no help in a patent dispute! Often I keep my old working areas around on my computer, and I use them if I forget what I did in my previous work.

This is pretty typical of what I see in people around me in the field. Other commenters made reference to more careful use of logbooks. I wonder how much usage style varies by field (medicine, physics (particle vs. condensed matter, theory vs. experiment), engineering, industry vs. academic, etc.)?

You put *WHAT* in there? March 16, 2010

It seems like I wrote the post asking folks what they used for their log book about a year ago. Ahem. Sorry.

The main goal of that post was to find out what people were actually using for a logbook. Looking over the responses and thinking a bit more, I decided that perhaps I should start off by a catalog of what people put in their logbooks first – before discussing what is out there. Actually, this might be a bit more interesting in the end than just a catalog of the stuff people do use.

Text/Prose

This is probably the most popular thing I saw referenced. I’m speaking of things that you can easily type in… thinking on a typewriterkeyboard, perhaps a summary of a conversation or a thought. From personal experience these can range from a short sentence to a long diatribe. I suppose basic formatting (similar to this blog) is nice – but for a logbook the main point seems to be to get an idea down so formatting isn’t so crucial. The other thing people mentioned liking is being able to quickly search for things – which text does very well.

A sub-category here, I suppose, are latex documents. Latex is the premiere mathematical typesetting language. Useful also for writing long books or papers. It has been mentioned to me more than once that when some new mathematical expression needs to be written into a logbook, the person will just latex it and then insert that in the logbook.

Pictures

I imagine many uses for this – especially if you are in a lab constructing things or perhaps inspecting something. As a result the logbook should have a simple capability to insert pictures from a camera. In general, what this turns into is insertion from a file (jpg, png, etc.). And if it to be easy on the computer it should be either automated or drag-and-drop.

This got me to thinking… if you had to do this often you could probably automate the process using something like the Eye-Fi SD card. You could take the pictures and by the time you got back to your office and had a cup of coffee, all the pictures you just took would be pasted into your logbook awaiting detailed annotation.

Ironically, the use I saw mentioned most in my post was to take pictures of handwritten notes. I’ve used this technique often to record a whiteboard or some paper scribbles – I can email it around to a student if we were discussing it or paste it into my logbook for later reference.

Screen Scrapes/Clippings & Plots

This was mentioned most commonly in conjunction with plots. You have your plotting program running one window and you now want that plot in your logbook. So you fire up your screen clipper (or perhaps it is part of your logbook) grab the plot, and then paste it into your logbook. Poof, the plot is in your logbook.

I’d like to point out two problems I have with this technique: when it comes time to make a presentation the screen clippings are rarely high enough quality for talks. The resolution is often small – and certainly not resizable. Second, the size of a screen capture of a plot is often quite a bit larger than the data in the plot (i.e. think about 10k or less for a histogram bin data vs the 100k or so for the image).  But for keeping track of some intermediate plots this sort of thing is just fine.

Storing the raw data of the plot would of course be ideal, however then displaying that is hard – if you are using a general program you would have to write a plug-in that would translate that data into the plot. Certainly possible, but…

Files

This was mentioned much less than I would have guessed in the responses to my blog posts. I find myself constantly dragging files into my logbook to keep the notes I’ve taken on a paper or something like that close to other similar ideas. The most common file types are probably PDF’s for me. But I’ve done other sorts of files, including Excel and small ROOT files.

Handwriting

Ok. For this you gotta remember who is talking here. I’m Mr. Tablet and I don’t think I could part with the ability to write things. However, I claim as long as one is living in a world where one wants to write down this sort of thing:

Now, I’m totally sympathetic to the idea that one can take a picture of a whiteboard (as I’ve done here). But if you have to do this sort of thing with any regularity – and perhaps on your own – being able to write it down is much better.

The other great use for writing is thinking (a few of you mentioned this in your comments to my post). Free flow drawing, arrows, etc. does work. However, as my wife has pointed out on numerous occasions when I bring this up – most people learn how to think on a keyboard now and they don’t suffer for it!

I did a quick scan of my log book to look where I had handwriting:

1. Markup of papers/notes I’m reviewing for physics correctness (i.e. I’m not going to send my markup back to the authors as much as I’m going to send a question like “If you make this cut here you are biasing your mass distribution here… please convince me you did this right.”
2. Free thinking. Probably about 50% of this in my own notes could have been done on a typewriter – it is just handwriting organized line-by-line
3. Class lectures and class lecture note preparation.
4. Note taking during a meeting. This is not mostly something that could be done by typewriter – I tend to cut/paste in clips of slides that are being presented and then scribble notes to myself over them.
5. Working through some problem involving mostly thinking. Say, an analysis problem. I’d say this follows the same 50% rule.
6. Working through some problem involving a plot(s). What I mean here is if I’m trying to think through some issue that is driven by a plot. Essentially, as soon as I paste a picture into my logbook I draw on it and my reasoning flows from there. So for me this is very much a handwriting thing.

So, I guess #1 and #3 aren’t strictly log-book functions. For #2 and #4-6 I find having pen input invaluable. If you eliminate #1 and #3 I think my logbook is about 20% handwriting content. If you keep those in it is closer to 50%.

YMMV.

So… what did I miss?

Ok – next time I’ll go through some of the ideas people had. Some were very cool – I’d never thought of them; now that I have I could see how they would make a good alternative to what I’m doing. I hope that the gap between this post and that won’t be as long!

Where is your logbook? January 24, 2010

This is old school:

What is new-school?

Some friends and I had a fairly heated discussion concerning logbooks the other day. As anyone who has been through a college (or high school) level lab-based science course knows, the importance of a logbook has been… well, drilled into your head. Use it to keep track of your experiments. It might be important in a patent dispute (this came up a lot during my high school lab courses for some reason), etc.

But now, in the modern age of computers and smart phones everywhere… I’m curious: where do you keep your logbook? In paper? An e-log? Do you keep plots in it? Do you even need a log book any longer? Just text files on a computer? What is the point of a log book now (as far as you are concerned)? Or any other log-book related thoughs. Please – dump them here.

Then I’ll collate them and tell you my opinion. And I’m sure you can guess that I have an opinion… 🙂