Generating headings for project files

I use the free software AutoHotkey (AHK) for automation of many things. One of those things is generating headings for my project file. I developed a simple AHK-script that asks me for the project name, the topic and then inserts the heading with additional information like my name, my E-mail address and a keyword for subversion.

The script is quite simple and I have adopted it for headings for several project files (tex, R and gams). The only difference being the characters used to comment out a line.

Here is the script

The line with ::tplt:: shows the characters that when typed start the template script (in this case for a tex-file). The lines with “Inputbox” produce a box where I can enter information

message

In this case, I enter the project name. What follows is the input box for the subject and then some additional information. Note, that the command “send” needs a variable (between %-values), therefore, everything  without variables should be on the same line.

Here the result for a new gams project file:

gams

You can easily adapt this script for your own use.

Backing up your subversion repository on a remote Windows server using batch files

My subversion repository is on a server by Webfaction (probably the best and cheapest way to host your repository. I have 100 Gigabyte space). I used to have a backup script that would save the backup on the same server, which then was transferred to my Windows 8 computer. Because I wanted to replace the complete backup with a new one, I ran into problems, as the backup procress was taking to much of the server CPU. Support pointed me to svnradmin, which is a tool comming with the command tools of tortoisesvn (otherwise you will find these tools on Collabnet).

I searched the net and found some nice scripts to run this process automatically. I have two batch files. The first one runs the complete backup which is run once, the second one is a daily backup, that checks if there are new revsions and then starts to backup these.

The first batch file (fullbackup.bat) looks like this:

I agree: this looks rather cryptic. The first line takes the first 3 items from the output of the date/t DOS-command is something like: 16.07.2015 (depending on your settings, this might be differt). The for command now reads this and separates this information in three parts (i,j,k) where the delimiter is given as “.”.  This information is assigned to the variable “datesvn” and will be stored as 20150716. The second line does the same kind of trick with the time. The third line gets the actual revision on my computer. I than set the filename where everything should be saved to the directory the batch file resides (in this case, it would be something like “d:\svnbackup\SVN_20150716_1305_rev1_3995.dmp”. The next line does the actual dump using svnrdump. Now I have a full backup I can build on. The last line saves the number of the last revision to a file. This whole process can take a couple of hours, if you have a big repository. You can also adjust this script, so you split the backup for revisions ranges like 1:1000, 1001:2000, etc.

The second batch file will check if there are new revisions and if necessary start an incremental backup:

It looks for the last revison in the text file, checks if there are new revisions and then runs the same kind of procedure as before. This batch file can be run on a regular basis using the task scheduler from Windows.

Using Org-mode to keep track of your project files

For every project I work on I have a org-file. Every file has the following structure:

  • Tasks
  • Repeat
  • Wait
  • Notes
  • Calls
  • Reading

I use remember-mode to capture tasks, notes and calls. Under the heading “Repeat” are tasks that are repeated on a regularly base. Under “Wait” are reminders for things I wait for (like a reply on an e-mail, an order, a call, etc.). These waits are also captured using org-remember (I have written about that in an earlier post). Under the heading “Reading” I have links to papers I am currently reading (using Bibtex; see an earlier post on that).

Sometimes it is nice to have information on the files in one of the project directories, if possible with links. This can be done with org-fstree. Just add (require ‘org-fstree) in your .emacs-file after installing. You now can add the following line somewhere in your org-project file:

#+BEGIN_FSTREE: d:/inbox/Gams

This will give me all the files in the directory d:/inbox/Gams with a direct link (this is my folder for answering questions in the Gams forums).

image

You can update this by putting the cursor in this line and hitting C-c C-c. The documentation gives you information on the options:

“#+BEGIN_FSTREE: <dir> :<optionname1> <optionvalue1> :<optionname2> <optionvalue2> …

Options are:

  • :non-recursive t , to suppress recursion into directories
  • :exclude-regexp-name <list of regexp strings> , exclude file/directory names matching either of the given regexp expressions
    • :exclude-regexp-name (“.*\\.pdf$” “.*\\.zip$”), excludes files/directories ending with either “.pdf” or “.zip”
    • :exclude-regexp-name (“^\\.git$“) , excludes files/directories named “.git”
    • – :exclude-regexp-fullpath <list of regexp strings>, same as :exclude-regexp-name but matches absolute path to file/directory
  • :relative-links t , generates relative instead of absolute links
  • :show-only-matches t , only files that are being linked to show up
  • :only-directories t , only directories are listed
  • :only-regular-files t , only regular files are listed
  • :dynamic-update t , [EXPERIMENTAL] dynamically update a subtree on visibility cycling.
  • :links-as-properties t, sets the links as properties Link1, Link2,… for use in column view [Does not work with dynamic-update!]
  • :no-annotations t, suppresses the search and display of file annotations

Managing papers with Org-Mode and Reftex

Writing papers means reading lots of papers, and I often lose track of all the papers I am reading: Where did I file a certain paper? Which paper did I start reading? Which paper might be interesting?

A reading list would be nice, but such a list should be easy to handle. I have all my papers catalogued in JabRef and use AucTex and RefTex for writing and Org-Mode for my tasks.The solution should therefore combine the strength of all three. I found a nice post on how to this (http://tincman.wordpress.com/2011/01/04/research-paper-management-with-emacs-org-mode-and-reftex/) and adopted it to my needs.

Here is how my solution looks like (for managing tasks with Org-Mode see one of my previous posts)

image

  1. For every paper I am writing I have an org-file with my tasks, reading list, notes, and calls.
  2. I have four lists for my reading:
    1. Actual reading
    2. Should read: papers I glanced through and that look promising
    3. Not yet read: lower priority papers to be read
    4. Already read
  3. I insert new papers by using C-c ) (as in LaTeX/AucTex/RefTex). More on that later. Automatically, a link to this paper and the title are inserted (I save all papers as pdf-file and named with the first author name and the year of publication. They are all saved in the same dropbox directory). With “C-c C-c” I can tag the entries for easy searching later on.
  4. If I want to work on my paper, I open the org-file and, if I want to read a paper, I just click on its link and start reading.
  5. If necessary I can make short notes directly under the link.

Org-Mode makes it possible to use RefTex. For this it is necessary to put the information on where the bib-file is (third line in the next figure)

image 

The .emacs file has to be adjusted as follows:

Note: as I can make the reading to a TODO or a MIT (most important task), it will also show up in my daily agenda and I can also start reading from here by clicking on the link:

image

Scripts for automating commits and updates with Subversion

Version control is good, but there are two possible pitfalls for me:
  1. In the morning, it can happen that I start working without updating my project. If this is a project I worked on, on  another computer the day before, as soon as I want to commit the new changes, I will have to merge all the changes in one file. Not a big issue, but unnessary work.
  2. Before I leave the office, I unfortunately often forget to commit.
 
I wrote a commit batch file that runs every two hours,  but the problem was that I manually had to define in which directories the batch file should look and, if there was nothing to commit, I still had to click the commit window away (if you have about 30 open projects, that isn’t much fun).
I decided to improve my batch file: It should automatically generate a list of all the root directories of projects under version control and then check if there is something to commit. If there is nothing to commit, the batch file should just continue without showing me the commit window.
 
One hour of surfing and my rudimentary powershell knowledge resulted in two small scripts that check for projects should be committed and should be updated. In order to run these scripts, you need a Windows computer and the command line utilities for Subversion (they are usually installed if you use TortoiseSVN). I think that a script like this can also be written in other scripting languages and for other operating systems.
If you want to run it on your computer, you will have to adjust the abolute paths in the scripts to your settings.
 
 
The commit script (svncommit.ps1):
 
 
Remarks:
  • Line 1: Changing to the root directory of my data drive with all my projects
  • Line 2: Here we look for .svn folders (subversion puts them in the root directory of a project. Powershell looks in all folders if there is a subfolder with this name (it continues if it encounters system directories) and saves the path of these directories in the file svn.txt.
  • Line 5-14: In the for loop, I loop through every project root directory, check the subversion status, write the result once again to a file (status.txt) and check  if it is not empty: if it is not empty, it does a commit.
  • I save the files svn.txt and status.txt in a directory that is not under version control, so it is not around when I check for changes in the project directory.
  • This script runs every two hours on my computers.
 
The update script (svnupdate.ps1) is also short and simple
 
 
 
It hast the same first two lines as the commit script. With the subversion update option /closeonend:2, only projects that are not up-to-date will be updated.
 

Todo's from Everywhere

I use Emacs for taking notes and project management using Remember-Mode and Org-Mode (see my blog entries Capturing and More. However, if I am working in Excel or reading a pdf , More and I want to make a note or I suddenly have a thought about a feature that I want to add to my model, I have to jump to Emacs,hit some keys, enter the note or the task and navigate back to the program I was working in.
Not much of a hassle, but inefficient as it really interrupts my work flow too much.
No problem, if you use some macro scripting tool (for example winautomation or autohotkey) it is rather easy.
Here is the macro in Winautomation:
In step 1 the macro saves the location of the active window (for example Excel). In the next step it jumps to Emacs, followed by sending the keys C-c r  which asks me for a template:
I then choose the template (for example t for ToDo) and enter the information. I quit with C-c C-c and my entry is saved. Then I hit Ctrl-g (which is the Emacs command for “break” and in this case does nothing and if I change my mind and don’t want to save the note or tasks it stops the remember mode). This keystroke combination is used as a trigger for the Winautomation macro to return to the macro. In the last step the focus jumps back to the program I was working with.
The macro is started when I hit the hokey “Ctrl-Shift+F11”, which I defined in Winautomation.
As I have the professional version of Winautomation, I can generate an executable and send it to you. Just mail me at renger(at)modelworks.ch.
The only thing you have to do, is add a keyboard trigger like Win+…
For this you can use free programs like winkey (on Win32)  or HotkeyBind (I did not test these programs).

OrgMode on my iPhone: MobileOrg and Dropbox

In one of my last year posts I wrote about using Org-Mode for organizing your projects and your notes/todo’s with links to your files. I am now using org-mode for lots of things but especially for my todo’s. Now this works fine if my notebook is up and running, but sometimes I want to have a look at those org-files without firing up my notebook. I use an iPhone and Richard Moreland developed a nice app (see http://mobileorg.ncogni.to/) that works together with dropbox (http://www.dropbox.com/).
 
 
It has some nice features:
  • Support for Dropbox or WebDAV as a transfer mechanism
  • Org files are stored offline and available anywhere
  • Complete editing support, including editable headings, body text, todo state, tags and priority.
  • Note-taking facility built in, including offline support
  • Full-text search, or search by keyword (TODO, DONE, WAITING) or by tag (errand, office)
  • Mark nodes as done, or flag them with an optional note for reference later
  • Document view of your outlines, for reading document-style Org files with long body text
  • Powerful support for linking between Org files

Here are some printscreens from my iPhone (by the way: printscreens can be produced by pushing the on/off and home button at the same time).
 
IMG_0045.PNG (320x480 pixels)IMG_0046.PNG (320x480 pixels)
IMG_0047.PNG (320x480 pixels)IMG_0048.PNG (320x480 pixels)
 

More on project management with Org

In one of my posts I talked about org-mode and how I use remember to organize my tasks, calls, etc. For every project I have such an org-file. I now added a file projects.org with all my projects for my different jobs. Here you see my projects for Ecoplan:
 
 
The project entry in this list is linked to my org-file for that project. In Emacs you can add a link by hitting C-u C-c C-l. It asks you for the path to the file and a description. The advantage of this file is that I now have an overview of my active projects and can quickly jump back and forth to organize my day.
 
Another thing I added in my org-files for a specific project is a list with file names. Because I work on a lot of projects I usually forget after a few days the content of my files. Below you see my active files for the project P-EVE. I used Org to make a table and quickly add every new file I work on with a short description. Of course: the file name in the table is linked to the file itself.
 
 

Capturing todo’s and notes in Emacs

Often when I work on a model or a statistic problem in Emacs I make a note that I have to correct or add something to the code whenever I have the time for it. I used to work with Outlook or with a piece of paper, but both have disadvantages. Writing on a piece of paper or with outlook disrupts my work flow and I have to write down to which part of my work the note relates. Emacs can be a great tool for keeping track of notes or todo’s and links them directly to the file you are working on. This is done with remember-mode developed by John Wiegley
http://www.emacswiki.org/emacs/RememberMode). I use remember-mode together with org-mode (http://www.emacswiki.org/emacs/OrgMode; the manual of org-mode chapter 9 gives a good introduction in how to use remember-mode). Both modes are part of the standard Emacs version.
Let us have a look at an example. I am working on a model in Gams and know that I have to document a part of the code, but don’t want to do it right away. I hit C-c r and Emacs shows me a menu with the following options:
I can select one of the templates I defined. I choose a “todo” which Emacs will link to the file and the place in the file I am at the moment. Emacs opens a new buffer which looks like this
I just can start typing (red arrow) and describe what I still have to do. The blue arrow is the link to the file where I am working on. I can add additional notes or tags. The green stuff gives you more information, but will not be part of the final todo. After filling out the template I hit C-c C-c and the todo is stored in the org-file “refile.org”. This is a file where all my notes, todo’s, calls etc. are temporarily stored. At the end of the day or in the morning I check this file and move the items to other files. I have for example a file phd.org with all my notes and todo’s for my phd-work.
Here is a list with todo’s in this file:
The first todo is opened and you can see the link to the file I was working on when I captured this todo. Each morning when I want to work on my phd-project I scroll through my todo-list and choose the todo’s I like to work on for that day and send them to my agenda (more on that in a later post).
I have also templates for ideas, notes, calls and journal items. The set-up in my emacs file looks like this:
(define-key global-map “\C-c\ r” ‘org-remember)
(setq org-remember-templates
‘((“Todo” ?t “* TODO %?\n %i\n %a” “d:/daten/org/refile.org” “Tasks”)
(“TodoOL” ?q “* TODO %?\n %i\n ” “d:/daten/org/refile.org” “Tasks”)
(“Note” ?n “* %^{Title}\n %i\n %a” “d:/daten/org/refile.org” “New Ideas”)
(“Idea” ?i “* %^{Title}\n %i\n %a” “d:/daten/org/refile.org” “New Notes”)
(“Call” ?h “* %U %?\n\n %i  \n ” “d:/daten/org/refile.org” “Calls”)
(“Journal” ?j “* %U  %?\n\n %i\n ” “d:/daten/org/journal.org” “Journal”)))
I defined a key for capturing (C-c r) and added several templates. Take a look at the org-manual for all the options (%,? {Title}, etc.).  The tools for building templates is quite versatile and using them makes your life much easier.