Generating headings for project files

I use the free software AutoHotkey (AHK) for automation of many things. One of those things is generating headings for my project file. I developed a simple AHK-script that asks me for the project name, the topic and then inserts the heading with additional information like my name, my E-mail address and a keyword for subversion.

The script is quite simple and I have adopted it for headings for several project files (tex, R and gams). The only difference being the characters used to comment out a line.

Here is the script

The line with ::tplt:: shows the characters that when typed start the template script (in this case for a tex-file). The lines with “Inputbox” produce a box where I can enter information

message

In this case, I enter the project name. What follows is the input box for the subject and then some additional information. Note, that the command “send” needs a variable (between %-values), therefore, everything  without variables should be on the same line.

Here the result for a new gams project file:

gams

You can easily adapt this script for your own use.

Automatic operator formatting mode in Emacs

Emacs has a nice new mode called electric-operator developed by David Spepherd. It helps when you write code by formatting all operators in a predefined way. For example, in R it adds spaces around the operator signs, when you write 1+1, this is automatically converted into 1 + 1.

You can define your own way of formatting for other modes. For gams-mode you add the following to your .emacs file

First you set gams-mode as a mode on which electric operator can work.Then you add some rules. After that you set a hook so electric-operator is automatically started when you use Gams (or ESS).

A special formatting is necessary for the comments in GAMS as they start with the same symbol as the multiplication. With the help of David, I could add a function gams-mode-* that changes the “*” used as a comment in “* “ and otherwise in “ * “.  It checks if the “*” symbol is at the beginning of the line and in that case will not add a space in front of it.

It probably needs some more tweaking, and I will keep you posted on improvements.

Column guide in Emacs

Many professional editors have a column guide. For example, below you see in the Powershell editor a veritcal line at column 80.

image

This marker comes in handy, if you want to print your files. In Emacs I usually check for the column 80 in the status line.

image

However, I usually forget to check, and when I print out stuff or publish, I got line breaks where I don’t want them. Passing the 80th column for comments is not a problem, as a simple Ctrl-q forces Emacs to break the lines nicely and add comment symbols at the beginning.

image

But after hitting Ctrl-q, Emacs produces this:

image

There is, however, a package, that mimics column guide seen in other editors. The package is called “fill-column-indicator”. Installing this package will do the trick. After installing, you only have to add the following lines to your .emacs file:

image

I added two hooks, so the column rule appears automatically in Gams and ess.

More information can be found here: http://www.emacswiki.org/emacs/FillColumnIndicator

Using file templates in Emacs

A good practice in modeling is to place information related to the project your are working on, the subject, the data and your personal information like E-mail address at the top of your file. In Emacs you can define skeleton functions that will prompt for the information and put it in the file. Below is an example for a file with as comment symbol “**: (the $Id: $ is for my version control system, that automatically will add the information on the last commit in this line).

image

Below you see how to make a function using skeleton. As I don’t want to repeat myself, the function asks as an input the comment symbol(s) that should be used. I now can use the function for different programming languages (R uses “#” , C++ uses two forward slashes, GAMS “*”, etc.). You can add the time and date by using the command “current-time-string”. All other inputs that have skeleton-read in front of it, are also prompted for. If you need the template for your whole project team, you could just make the author and contact field also input-dependent.

image 

More information can be found at: SkeletonMode. (note that this is part of Emacs since version 22).

PS. The function puts an extra comment symbol at the end (not shown). This is a bug in my function. Any ideas how to get rid of this are welcome.

Automating writing LaTeX papers even further

In an earlier post I wrote on how to use R to produce all kind of figures and tables for a LaTeX paper. This time, I will show, how to automate this process even more.

My work flow consists of a batch file, that runs my model, sends the results to R, produces my tables and figures and sends these to my paper: If I change my model, I just have to run this batch file, and the results in my paper are automatically adjusted. One smaller problem is that I often write in my text on a specific value from a table (for example, the minimum time used). Every time I change my model, I have to check these entries by hand (of course, I could use knitr, but that is on my list for the future and at the moment, I want to keep my R-files separate from my LaTeX files).

Here is a neat trick, to automate this process. Suppose I have calculated the reduction in time in my model and exported the results to R. In my paper I have the following  sentence:

“As you can see in Table 1,in the scenario “Cruising” the reduction in driving time is 12.4%.”

In my R-script, I have the R-Code that produces this value or takes the value from my model results directly. Now I use some simple R-commands to produce a LaTeX file that can be included in my paper:

In my LaTeX-paper, I now just change the sentence to:

This produces the same sentence as before. However, if I change my model, the minimal reduction is changed and automatically adjusted.

Migrating to R 3.0.0: TikzDevice and gdxrrw

I am an avid user of R. Most of the time, I use R for preparing nice plots and tables for my LaTeX papers. For my plots, I use TikzDevice and for my tables I use xtable. Gams, my modeling software, developed a nice interface for using data from Gams in R and the other way around (see http://support.gams.com/doku.php?id=gdxrrw:interfacing_gams_and_r). When I upgraded from R 2.15 to R 3.0, I had some problems on my windows 8 machine, especially with TikZDevice. I finally managed to get it installed. The necessary trick is described on “The Stata Things” (!!!, http://enoriver.net/index.php/2013/04/12/keeping-knitr-happy-after-upgrading-to-r-3-0-0/ )
Here is what you have to do, to get TikzDevice up and running:
  • Install R 3.0
  • Update your packages, using the command: update.packages(checkBuilt=TRUE)
  • Install Rtools (http://cran.r-project.org/bin/windows/Rtools/) and adjust your path (sytem variable PATH) by adding the bin-directory of Rtools (something like C:\Program Files\Rtools\bin)
  • Download tikzdevice from http://r-forge.r-project.org/R/?group_id=440
  • Run the following commands in the R Console
  • remove.packages(“tikzDevice”)
  • install.packages(“Path to where the file on your computer is/tikzDevice_0.6.3.tar.gz”, repos = NULL, type=”source”)
For Gams users: download the gdxrrw 0.3.0 (not 0.2.0) from the homepage (http://support.gams.com/doku.php?id=gdxrrw:interfacing_gams_and_r ). Extract the files in your R library and add the following line to your Rprofile.site (in the directory etc of your R-installation), where you have to set the gams directory properly.
.First <- function() {
     library(gdxrrw)
     a <- R.Version()$arch
     if (a == “x86_64”)  igdx(“C:/program files/gams_64/23.9.4”)
     else                igdx(“C:/program files/gams_32/23.9.4”)
   }
PS. I like the fortunes package. It produces nice quotations when you start up R. Just install the package and add the following lines to your Rprofile.site
   if (interactive())
   fortunes::fortune()
Here an example
I wish to perform brain surgery this afternoon at 4pm and don’t know where to start. My background is the history of great statistician sports legends but I am willing to learn. I know there are courses and numerous books on brain surgery but I don’t have the time for those. Please
direct me to the appropriate HowTos, and be on standby for solving any problem I may encounter while in the operating room. Some of you might ask for specifics of the case, but that would require my following the posting guide and spending even more time than I am already taking to
write this note.
  — I. Ben Fooled (aka Frank Harrell)
     R-help (April 1, 2005)

Viewing all the objects in R in a buffer in Emacs

You also probably did not read the complete Emacs-Speaks-Statistics manual… ? By chance, I discovered a nice feature, called Rdired.
This command gives you a dired-buffer with all your objects. In this buffer you can view, edit and plot your R-objects. It is documented in chapter 12.9 of the manual.
 
Here is small example: In the buffer in the middle, you see the original code taken from the manual. I invoked ess-rdired and the lower buffer is shown. I then put the cursor on the line with the parameter s and hit “v”. This gave me the upper buffer with all the values. Then I hit the “p” and a nice plot is shown. With “d” you could delete the object.
 
 
 
 
You have to put the following lines in your .emacs file:

Using Gams, R and LaTeX I

Wow, I just discovered how easy it is to produce nice LaTeX tables from your Gams results. Usually I capture my model results in some parameters that are exported (using gdx or gdxxrw) to excel. Then I make my tables in Excel and use an excel add-in to export the table to LaTeX (I have written about this add-in in this blog). This works fine, but is tedious because as soon as my results are in excel, I can’t do things automatically and have to click my way to get my LaTeX tables.
Today, I was working on the parking model and wanted to automate my LaTeX tables. I already wrote on how to produce nice plots using R Statistics, but that might have been too geeky for a lot of you (“I don’t do network modelling and Excel or Gnuplot produces plots that will do…”). However, what I show you today is worth trying to start working with R if you are doing your modeling with Gams.
My workflow now looks like this:
  1. Run my model and produce some parameter with results and export the parameters to a gdx file
  2. Read the parameters from the gdx file with R using the R package (gdxrrw).
  3. Recast the data using “cast” and “mold” from the package reshape (see for an excellent introduction (http://www.jstatsoft.org/v21/i12/paper)
  4. Use the R package “xtable” (see http://cran.r-project.org/web/packages/xtable/xtable.pdf) to produce for each table a separate tex file.
  5. Write my paper and include the tex files with tables.
Let us have a look at the code for every step.
1. Run my model and produce some parameter with results  and export the parameters to a gdx file
Most of this is straight forward. Note that I save my gdx files in my R directory. In this directory I have my R-scripts.
2. Read the parameters from the gdx file with R using the R package (gdxrrw).
Gams Corporation is working on a R-package that allows you to transfer data from R to Gams and the other way around. The package is called gdxrrw (the “rrw” stands for R read and write)  is still work-in-progress, but you can already use it (just download it from (http://support.gams.com/doku.php?id=gdxrrw:interfacing_gams_and_r).
When you have it installed, you can use it as follows:
The gams parameter is now a R data frame with the following structure:
3. Recast the data using “cast” and “mold” from the package reshape
I want to produce two different tables: one for the model statistisc and one for the road results. I saved all this results in one parameter, so now I have to take a subset of the results for every table.
I defined a subset “modelstatistics” and use them to subset my dataframe using %in%. This gives us the following dataframe
I want to have a table with the parameter as columns and the scenarios as rows. For this I first rename the columns and then I use the cast command:
If I combine this with the xtable command, I already have my LaTeX table:
4. Use the R package “xtable” to produce for each table a separate tex file.
Now we are almost ready: I set some options for saving the LaTeX table (table title, label and filename) and save the file using the print command on the table:
5. Write my paper and include the tex files with tables.
I jump in my paper tex file and add the following line for including the table:
and produce my LaTeX file:
In your xtable-code in the R-script you can a lot of additonal options to format your table.
The great thing is that I can do it all now without having to adjust my tables every time I run a different or addtional scenario. I just add in gams the following line to run the R-script:
And for the Emacs fans: I didn’t have to leave Emacs at all: I run my Gams model, my R code and my LaTeX from within Emacs (and for those with the good eyes: I can even clock this task…)
In a next post, I will write more on the use of cast: you can use this R command for lots of interesting stuff (building tables with the sum, mean, min, max, etc.).

R-Graphics for Networks

With R you can make beautiful graphics and the nice thing is, you can design your own customized graphics. For some nice graphs produced with R, have a look at http://addictedtor.free.fr/graphiques/.
 
At the moment I am working on a parking model (searching for parking lots in the city) and have a simple grid for the streets and the junctions. If you start working with network models you have the problem that there is basic information for every node and on top of that you have for every node your results. Trying to interprete the results is hard. Either you have a huge excel sheet with lots of numbers for every node or you try to catch specific results in your model by filtering the output and looking at your model results listing.
 
R gives you the possibility to draw your network with all the information near the nodes. I first wrote all the numbers near the nodes with different colors, then I used arrows in different colors and different sizes. That was all nice, but still hard to read.
What I really wanted was to have something like a piechart or bars for every node directly plotted on the grid.
 
In R this is possible. You can use the x- and y-coordinates of your nodes and the drawing function polygon(). With some basic geometry (wihich I had to look up….) and ideas from Bálint Czúcz (http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=143) ,
I managed to write a R-script that takes the number of results per nodes (in my case 3, but you can have as many as you want), the grid size (in my case 5×5, but you can also use a real grid for a specific citiy). The script calculates the relative sizes of the results or basic information (for example number of old people, number of young people, number of adults = (0.3, 0.2,0.5)) and the relative size of the total (in node a we have the most people, in node b we have only 50%) and uses this information to draw scaled piecharts.
 
In the nex figure you see the results. You  now can easily see that in this case mose people live in the node with the coordinates (4,6) and that every group has about the same size.
 
 
Another way would be using bars:
 
 
Here is the most important of the code (pie chart)
 
 
 

Template Mode for Emacs

If you write a lot of R-, Stata-, Gams and/or LaTeX-files templates can make your life more relaxed. Emacs doesn’t have a template system (well there is auto-insert, but I did not find enough information on how to use if for templates). I used to have some templates in a directory and usually I would start with them (and forget that I should save it under another name…). The disadvantage is that you have to jump around in the file to fill out the information you want to have at the top of the file (like project name, topic, file name etc.). Today I discovered templatemode (see http://emacs-template.sourceforge.net/details.html). It works like a breeze and makes my writing life much easier.
 
I have templates for the above mentioned file types. The Gams template looks like this:
 
Now if I start a new file, TemplateMode asks me if I want to use the Gams-template, it inserts all the information between >>> and <<< or asks for information to insert. For example DIR, DATE will grab the directory and the date. At the end of the template file you see pairs of words. These words define the interactive part. Templatemode asks me for the project name with Project:, the job with Job: and the Gams title with Title:.
 
The line with the Version Control Information grabs information from my verison control system as soon as I commit the file (and is not part of templatemode).
 
This is how a new Gams file would look like:
 
 
Installation is easy and you can find lots of examples in the directory.