Emacs and Stata

Although I am not a great fan of multi-tasking I often use a lot of programs simultaneously for one task: For example, when I am writing a report in Word or with Emacs, I take notes in Emacs, search the internet and have my model results in the gdx-viewer.  Additionally my mail client is somewhere in the background and nowadays I have a messenger program open to talk to my collaborators. I don”t like this jumping from one program to another, but it is often necessary. This is also one of the reasons I switched to Emacs, because I can do my main work in Emacs (modelling, note taking, statistics and report writing).
For my statistical work I use R, SPSS, Eviews and Stata. Emacs has a nice mode for working with R (ESS: Emacs-Speaks-Statistics). You can edit your scripts and  run it  in R without leaving Emacs. ESS has highlighting and lots of other features. ESS supports Stata too, but you can”t run your Stata commands in Emacs. So you have to copy-paste from Emacs to Stata. The editor of Stata (I use version 10) is  a plain text editor without syntax highlighting and almost no other features (version 11 has a much better editor but the update is too expensive).
So what can you do, if you still want to use Emacs (or any other text editor other than the Stata editor)? Use a lot of keyboard strokes switching back and fort from one program to another is one option, but here you run into other problems: if you have multi line commands copy-paste doesn’t work for Stata as you have to strip off the linefeeds. OK, you can write your commands on one line, but then you have to adjust your output if you want to document your scripts nicely. Another more practical way is using autohotkey, a free, open-source utility for Windows
(there are similar programs for Unix or Mac). Here are the great features of Autohotkey. With Autohotkey, you can (taken from the manual):
  • Automate almost anything by sending keystrokes and mouse clicks. You can write macros by hand or use the macro recorder.
  • Create hotkeys for keyboard, joystick, and mouse. Virtually any key, button, or combination can become a hotkey.
  • Expand abbreviations as you type them. For example, typing “btw” can automatically produce “by the way”.
  • Create custom data-entry forms, user interfaces, and menu bars. See GUI for details.
  • Remap keys and buttons on your keyboard, joystick, and mouse.
  • Respond to signals from hand-held remote controls via the WinLIRC client script.
  • Convert any script into an EXE file that can be run on computers that don”t have AutoHotkey installed.
For the problem described above I wrote a small script and linked it to the key combination Windows+s.
The script looks like this:

#s::                               ; the short cut (Windows key + s
clipboard =                      ; Empty the clipboard.
Send !w                          ; Copy the highlighted region in Emacs (Alt-w)
; Replace /// (necessary in Stata for multiline commands) and line feeds:
StringReplace, clipboard, clipboard, ///rn,  , All
WinActivate, Stata/SE       ; Activate the Stata window
Send ^v                           ; Copy the clipboard (Ctrl-v)
Send {Enter}                     ; Execute the commands
; Jump back to Emacs (Renger”s Emacs Mode is my Emacs window title)
WinActivate, Renger”s Emacs Mode      
return

If I mark a region in my Stata script in Emacs and hit Windows+s, the line is copied to the clipboard, all line end feeds (only in multi line commands) are stripped off and the edited region is copied to Stata and executed. After the execution the cursor jumps back to Emacs. This with just one key stroke!
I use Autohotkey for lot of other tasks (one recently was taking a sample from a telephone directory and copying the addresses to an Excel sheet). Another nice feature I implemented is automating copy-paste from a pdf-file I am reading to a text file with my notes. Once again: one click and the text is in my notes.

Golden Rule 2: Comments and Descriptions

The next golden rule might also be obvious:Always add comments to your work and label your sets, variables, parameters and equations
Now this might be obvious if you have a lot of time, but usually the rule is forgotten if you are stressed and want to finish what you are working on. Not writing comments will probably cost you a lot of time, if you take your model a year from now and see your code: Wow, that looks interesting and very ingenious, but what the %*?!  did I do here? If you send your model to somebody else for help, this person usually don”t want to dechiffer your code. Comments make her life easier too and the chance you get an answer will increase.
In Gams you have many options to add comments:
  • Start a line with a “*” and write your comment, like
* This is a comment
You can change the character “*” in a different character with this command
$comment #
# This is a comment with the new character
  • Put your comment between $ontext and $offtext:
  • Put your comment after some code using  the end-of-line comment possibility after the two characters “!!”
C = IOT(”H”,”CONS”)   !! This is the consumption initialization
Note that this possibility of setting comments is turned off by default. You have to activate it by inserting $oneolcom (on end of line comment)  or $inlinecom (inline comment)
You can also change the default setting of “!!” to two other characters by using $eolcom
Gams-mode for Emacs has a nice feature that allows you to hide multi-line comments (just use Ctrl-c Ctrl-h to hide or show the comments). Hiding the comments gives you a good view on your code
Before hiding:
After hiding:
Writing comments is good practice.
Adding descriptions to your parameters, variables and equations is also good practice. If you use short variable or parameter names, this might save your time, but once again, will make life hard if you look at your model after some time or if your model is big. Was “y” domestic production level or was it income?
If you use descriptions your output is easier to read:
compared with:
Gams-mode for Emacs has another nice feature: if you put your cursor on a parameter or variable in your code and hit F7, another window opens and shows you the description.

Golden Rule 1: Display and use your intuition

The first golden rule for modeling is: Always display the results of your calculations and check them with your expectations about the results.
Often you are working on a model doing a lot of preparing calculations (calibrating the model, preparing the data). Those calculations might be easy, for example calculating the shares for some parameter, but if you make an unnoticed error can cost you a lot of time and frustration.
Here is a simple example. I want to calculate the shares for the investment function
 
 
Calculating the share is not difficult thing, but you can make small mistakes. When I teach I often see people write things like
 
      shareinv(s) = inv(s)/sum(as, inv(as));

There is only a small mistake: instead of summing over inv(as,”inv”)  people sum inv(s,”inv”).
 
Gams in this case just multiplies the value three times instead of summing over all the elements
 
The best way to guard yourself against errors like this is by displaying the results of your calculations. Just add a display statement and a $exit  on the next line (the last command stops gams even if there is code after the $exit. Don”t forget to write the $ of $exit in the first column).
 
But not only should you display your results, you should before that ask yourself what the result of your calculations will be. In this case you expect values between 0 and 1. Thinking about the expected results before you see the results is perhaps one of the most important things in modeling.
Perhaps this sounds all a little bit exaggerated, but mistakes are easily made and if you don’t check your results, you can loose a lot of time.
I have seen models with negative values for output, shares bigger than 1 and so on. Displaying your results and checking them with your intuition doesn’t take much time and can help you find bugs in an early stage of modeling.

How to update Gams and keeping an older version

Gams is often updated and sometimes it can be good to have the old version still around (especially if you are not sure if the new version is working properly or if you are trying the beta version and want to switch back to the old version).
What I usually do is the following:
  1. I rename the directory of my installed version to GamsOld
  2. Install the new version (directory “Gams”)
  3. That is almost all.

The advantages ares:

  1. first of all, I still have the older version around (I usually keep one older version and delete the others).
  2. As I work a lot using the command line I do not have to adjust my path environmental variable every time I update my gams installation. The path where my computer remains “c:\program files\gams.”
  3. if I want to run my model with an older version of Gams I only have to rename the directory where my new version resides into something like c:\program files\gamsnew and rename the directory of the older version back to c:\program files\gams. No other adjustments are necessary. The path is still correct and the link to the Gamside is also still working. After working with the older version I just rename the directories back to their previous names.
There is only one smaller problem: include files and other files I installed will not be available in the newer version. For this  I use a batch file that copies those files from the old version to the new version. The batch files contains commands like
Copy  c:\program files\gamsold\inclib\plot.gms   c:\program files\gams\inclib\plot.gms

Guidelines for Posting to the Gams Mailing List

The Gams mailing list can be very helpful. There are a lot of people who are willing to use their free time to answer your questions. My advice: subscribe to the mailing list at http://www.listserv.dfn.de/cgi-bin/wa?SUBED1=gams-l&A=1 ( and even if you don’t post to it, you can learn a lot of the questions and answers by others. If you post: don’t be afraid to ask a stupid question. The worst thing that can happen is that you get an answer like I once got. I asked a really stupid answer and got the answer: “Think”!

Tom Rutherford posted some nice guidelines for writing to the Gams mailing list. They might be useful for any mailing list you use. Here are the guidelines:

You will get more mileage out of the GAMS list if you adhere to a few guidelines (nothing official here, only my unsolicited advice):

1. Always take time to compose a clear statement of the problem from first principles.. Often times, the process of clearly explaining the problem is sufficient impetus for finding your own logical error.

2. If you have a problem specifically related to GAMS syntax or execution, write down the minimal operational GAMS program which illustrates your problem. Minimize the number of symbols you use in this program, but leave no descriptive field blank. Add comments to the program to the extent that they are needed to explain what you are doing.

3. If you are working on something more complex, write up the mathematics in a PDF document which explains what you are trying to accomplish.

4. If possible, upload a zip file containing the GAMS code and documentation to a web site (these are freely available everywhere these days my daughters tell me), and send a nice short message to the GAMS list, briefly indicating your field/problem type and an operational link to your zip file. (My policy, whenever posting a zip file, is to be sure that I am able to download and unzip the file on my own machine before asking someone else to try).

The advantage of using a zip file is that you can include all of the data files and perhaps a restart file which can save someone lots of time. The idea here is to provide the support staff with a “smoking gun”. We need to see precisely where the error occurs, and we need to be able to reproduce it on our own platform if there is to be a high probability of providing assistance.

Outline Mode in Gams-Mode for Emacs

Let’s get to know the great things you can do with Gams-mode. In the previous post I showed how Gams-mode nicely colors your code, the syntax-highlighting. In this post we will look at the great way Gams-mode shows the results of your model run. This is done in the Gams-Outline mode.
First run your model by hitting C-c C-s (C stands for Ctrl) or use the same command from the menu. Emacs will open a new window and you see how Gams solves your model. When Gams is finished you can hit F10 (or C-c C-v)  to show the listing file in a new buffer. We will however assume that the model solved without errors and we are interested in the results. Of course: you can scroll through your listing file using the search commandos from Emacs. Gams-mode however has a great way of showing all your results in a nice outline mode. This mode also allows you to jump from one result (for example the output of a variable to the definition of a set). Instead of jumping to the listing file, we therefore jump with F11 to the outline-view of Gams-mode.
You can also jump to the outline-mode from your gams file by hitting C-c Tab or hitting C-c C-ii (this command cycles through gams-, listing- and outline-mode).
As soon as you are in outline-mode you will see that the menu bar will have changed and the menu item GAMS-OUTLINE appears. Here you will find all the commands related to the outline-mode (with the short-cuts after the command name):
You will see a buffer with a grouped list of your variables, sets, parameters, etc. The next picture shows you the outline-mode for the library file abel.gms
First notice, that the information is grouped and colors are used to make a distinction between the different groups.
If you put your cursor on one of the items, for example on PAR w, your screen is split in two halfs: the upper part is still the list with all the items. The lower buffer shows the part of the listing file with the output for the chosen item. You can jump to the next item by pressing the key “n” (for next) and to the previous by pressing “p” (yes: for previous…).
Often you want to have a look at two items at the same time: just press the key “m” for the first item (you will see a “*” in front of this item). Now choose the next item by hitting the space bar: both items will be show in separate buffers.
Gams-outline allows to split the window horizontally or vertically (I prefer the vertical version). If you are in outline-mode and you have selected an item, you will
See in the status bar at the bottom of your Emacs help: on outline mode:
Hit the “x” key and you are in vertical mode. Hit it another time and you are back to horizontal mode.
If you don’t want all the items being listed, you hit the “t”-key and define if you want to have the summaries (SUM), variables (VAR) etc. being listed or not.

Emacs Goodies

Often you have to repeat a series of keystrokes. That is probably boring if the number of repetitions is low, but gets really annoying if the number gets higher. A typical example is formatting data or text. Take the following text snippet where we have data coming from a comma separated file:
A;410
B;243
C;134
A;410
B;324
C;134
A;410
B;324
C;134
and you would like to have this text in the following format
A        410,
B        243,
Emacs makes this a very easy job: Start the macro recording with C-x (  or F3 (the C stands for the Control key) and do the editing of the first line. After you have finished editing the first line, set your cursor at the beginning of the second line and hit C-x ) or F4 (stop recording). Now hit C-x e (start keyboard macro) and Emacs does the second line for you. Keep hitting “e” and it will do the next lines.
If you have 10000 lines like this, hitting the “e” 10000 times might be not a very good option. Instead keep the M key (on windows this is the Alt-key) pressed and punch in 10000 followed by C-x e. Emacs now runs the recorded macro 10000 times.

Syntax Coloring in Gams-Mode

I suppose you have installed emacs and gams-mode for emacs (if not see my preceding posts) and now are eager to start working on your model.
Let us start simple and get a model from the Gams-library. If you started emacs you are not in gams-mode (you can check this by looking at the menu bar: there is no gams-entry). If you load a gams model (with the extension “gms”), Emacs automatically turns on gams-mode and the gams entry appears in the menu bar (blue arrow). The first thing you probably notice is the syntax coloring. The coloring you will see might slightly differ from what you see in the picture. Gams-mode is highly adjustable and I changed some of the settings for syntax coloring. All gams keywords are  bold blue, the comments in a $ontext-$offtext block are green, etc.
This feature is of great help: if you are writing your code and mistype a keyword it doesn’t get colored and you know that mistyped something. In the picture you see that I mistyped “Table”. Compare it to the picture above and you see the difference.
If you don’t like the default syntax coloring (like I did), you can change the settings easily: go to the menu entry “Customize Gams Mode for Emacs“. Emacs now opens a new buffer with all the settings for Gams mode. Go to the end of the buffer to the group “Gams Facesand click on the buttonGo  to Group“. The  first setting you will see is the one for coloring comments. The green arrow shows you how comments will appear (green). Now click on the button “Show Face“. There are many options that can be adjusted.
I usually adjust only two of them: “Weight” (bold, normal, etc) and “Foreground”, the color. For the weight property you can choose from a drop down menu. For the color property you can either enter the Hexadecimal Color Code or just use the plain names for colors (green, red, etc.). Here is a site that shows you the hexadecimal color codes (http://www.december.com/html/spec/colorhslhex6.html#66).
If you are finished adjusting the settings, you finally have to save them. Scroll up to the beginning of the buffer and set the new settings for the current session and save them for future sessions.

Using a Newsreader

You can now automatically download the posts from this side using your newsreader (for example Outlook 2007/2010). Click on the “site feed” on the right and follow the instructions.