Content
Thoughts on Modelling and Statistics using Gams, Emacs and Stata or R
Tuesday 22 June 2010
-
Filed under
Emacs + Stata
Although I am not a great fan of multi-tasking I often use a lot of programs simultaneously for one task: For example, when I am writing a report in Word or with Emacs, I take notes in Emacs, search the internet and have my model results in the gdx-viewer. Additionally my mail client is somewhere in the background and nowadays I have a messenger program open to talk to my collaborators. I don”t like this jumping from one program to another, but it is often necessary. This is also one of the reasons I switched to Emacs, because I can do my main work in Emacs (modelling, note taking, statistics and report writing).
For my statistical work I use R, SPSS, Eviews and Stata. Emacs has a nice mode for working with R (ESS: Emacs-Speaks-Statistics). You can edit your scripts and run it in R without leaving Emacs. ESS has highlighting and lots of other features. ESS supports Stata too, but you can”t run your Stata commands in Emacs. So you have to copy-paste from Emacs to Stata. The editor of Stata (I use version 10) is a plain text editor without syntax highlighting and almost no other features (version 11 has a much better editor but the update is too expensive).
So what can you do, if you still want to use Emacs (or any other text editor other than the Stata editor)? Use a lot of keyboard strokes switching back and fort from one program to another is one option, but here you run into other problems: if you have multi line commands copy-paste doesn’t work for Stata as you have to strip off the linefeeds. OK, you can write your commands on one line, but then you have to adjust your output if you want to document your scripts nicely. Another more practical way is using
autohotkey, a free, open-source utility for Windows
(there are similar programs for Unix or Mac). Here are the great features of Autohotkey. With Autohotkey, you can (taken from the manual):
- Automate almost anything by sending keystrokes and mouse clicks. You can write macros by hand or use the macro recorder.
- Create hotkeys for keyboard, joystick, and mouse. Virtually any key, button, or combination can become a hotkey.
- Expand abbreviations as you type them. For example, typing “btw” can automatically produce “by the way”.
- Create custom data-entry forms, user interfaces, and menu bars. See GUI for details.
- Remap keys and buttons on your keyboard, joystick, and mouse.
- Respond to signals from hand-held remote controls via the WinLIRC client script.
- Convert any script into an EXE file that can be run on computers that don”t have AutoHotkey installed.
For the problem described above I wrote a small script and linked it to the key combination Windows+s.
The script looks like this:
#s:: ; the short cut (Windows key + s
clipboard = ; Empty the clipboard.
Send !w ; Copy the highlighted region in Emacs (Alt-w)
; Replace /// (necessary in Stata for multiline commands) and line feeds:
StringReplace, clipboard, clipboard, ///`r`n, , All
WinActivate, Stata/SE ; Activate the Stata window
Send ^v ; Copy the clipboard (Ctrl-v)
Send {Enter} ; Execute the commands
; Jump back to Emacs (Renger”s Emacs Mode is my Emacs window title)
WinActivate, Renger”s Emacs Mode
return
If I mark a region in my Stata script in Emacs and hit Windows+s, the line is copied to the clipboard, all line end feeds (only in multi line commands) are stripped off and the edited region is copied to Stata and executed. After the execution the cursor jumps back to Emacs. This with just one key stroke!
I use Autohotkey for lot of other tasks (one recently was taking a sample from a telephone directory and copying the addresses to an Excel sheet). Another nice feature I implemented is automating copy-paste from a pdf-file I am reading to a text file with my notes. Once again: one click and the text is in my notes.
::
Share or discuss
::
2010-06-22 ::
admin
Tuesday 22 June 2010
-
Filed under
GAMS + Gams-Mode
Now this might be obvious if you have a lot of time, but usually the rule is forgotten if you are stressed and want to finish what you are working on. Not writing comments will probably cost you a lot of time, if you take your model a year from now and see your code: Wow, that looks interesting and very ingenious, but what the %*?! did I do here? If you send your model to somebody else for help, this person usually don”t want to dechiffer your code. Comments make her life easier too and the chance you get an answer will increase.
In Gams you have many options to add comments:
- Start a line with a “*” and write your comment, like
* This is a comment
You can change the character “*” in a different character with this command
$comment #
# This is a comment with the new character
- Put your comment between $ontext and $offtext:
- Put your comment after some code using the end-of-line comment possibility after the two characters “!!”
C = IOT(”H”,”CONS”) !! This is the consumption initialization
Note that this possibility of setting comments is turned off by default. You have to activate it by inserting $oneolcom (on end of line comment) or $inlinecom (inline comment)
You can also change the default setting of “!!” to two other characters by using $eolcom
Gams-mode for Emacs has a nice feature that allows you to hide multi-line comments (just use Ctrl-c Ctrl-h to hide or show the comments). Hiding the comments gives you a good view on your code
Before hiding:
After hiding:
Writing comments is good practice.
Adding descriptions to your parameters, variables and equations is also good practice. If you use short variable or parameter names, this might save your time, but once again, will make life hard if you look at your model after some time or if your model is big. Was “y” domestic production level or was it income?
If you use descriptions your output is easier to read:
compared with:
Gams-mode for Emacs has another nice feature: if you put your cursor on a parameter or variable in your code and hit F7, another window opens and shows you the description.
::
Share or discuss
::
2010-06-22 ::
admin
Tuesday 22 June 2010
-
Filed under
GAMS
The first golden rule for modeling is: Always display the results of your calculations and check them with your expectations about the results.
Often you are working on a model doing a lot of preparing calculations (calibrating the model, preparing the data). Those calculations might be easy, for example calculating the shares for some parameter, but if you make an unnoticed error can cost you a lot of time and frustration.
Here is a simple example. I want to calculate the shares for the investment function
Calculating the share is not difficult thing, but you can make small mistakes. When I teach I often see people write things like
shareinv(s) = inv(s)/sum(as, inv(as));
There is only a small mistake: instead of summing over inv(as,”inv”) people sum inv(s,”inv”).
Gams in this case just multiplies the value three times instead of summing over all the elements
The best way to guard yourself against errors like this is by displaying the results of your calculations. Just add a display statement and a $exit on the next line (the last command stops gams even if there is code after the $exit. Don”t forget to write the $ of $exit in the first column).
But not only should you display your results, you should before that ask yourself what the result of your calculations will be. In this case you expect values between 0 and 1. Thinking about the expected results before you see the results is perhaps one of the most important things in modeling.
Perhaps this sounds all a little bit exaggerated, but mistakes are easily made and if you don’t check your results, you can loose a lot of time.
I have seen models with negative values for output, shares bigger than 1 and so on. Displaying your results and checking them with your intuition doesn’t take much time and can help you find bugs in an early stage of modeling.
::
Share or discuss
::
2010-06-22 ::
admin
Tuesday 15 June 2010
-
Filed under
Blogging
Unfortunately, my blog was hacked (that happens sometimes if you do not update WordPress to the newest, secure version…). I have reinstalled my blog and will post my past blogs to it in the next few days. Sorry if you get the old posts again in your RSS feader…
I haven’t written for a long time, because I had a busy time. I am now part-time at the CEPE (Centre for Energy Policy and Economics) as PhD-student of Tom Rutherford, working for Ecoplan, doing policy consulting and for Modelworks, my own little firm. From July onwards I will concentrate on my PhD.
::
Share or discuss
::
2010-06-15 ::
admin
Tuesday 15 June 2010
-
Filed under
GAMS
The Gams mailing list can be very helpful. There are a lot of people who are willing to use their free time to answer your questions. My advice: subscribe to the mailing list at http://www.listserv.dfn.de/cgi-bin/wa?SUBED1=gams-l&A=1 ( and even if you don’t post to it, you can learn a lot of the questions and answers by others. If you post: don’t be afraid to ask a stupid question. The worst thing that can happen is that you get an answer like I once got. I asked a really stupid answer and got the answer: “Think”!
Tom Rutherford posted some nice guidelines for writing to the Gams mailing list. They might be useful for any mailing list you use. Here are the guidelines:
You will get more mileage out of the GAMS list if you adhere to a few guidelines (nothing official here, only my unsolicited advice):
1. Always take time to compose a clear statement of the problem from first principles.. Often times, the process of clearly explaining the problem is sufficient impetus for finding your own logical error.
2. If you have a problem specifically related to GAMS syntax or execution, write down the minimal operational GAMS program which illustrates your problem. Minimize the number of symbols you use in this program, but leave no descriptive field blank. Add comments to the program to the extent that they are needed to explain what you are doing.
3. If you are working on something more complex, write up the mathematics in a PDF document which explains what you are trying to accomplish.
4. If possible, upload a zip file containing the GAMS code and documentation to a web site (these are freely available everywhere these days my daughters tell me), and send a nice short message to the GAMS list, briefly indicating your field/problem type and an operational link to your zip file. (My policy, whenever posting a zip file, is to be sure that I am able to download and unzip the file on my own machine before asking someone else to try).
The advantage of using a zip file is that you can include all of the data files and perhaps a restart file which can save someone lots of time. The idea here is to provide the support staff with a “smoking gun”. We need to see precisely where the error occurs, and we need to be able to reproduce it on our own platform if there is to be a high probability of providing assistance.
::
Share or discuss
::
2010-06-15 ::
admin
Tuesday 15 June 2010
-
Filed under
Emacs + GAMS + Gams-Mode
First run your model by hitting C-c C-s (C stands for Ctrl) or use the same command from the menu. Emacs will open a new window and you see how Gams solves your model. When Gams is finished you can hit F10 (or C-c C-v) to show the listing file in a new buffer. We will however assume that the model solved without errors and we are interested in the results. Of course: you can scroll through your listing file using the search commandos from Emacs. Gams-mode however has a great way of showing all your results in a nice outline mode. This mode also allows you to jump from one result (for example the output of a variable to the definition of a set). Instead of jumping to the listing file, we therefore jump with F11 to the outline-view of Gams-mode.
You can also jump to the outline-mode from your gams file by hitting C-c Tab or hitting C-c C-ii (this command cycles through gams-, listing- and outline-mode).
As soon as you are in outline-mode you will see that the menu bar will have changed and the menu item GAMS-OUTLINE appears. Here you will find all the commands related to the outline-mode (with the short-cuts after the command name):
You will see a buffer with a grouped list of your variables, sets, parameters, etc. The next picture shows you the outline-mode for the library file abel.gms
First notice, that the information is grouped and colors are used to make a distinction between the different groups.
If you put your cursor on one of the items, for example on PAR w, your screen is split in two halfs: the upper part is still the list with all the items. The lower buffer shows the part of the listing file with the output for the chosen item. You can jump to the next item by pressing the key “n” (for next) and to the previous by pressing “p” (yes: for previous…).
Often you want to have a look at two items at the same time: just press the key “m” for the first item (you will see a “*” in front of this item). Now choose the next item by hitting the space bar: both items will be show in separate buffers.
Gams-outline allows to split the window horizontally or vertically (I prefer the vertical version). If you are in outline-mode and you have selected an item, you will
See in the status bar at the bottom of your Emacs help: on outline mode:
Hit the “x” key and you are in vertical mode. Hit it another time and you are back to horizontal mode.
If you don’t want all the items being listed, you hit the “t”-key and define if you want to have the summaries (SUM), variables (VAR) etc. being listed or not.
::
Share or discuss
::
2010-06-15 ::
admin
Tuesday 15 June 2010
-
Filed under
Emacs
Suppose you have the following information in a text file:
A;410
B;243
C;134
A;410
B;324
C;134
A;410
B;324
C;134
…
and you would like to have this text in the following format
A 410,
B 243,
Emacs makes this a very easy job: Start the macro recording with C-x ( or F3 (the C stands for the Control key) and do the editing of the first line. After you have finished editing the first line, set your cursor at the beginning of the second line and hit C-x ) or F4 (stop recording). Now hit C-x e (start keyboard macro) and Emacs does the second line for you. Keep hitting “e” and it will do the next lines.
If you have 10000 lines like this, hitting the “e” 10000 times might be not a very good option. Instead keep the M key (on windows this is the Alt-key) pressed and punch in 10000 followed by C-x e. Emacs now runs the recorded macro 10000 times.
::
Share or discuss
::
2010-06-15 ::
admin
Tuesday 15 June 2010
-
Filed under
Emacs + GAMS + Gams-Mode
Let us start simple and get a model from the Gams-library. If you started emacs you are not in gams-mode (you can check this by looking at the menu bar: there is no gams-entry). If you load a gams model (with the extension “gms”), Emacs automatically turns on gams-mode and the gams entry appears in the menu bar (blue arrow). The first thing you probably notice is the syntax coloring. The coloring you will see might slightly differ from what you see in the picture. Gams-mode is highly adjustable and I changed some of the settings for syntax coloring. All gams keywords are bold blue, the comments in a $ontext-$offtext block are green, etc.
This feature is of great help: if you are writing your code and mistype a keyword it doesn’t get colored and you know that mistyped something. In the picture you see that I mistyped “Table”. Compare it to the picture above and you see the difference.
If you don’t like the default syntax coloring (like I did), you can change the settings easily: go to the menu entry “Customize Gams Mode for Emacs“. Emacs now opens a new buffer with all the settings for Gams mode. Go to the end of the buffer to the group “Gams Faces” and click on the button “Go to Group“. The first setting you will see is the one for coloring comments. The green arrow shows you how comments will appear (green). Now click on the button “Show Face“. There are many options that can be adjusted.
I usually adjust only two of them: “Weight” (bold, normal, etc) and “Foreground”, the color. For the weight property you can choose from a drop down menu. For the color property you can either enter the Hexadecimal Color Code
or just use the plain names for colors (green, red, etc.). Here is a site that shows you the hexadecimal color codes (
http://www.december.com/html/spec/colorhslhex6.html#66).
If you are finished adjusting the settings, you finally have to save them. Scroll up to the beginning of the buffer and set the new settings for the current session and save them for future sessions.
::
Share or discuss
::
2010-06-15 ::
admin
Tuesday 15 June 2010
-
Filed under
Blogging
You can now automatically download the posts from this side using your newsreader (for example Outlook 2007/2010). Click on the “site feed” on the right and follow the instructions.
::
Share or discuss
::
2010-06-15 ::
admin
Tuesday 15 June 2010
-
Filed under
Emacs + GAMS + Gams-Mode
First of all download gams-mode from Shiro Takeda’s website: You will find it here:
gams-mode-download.
The files are zipped and should be unzipped to a directory Emacs will find. I show you my settings (as usual, you can do it in an other way).
I have all my additional features in a directory that resides in the Emacs program directory:
The full name of the directory is C:\Programme\Emacs\site-lisp\gams. Notice that I don’t call the directory “..\gams-2.7.1″ (the actual version). If there is a new version of gams-mode out, I just drop the files in this directory and I do not have to adjust any settings I make in Emacs. You can see the other modes I use (for example python-mm, ess, planner and auctex).
The next thing is to tell Emacs where it can find gams-mode. For this open your “.emacs” (dot-emacs) file that will be in your home directory (see my previous post). If there is no .emacs-file, you just create one.
Add the following text:
(setq load-path
(cons “c:/Programme/Emacs/site-lisp/gams/” ;; Set the installed directory!
load-path))
(require ‘gams)
The first 3 lines tell Emacs where to find gams-mode and the fourth line loads gams-mode.
Now you are ready to start using gams-mode. Open a gams-file in Emacs and if all worked fine, you will see the gams-mode menu items (the red arrow in the next picture). In the picture you can also see that I have some features installed that do not come with Emacs (blue arrow). I will talk about that in another post.
In the next post I will tell you something about the great things that you can do with gams-mode and which will make your programming life more relaxed…
::
Share or discuss
::
2010-06-15 ::
admin
