pleasant things work better
I watched Don Norman's TED Talk "3 ways good design makes you happy" last night:
This part of his talk really stuck out for me (transcription errors mine):
I really had the feeling that pleasant things work better and that never made any sense to me, until I finally figured it out. Look:
I'm gonna put a plank on the ground. So imagine I have a plank about 2' wide and 30' long. And I'm going to walk on it. See I can walk on it without looking, and go back and forth, and I can jump up and down. No problem. Now I'm going to put the plank 300' feet in the air... and I'm not going to go near it, thank you. Intense fear paralyzes you. It actually affects the way your brain works.
...
If you're happy, things work better because you're more creative. You get a little problem, you say "Ah, I'll figure it out. No big deal."
Read more » 4 comments
Komodo 5.1.3 released
We (ActiveState) released Komodo 5.1.3 today. Get it here:
Or, if you are currently running any previous Komodo 5, click "Help > Check for Updates...". This is a bug fix release and is recommended for all users. See below for details.
Read more » 6 commentsKomodo 5.1.1: fixes, path mode in Fast Open dialog, Perl::Critic integration
We (ActiveState) released Komodo 5.1.1 today. Get it here:
Or, if you are currently running Komodo 5.1.0, click "Help > Check for Updates...". This is a bug fix release and is recommended for all users. See below for details.
Read more » 1 commentsKomodo 5.1 released (fast open, history, hyperlinks, etc.)
We (ActiveState) released Komodo 5.1 today! Get it here:
Komodo IDE 5.1 is a free upgrade for Komodo IDE 5.x license holders. Your license entitles you to run Komodo IDE on any of the platforms we support (Windows, Mac OS X and Linux). Komodo Edit 5.1 is, as ever, open-source and free.
Read more » 26 commentshyperlinks in Komodo 5.1.0b1
We released Komodo 5.1 beta 1 yesterday! Get it here:
Please try it out and give us your feedback:
| http://listserv.activestate.com/mailman/listinfo/komodo-beta | |
| bugs | http://bugs.activestate.com/enter_bug.cgi?product=Komodo |
| forums | http://community.activestate.com/products/Komodo |
This is the third release of Komodo 5.1 on the way to a planned final release very soon (hopefully within a week or two). Here are a few goodies in this release. (See my previous posts about the Komodo 5.1a1 and Komodo 5.1a2 releases.)
Hyperlinks
Many IDEs that have code intelligence support allow you to Ctrl+click (Cmd+click on a Mac) on a symbol to Go to Definition. Here was Komodo's feature request for that. This is now implemented in Komodo 5.1b1.
However, we've gone one step further and made a generic system where Ctrl+mouse-hover will underline interesting regions (hyperlinks) in your text for clicking on. The most common type of hyperlink is a symbol for "Go to Definition". However other types of hyperlinks include:
Colors in CSS (including CSS in HTML files):
As in Firebug, you'll get a swatch of the color when hovering over the color. What's more,
Ctrl+clickwill bring up the system's color picker with which you can change the color.HTTP and FTP URLs:
Ctrl+clickwill load that URL in your browser.Regular expression mapping to an HTTP URL.
Currently 5.1.0b1 includes a regex to map occurrences of "bug \d+" to the appropriate bug in ActiveState's bug database. Eventually we'll have a preferences dialog where adding these mappings will be easier, but for now here is how you can add your own.
A generic handler to do whatever you can think of.
The hyperlinks above are all implemented with a simple mechanism in Komodo's
ko.hyperlinksJavaScript namespace. You can add your own handlers to do other things. Following how the regexhandler works is a good place to start. We'll try to give more examples later.
Note: The hyperlink types other than "Go to Definition" are only in Komodo IDE (i.e. not in Komodo Edit) for the 5.1.0b1 release. So, if you want to play you should either use Komodo IDE 5.1.0b1 or use the latest nightlies.
koext updates
koext is a command-line tool for helping in building Komodo extensions. There are some great Komodo extensions that users have been building here. However, it is far from as easy as it should be to dig in and build Komodo extensions. Part of the solution is the koext tool. (Another part is documentation for extension authors, but that is a story for another time.)
We've started doing some updates to koext again (see the change log). Recent changes are working towards making it easier to have a quick development cycle -- i.e. make it so that to test a change to your extension you just need to:
- make your edit;
- possibly run
koext build --dev, e.g. if you changed an IDL file; and - re-start Komodo
Instead of the more laborious:
- make your edit;
- run
koext buildto build a new.xpifile; - re-install that
.xpiin Komodo; - re-start Komodo
In a subsequent post I'll describe how I setup to build a Komodo extension. Here is a brief intro to koext from way back
Other stuff
A quick list of other feature work, notable bug fixes and fixed annoyances in Komodo 5.1.0b1:
Komodo's new "History" feature now has session support which is currently used to make your history specific to a single Komodo window.
In Komodo IDE, the History now shows the section title for locations in the History. This can make the "Recent locations" menu a lot more useful:
In XML (and HTML and PHP, etc.) files, clicking on a tag will briefly flash (highlight) the matching tag (bug 81606).
Komodo's "jump to matching brace" now works as you'd expect for opening and closing tags in XML/HTML (bug 43239).
We've done some crash fix work so that Komodo 5.1b1 should be more stable that 5.1a2. It is hard to quantify and we continue to look for crash issues in Komodo.
Greatly improved the annoyance of it being very hard to grab the bottom-pane and sidebar splitters on Mac OS X (bug 80756).
Fixed the annoyance of being unable to resize Name/Type/Value panes in Locals/Globals debugger window (bug 80566).
fix: codeintel: calltips cannot show unicode doc comments http://bugs.activestate.com/show_bug.cgi?id=70448
Added Korean and Japanese JIS encodings (bug 80890).
As well there is more coming. Try out the Komodo nightly builds for the very latest stuff:
22 commentsFind highlighting and Linux/x86_64 support in Komodo 5.1a2
We released Komodo 5.1 alpha 2 a couple of days ago (shame on me for not announcing until now). Get it here:
Please try it out and give us your feedback:
| http://listserv.activestate.com/mailman/listinfo/komodo-beta | |
| bugs | http://bugs.activestate.com/enter_bug.cgi?product=Komodo |
| forums | http://community.activestate.com/products/Komodo |
This is the second release of Komodo 5.1 on the way to a planned final release around mid-May. There are a few goodies worth talking about in this release. (See my post about Komodo 5.1a1 here.)
Find highlighting
Komodo now highlights find/search matches in your buffer. Here is a short video showing it off. Sorry, no sound. This is my first screencast. :)
Here I'm doing a couple of searches using Komodo's Vi mode. That highlighting makes a big difference for helping your eyes find where you want to navigate to.
Find highlighting is one of those "well, duh" features that we are now able to add with indicator support in Komodo's editing component Scintilla. Indicators in Scintilla allow one to put visual markers on regions of the editor buffer independent of the syntax coloring information. Before indicators, syntax coloring styling and other styling (squiggly underlining for syntax errors/warnings, find highlighting, ...) had to share 8-bits of data for each position (i.e. each character). That was awkward (playing with bit masks) and limiting (ran out of space in, e.g., HTML which uses 7 of those 8 bits for all the different syntax coloring styles). Another example is Komodo's Tabstops -- which were made a lot more usable in 5.0 because of what we could do with indicators.
Linux/x86_64 support
We've added support for a new platform: Linux/x86_64. This is our first native 64-bit platform build. Linux x86_64 installs are getting to be quite common, and typically the default install of Linux distros on x86_64 don't include the 32-bit compatibility libraries. This means that attempting to use Komodo's 32-bit Linux build wouldn't work out of the box (it tends to work fine once the distro's 32-bit compat libs are installed) -- and hence was a common support issue. Hopefully, no more. As well, Linux/x86_64 users will possibly enjoy a slight performance benefit.
Localization patches from Davide Ficano (l10n)
Way back in the heady days of 2008 (before Komodo 5.0 was released) Davide Ficano (aka dafi) made this post on Komodo's forums:
that kicked off some starter work towards localizing Komodo. One of the necessary steps to getting good localizations of Komodo was to update Komodo's chrome to more rigorously use DTDs for XUL (we were using these fairly well) and string bundles for JavaScript code (we weren't doing so well here). Dafi whipped up a bunch of patches for this. Todd has finally managed to get those all checked in, so that Komodo is now in a pretty good state to start being localized.
Other stuff
In addition to the above, Komodo's History feature is coming along:
- The same keybindings as your browser for Back/Forward should be working on all platforms.
- The side mouse buttons on 5-button mice should work for navigating the history.
- This history will now properly handle cleaning out URLs from finished remote debugging sessions.
As well we've a few more goodies that should be ready to show for a beta 1 release in a week or two. As ever, try out the Komodo nightly builds for the very latest stuff:
13 commentsHistory feature in Komodo 5.1.0 alpha 1
We released Komodo 5.1.0 alpha 1 today! Get it here:
Please try it out and give us your feedback:
| http://listserv.activestate.com/mailman/listinfo/komodo-beta | |
| bugs | http://bugs.activestate.com/enter_bug.cgi?product=Komodo |
| forums | http://community.activestate.com/products/Komodo |
This is the first release of Komodo 5.1 on the way to a planned final release
around mid-May [Update: mid-March. Subconsciously I keep hoping for more time. :)]. (I'll write about our Komodo 5.1 plans in a separate post).
Here I want to talk about Komodo's new "History" feature.
History overview
Komodo's History feature is like your browser's history, but for the editor. Back and Forward buttons in the toolbar. Same default keybindings as in Firefox 1. Simple.
Komodo's history is a bit different than a browser's. In a browser, you have a separate history session for each tab. This doesn't make as much sense for an editor. Komodo's history is per-window 2. That means that the Back button will move you back to the last place you were, be that in the current file or in another file in the same window. This means that jumping back:
- after a Go To Definition, or
- after opening a new file, or
- after jumping to a find result
is as easy as clicking "Back".
More than any new feature in Komodo, the first time we hooked the feature up it felt immediately useful. Of course, this is just an alpha release so there is lots of polishing to do. Read on for some of the other things we hope to do with this.
Future work
Some other things Komodo will be able to support with this:
- Opening recent files quickly. Chances are good that a file you want to open in your editor is a file you've edited before (and recently). The history database now provides Komodo with the data it needs to support that.
- Find in recent files. Often I'll want to look at some snippet of code that I remember writing in the last few days, but can't remember what file (or even what project) that was in.
Support
''command in Vi-mode. From the Vim help:'' `` Move to the position before latest jump.Hooking
BackandForwardinto the MS Intellimouse's (and other mice, I suspect) side buttons -- as is the default in Firefox.
Backend code
For those interested, most of the backend of the history system is here in "editorhistory.py" in the Open Komodo subversion repository.
For JavaScript code (most interesting to Komodo extension developers) there is
a new ko.history
API
with the most relevant methods being:
ko.history.note_curr_loc(view): Tell the history system to note the current editor location before jumping somewhere.ko.history.history_back(n): Go backnlocations.ko.history.history_forward(n): Go forwardnlocations.
Komodo's history database shares some ideas with Firefox 3's Places database. In particular the idea of splitting visited locations (URLs in Firefox, editor locations in Komodo) and visits into separate database tables was helpful. They are, of course, both SQLite3 databases.
ActiveState Code: lauched!
We launched ActiveState Code today, and there was much rejoicing. Yaaaah!
ActiveState Code is a site for sharing code recipes. It is the replacement for the popular ASPN Cookbooks (especially the Python Cookbook, which was a collaboration with O'Reilly and Associates that resulted in two print cookbooks using recipes from the site). The new site adds things like tagging, the ability to add recipes in a number of other languages, and a fresher and hopefully more usable site.
Migration should be easy. All recipes from the Python, Tcl and PHP Cookbooks have been carried over. Redirects maintain all old aspn.activestate.com/ASPN/Cookbooks links. Recipe id and author ids have been maintained. The ASPN Cookbook categories have been translated into tags in the new system -- full details here.
I welcome any feedback on the site.
0 commentsTranslating Komodo on babelzilla.org
At the prompting of dafi (one of Komodo's more active users) and with goofy's help I've uploaded a Komodo Language Pack with Komodo's current en-US localization to BabelZilla.
We get requests for localizations of Komodo every so often, but lack the resources to do this ourselves. Given a number of good translations here, I can start providing language pack extensions to Komodo's add-ons site.
This is a trial balloon that I really hope turns out well. I remember having a great chat with Michal Berman about Mozilla's l10n infrastructure at FSSOS 2007 waaay back in October last year. I dropped that ball following up after that discussion. I'd also like to take a look at Launchpad Translations to see if that would work well.
If you are interested in helping out with translating Komodo to your language, please take a look at this forum thread and/or let me know (trentm at activestate dot com).
2 commentsnightly updates for Komodo (and the Komodo AUS)
I and others here have been hard at work on Komodo 4.3 (due to go final this week) so it has been a while since I've posted. One thing I've wanted to post about for quite a while is the Komodo auto-update system. I alluded to it waaay back when working on adding auto-update support to Komodo 4.2 but haven't written anything about it since.
Last Friday gives me good reason to post about it: We now have a "nightly" channel for Komodo Edit!
Update (4-Nov-2008): Nightly updates should not work for Komodo IDE as well!
Komodo Auto-update channels
There are three Komodo auto-update "channels":
"release": This is the typical (and default) channel for installations of a final release of Komodo (e.g. 4.2.0, 4.2.1, 4.3.0). On this channel, Komodo will only update itself to the latest final Komodo release.
"beta": This is the typical (and default) channel for Komodo alpha/beta builds. On this channel, Komodo will update itself to the final or pre-release (i.e. alphas and betas).
"nightly": This is a channel that I finally got working on the server-side on Friday. Since the announcement of OpenKomodo and open sourcing of Komodo Edit we've been doing "nightly" builds of Komodo Edit (built on most nights :). These are publicly available here: http://downloads.openkomodo.com/komodoedit/nightly/
As of last Friday, if you are on the nightly update channel Komodo Edit will update to the latest nightly build. These are quite a bit more burning-edge that the "beta" channel. Often the only criteria for putting up a nightly is that the build worked for all platforms. So, occasionally some features are broken -- though I think we do pretty well.
This channel is quite new for us though, so there may be some growing pains in the first couple of weeks. Please let me know if you have any problems with it. I think it will be pretty cool to easily always be running the very latest Komodo Edit.
At this time we aren't yet doing public nightlies of Komodo IDE.
Setting the update channel for your Komodo installation
Currently there isn't a prefs panel in Komodo to tweak auto-update settings -- such as the channel you are one. There should be. I hope to get one in sometime after 4.3.0.
To set your Komodo channel edit "channel-prefs.js" in your Komodo installation as appropriate. On Windows and Linux this file is here:
INSTALLDIR/lib/mozilla/defaults/pref/channel-prefs.js
and on Mac OS X here:
INSTALLDIR/Contents/MacOS/defaults/pref/channel-prefs.js
It is a very simple file that looks like this:
// Valid values are "release", "beta" and "nightly" (internal-only).
pref("app.update.channel", "beta");
Happily that "internal-only" is no longer correct for Komodo Edit.
Update (4-Nov-2008): Ditto for Komodo IDE now.
Other Komodo AUS Stuff
Komodo's auto-update system, on the client side (i.e. the app), pretty much just uses the excellent Mozilla auto-update system. On the server-side we have our own (very simple) Django-based update server. On the build-side, we have our own Python scripts (mozupdate.py et al) for building all relevant partial and complete update packages as part of full builds.
I hope to post more about our AUS server and about our build tools. I think I could fairly easily package up our tools to provide a possible answer to Mozilla Bug 415181 (Package the MAR generation tools for easy external usage).
There used to be an open bug (found it: Mozilla Bug 375752) to convert some of the Bash shell scripts for Mozilla update package building to Python scripts. I see (by way of Mozilla Bug 410806) that that has at least partially happened with make_incremental_updates.py.
open komodo is out
The wraps are finally off Open Komodo!
- source: http://svn.openkomodo.com/openkomodo/browse/openkomodo/trunk
- grok (an alternative to lxr): http://grok.openkomodo.com/source/search?q=koDirs
- wiki: http://wiki.openkomodo.com/index.php/Main_Page
- mailing lists: http://lists.openkomodo.com/mailman/listinfo
- irc: irc.mozilla.org #komodo
Check it out.
Quick build notes:
# Get the source
svn co http://svn.openkomodo.com/repos/openkomodo/trunk openkomodo
# Build Mozilla
cd openkomodo/mozilla
python build.py configure -k 1.0 --moz-src=cvs:1.8 --release \
--no-strip --shared --tools
python build.py distclean all
cd ..
# Build Komodo
export PATH=`pwd`/util/black:$PATH # Komodo's "bk" build tool
bk configure
bk build
# Run Komodo
bk run
If you have the build prerequisites setup, you should be able to cut 'n paste the above. (Windows users, use the Windows-specific quick build steps in the README.txt.)
I have some (longer term) plans to reduce those build steps to:
./configure.py
mk
off to FSOSS
I'm off to FSSOS in a few hours. We'll be opening up the Komodo sources next Wednesday, so Shane and I will be there to start the discussion about what Open Komodo and Snapdragon should focus on to best improve the tool story for open web development. (Currently the Open Komodo sources are available to a small group of mozillians. If you have some ideas and would like to take a peek, let me know and I'll hook you up!)
I'm hoping to meet a few of the Mozilla folks that will be there: Benjamin to ask about breakpad (I want to get breakpad running for Open Komodo and Komodo builds), looking forward to Mike Beltzner's talk on UE design at Mozilla, Eric Shephed to ask about how mozilla handles localization of their docs.
As well, if any of the mozilla folks involved in the AUS will be there, I'd love to talk with you to compare notes with Komodo's auto-update system.
1 commentsmercurial needs better end-of-line support
One real world issue with source control systems is the handling of end-of-line characters in text files. Currently Mercurial pretty much punts. The hg book says:
Note: The Windows version of Mercurial does not automatically convert line endings between Windows and Unix styles. If you want to share work with Unix users, you must do a little additional configuration work. XXX Flesh this out.
NOTE: the tempfile mechanism is recommended for Windows systems,
where the standard shell I/O redirection operators often have
strange effects. In particular, if you are doing line ending
conversion on Windows using the popular dos2unix and unix2dos
programs, you *must* use the tempfile mechanism, as using pipes will
corrupt the contents of your files.
Tempfile example:
[encode]
# convert files to unix line ending conventions on checkin
**.txt = tempfile: dos2unix -n INFILE OUTFILE
[decode]
# convert files to windows line ending conventions when writing
# them to the working dir
**.txt = tempfile: unix2dos -n INFILE OUTFILE
However (1) unix2dos and dos2unix are generally not available on Windows machines and (2) if dos2unix isn't available the "encoding" here will silently wipe out your file to empty content on checkin.
How is Mozilla handling this in their hg repository? Is it mandated that new files added on Windows use Unix line endings or is some kind of conversion for Windows attempted?
3 commentspatch for "custom action" for mozilla updater
Here is a patch (against a slightly out of date updater.cpp on the Mozilla 1.8 branch) that would add support for a:
customaction "relative-path-to-executable"
action in the "update.manifest" for a partial or full update (.mar file) for the Mozilla update system.
I'm just chucking this up here quickly for lack of a better place to put it right now. Komodo uses the Mozilla update system and will possibly need this patch at some point.
Limitations:
- It ignores the return value of the spawned executable.
- It doesn't support arguments to the executable.
an intro to Komodo extensions
Komodo uses the Mozilla extension mechanism -- same .xpi files as Firefox to install an extension, same kind of bundle content in an extension. However, Komodo adds a number of "hooks" that can be used to customize Komodo with an extension (see the end of this post).
In Komodo 4.2 (currently in beta) we've been working at improving the extension story. Part of my work there has been to improve the tools for building them. To that end Komodo 4.2 now includes a sort of "SDK" with a few tools:
- koext
- A tool for building and generating stubs for Komodo extensions. A recently added a (very brief) intro to using koext to Komodo's extension forum.
- luddite
- A tool for working with Komodo's UDL (User-Defined Languages) system. The UDL system (new in Komodo 4.0) provides a way to define lexers for new languages. Lexers are used mainly for syntax coloring, but can also be used by Komodo Code Intelligence system for provide autocomplete and calltips. Eric wrote up a long intro to UDL a while back. UDL currently isn't for the faint of heart, but it provides an execellent system for robust lexing of code languages -- in particular it supports *multi-language* code (e.g. JavaScript in HTML, Ruby in RHTML, CSS in Django HTML).
- codeintel
- A tool to help writing a language support for Komodo's Code Intelligence system. I'll write more on this later.
These tools are all works in progress but they are used internally as part of normal Komodo development, so should be usable for Komodo extension authors.
Komodo's koext tool briefly describes all the current Komodo extension "hooks":
$ koext help hooks
Many parts of Komodo's functionality can be extended with a
Komodo extension. We call those "hooks" here. The following is
a list of all extension hooks that Komodo currently supports.
The "source tree files" sections below are conventions for
placement of sources files. If you use these conventions, then
`koext build' will automatically be able to build your extension
properly.
chrome
Chrome is the collective term for XUL (content), JavaScript
(content), CSS (skin), images (skin) and localized files
(locale, typically DTDs) that can be used to extend the
Komodo UI. This works in Komodo extensions in exactly the
same way as any other Mozilla-base application (such as
Firefox). See `koext help chrome' for some tips.
source tree files:
chrome.manifest
content/ # XUL overlays, dialogs and JavaScript
skin/ # CSS
locale/ # localized files (typically DTDs)
XPCOM components
XPCOM components are placed here. These can be written in
Python or JavaScript. (C++-based components are possible
as well, but currently the Komodo SDK does not support
building them.)
source files:
components/
*.idl # interface definitions
*.py # PyXPCOM components
*.js # JavaScript XPCOM components
templates
A file hierarchy under here maps into Komodo's "New File"
dialog. For example, "templates/Common/Foo.pl" will result
in a new Perl file template called "Foo" in the "Common"
folder of the "New File" dialog.
source files:
templates/
lexers
Komodo User-Defined Languages (UDL) system provides a
facility for writing regular expression, state-based lexers
for new languages (including for multi-lang languages).
".lexres" files are built from ".udl" source files with
the "luddite" tool (in this SDK). See `koext help udl' and
Komodo's UDL documentation for more details.
source files:
udl/
*-mainlex.udl # a .lexres will be build for each of these
*.udl # support files to be included by
# "*-mainlex.udl" files
XML catalogs
An extension can include an XML catalog (and associates
schemas) defining namespace to schema mappings for XML
autocomplete.
source files:
catalog.xml # Note: This may move to xmlcatalogs/...
API catalogs
An extension can include API catalogs to provide autocomplete
and calltips for 3rd party libraries. An API catalog is a CIX
file (an XML dialect) that defines the API of a
library/project/toolkit.
source files:
apicatalogs/ # .cix files here will be included
# in the API catalog list in the
# "Code Intelligence" prefs panel
Python modules
An extension can supply Python modules by placing then in
the "pylib" directory of the extension. This "pylib" directory
will be appended to Komodo's Python runtime sys.path.
source files:
pylib/
codeintel
An extension can provide the Code
Intelligence logic (for autocomplete and calltips, for
"Jump to Definition" and for the Code Browser in Komodo IDE)
for new languages.
source files:
pylib/ # lang_*.py files here are picked up
# by the codeintel system.
6
comments
open komodo and the code
Yesterday we (ActiveState) announced Open Komodo, an open-source project seeded with much of the core of Komodo Edit and Komodo IDE with the goals of produce a platform/framework for and (codename Komodo Snapdragon) an IDE for client-side open web development.
That's a mouthful. Shane and David have done a good job giving some wider perspective on what the Open Komodo project could mean (if all goes well). David went so far as to invent new language to make his points.
Some quick thoughts from a coder's perspective:
The source will be available in a Mercurial repository in (quoting Shane paraphrasing Mike Shaver) "Two F**king Months!". Early November -- or earlier if we can.
Komodo is a Mozilla-based application with the added heavy use of PyXPCOM for much of the core logic. That means the app comes together like this:
- Get a slightly tweaked mozilla build (C++, JavaScript, XUL).
- Get a slightly tweaked Python build (C).
- Add a bunch of core logic (Python). For example, the guts of Komodo's Find/Replace system is written in Python -- using Python's unicode-aware regular expression engine.
- Add Komodo chrome (XUL, JavaScript, CSS, DTDs).
What this means is that to work on and add significant functionality to Komodo, all you tend to need to know is XUL, JavaScript and Python. From early on in Komodo's development we've felt that this is one of Komodo's aces in the hole: developing in the dynamic languages is so much faster. I remember David Ascher making the comment way back that if Subversion had been written in Python, it would have been ready years sooner. And now two of the primary DVCS, Mercurial and Bazaar, are written in Python.
Komodo uses the same extension mechanisms as Firefox. It is easy to build a .xpi to add functionality to Komodo. We really hope that a community of Komodo extension authors will develop.
Komodo builds and runs on Windows, Linux and Mac OS X. Given some work there is little reason the Open Komodo code base couldn't be made to run well on Solaris, BSD, etc.
If any of this sounds interesting to you as an open-source tinkerer, then give Komodo Edit or Komodo IDE a try. The first app that will come out of the Open Komodo project (Komodo Snapdragon) will look and feel a lot like them.
In subsequent posts, and especially once the source code repository is up, I plan to blog here about Komodo's internals.
3 commentsbuilding MSI patch packages (.msp) with WiX
This post includes a complete and concrete example of building an MSI patch package (a .msp file to upgrade an existing .msi installation) with WiX.
Background
I'm responsible for building the ActivePython and Komodo installers at ActiveState. On Windows we build MSI packages for installation.
Currently I'm investigating auto-update support for Komodo 4.2. Because Komodo is based on Firefox/Mozilla we can benefit from the excellent Mozilla update system (I'll write another post about our experience with it). However, integrating with an MSI-based installation isn't something the Mozilla update system does out of the box: Firefox and Thunderbird don't use MSI for their installers (they use NSIS), hence I suspect working with MSI was never a design consideration.
While working out how to best marry MSI and Moz update, I investigated producing MSI patch packages (.msp files) for Komodo updates. MSI is a complex and complicated technology (it would be nice if the latter, at least, wasn't the case). Back in the day I used InstallShield for building our MSI packages, but now WiX is the best way to build .msi's -- by far. WiX helps a lot, but building appropriate MSI packages is still quite difficult. The following two pages helped me get to successfully building .msp's. Hopefully this concrete example will help others too.
ActiveFoo 1.0
For this example we'll build .msi installers for versions 1.0.0 and 1.0.1 of the the mythical "ActiveFoo" app ("activefoo-1.0.0.msi" and "activefoo-1.0.1.msi"). Then we'll build a '.msp' that will upgrade a 1.0.0 install to 1.0.1. We'll have the following files:
1.0.0/
activefoo.wxs # This describes "activefoo-1.0.0.msi"
config.wxi
installimage/ # The ActiveFoo install image
CHANGES.txt
foo.exe
README.txt
1.0.1/
activefoo.wxs # This describes "activefoo-1.0.1.msi"
config.wxi
installimage/ # The install image with changes for 1.0.1
CHANGES.txt
README.txt
upgrade-1.0.0.wxs # This describes the '.msp'.
make.py # 'python make.py' to build everything
README.txt
Here is a zip of the working files for this example, if you'd like to play along.
We have a simple install image with three files (foo.exe, README.txt and CHANGES.txt). The WiX code to build an installer for ActiveFoo 1.0.0 is 1.0.0/activefoo.wxs:
<?xml version="1.0" encoding="utf-8"?>
<?include config.wxi ?>
<Wix xmlns="http://schemas.microsoft.com/wix/2003/01/wi">
<Product Name="$(var.ProductName)" Id="$(var.ProductCode)"
Language="1033" Codepage="1252" Version="$(var.ProductVersion)"
Manufacturer="Acme" UpgradeCode="$(var.UpgradeCode)">
<Package Id="????????-????-????-????-????????????" Keywords="Installer"
Description="$(var.ProductName)"
Comments="blah blah" Manufacturer="Acme"
InstallerVersion="200" Languages="1033" Compressed="yes"
SummaryCodepage="1252" />
<Media Id="1" Cabinet="media.cab" EmbedCab="yes" />
<!-- Define some of the dir-structure. -->
<Directory Id="TARGETDIR" Name="SourceDir">
<Directory Id="ProgramFilesFolder" Name="PFILES">
<Directory Id="INSTALLDIR" Name="$(var.InstallId)"
LongName="$(var.InstallName)" />
</Directory>
</Directory>
<!-- Define the feature hierarchy (just one feature in this simple
example). -->
<Property Id="INSTALLLEVEL" Value="1000" />
<Feature Id="core" Title="ActiveFoo" Description="The Foo core"
Level="1">
<ComponentRef Id="MainExe" />
<ComponentRef Id="ReadMeFiles" />
</Feature>
<!-- Define all the components. -->
<DirectoryRef Id="INSTALLDIR">
<Component Id="MainExe" Guid="6ee6fda3-6f50-47bf-99b9-6031c720428e">
<File Id="MainExe" Name="foo.exe" DiskId="1"
src="installimage\foo.exe" Vital="yes" />
</Component>
<Component Id="ReadMeFiles" DiskId="1"
Guid="8f2255f3-3eaf-4c82-9688-3545cd9b2018">
<File Id="README.txt" Name="README.txt"
src="installimage\README.txt" />
<File Id="CHANGES.txt" Name="CHANGES.txt"
src="installimage\CHANGES.txt" />
</Component>
</DirectoryRef>
</Product>
</Wix>
with some configuration variables included from 1.0.0/config.wxi:
<?xml version="1.0" encoding="utf-8"?> <Include> <?define ProductCode = "cdc5e50f-b490-4a37-8ff6-22e3cb3d690e" ?> <?define UpgradeCode = "ed340ed8-aa91-4bf6-9dcf-d7f6f4d43737" ?> <?define ProductName = "ActiveFoo" ?> <?define InstallName = "ActiveFoo" ?> <?define InstallId = "AFoo10" ?> <?define ProductVersion = "1.0.0" ?> <?define ProductURL = "http://www.example.com/products/activefoo/" ?> </Include>
(Note that this WiX project is simplistic. In a real world WiX project you'd likely have a UI element for a user UI, define Add/Remove Programs -- ARP -- properties, etc.)
Use the provided "make.py" script to build "activefoo-1.0.0.msi":
C:\tmp\wix_and_msp> python make.py -v 100 INFO:make:build target '100' DEBUG:make:running 'candle -nologo activefoo.wxs' in '1.0.0' activefoo.wxs DEBUG:make:running 'light -nologo -o ../activefoo-1.0.0.msi activefoo.wixobj' in '1.0.0' INFO:make:'activefoo-1.0.0.msi' created
and install it. You should now have a "ActiveFoo" folder in your "Program Files".
ActiveFoo 1.0.1
Version 1.0.1 has the following changes:
- The ProductVersion is incremented to 1.0.1. We aren't change the ProductCode so this qualifies in MSI parlance as a "minor upgrade", as opposed to a "small update" or a "major upgrade").
- We've added a note to "CHANGES.txt" for the new release.
- We've removed the "foo.exe" file from the install image. This is so we can see how file removal can be accomplished with a "minor upgrade". There is a lot of documentation out there than says that file removal can't be done with an MSI minor upgrade. We'll see that that isn't true. I haven't seen any justification for why minor upgrades shouldn't remove files.
Normally, for these changes, the only updates to the WiX sources to build "activefoo-1.0.1.msi" would be to (a) update the "ProductVersion" string and (b) remove the File and Component elements for "foo.exe". However, working from this comment in Minor and Major Upgrades Using IPWI:
If you need to remove any files or registry data during the upgrade, add
records to the RemoveFile or RemoveRegistry tables of the newer database.
I've found that to get WiX to put a RemoveFile entry for, in this case, "foo.exe", I needed to add an explicit RemoveFile element:
...
<Component Id="MainExe" Guid="6ee6fda3-6f50-47bf-99b9-6031c720428e">
<!-- Note: This is how to explicitly remove files in an update. -->
<RemoveFile Id="removefile1" On="install" Name="foo.exe"/>
</Component>
...
The ProductVersion we updated in "1.0.1\config.wxi":
C:\tmp\wix_and_msp>diff -u 1.0.0\config.wxi 1.0.1\config.wxi --- 1.0.0\config.wxi Mon May 28 17:33:01 2007 +++ 1.0.1\config.wxi Mon May 28 17:33:03 2007 @@ -6,7 +6,7 @@ <?define ProductName = "ActiveFoo" ?> <?define InstallName = "ActiveFoo" ?> <?define InstallId = "AFoo10" ?> - <?define ProductVersion = "1.0.0" ?> + <?define ProductVersion = "1.0.1" ?> <?define ProductURL = "http://www.example.com/products/activefoo/" ?> </Include>
Now we can build "activefoo-1.0.1.msi":
C:\tmp\wix_and_msp> python make.py -v 101 INFO:make:build target '101' DEBUG:make:running 'candle -nologo activefoo.wxs' in '1.0.1' activefoo.wxs DEBUG:make:running 'light -nologo -o ../activefoo-1.0.1.msi activefoo.wixobj' in '1.0.1' INFO:make:'activefoo-1.0.1.msi' created
ActiveFoo 1.0.1 update
The basic process for building a '.msp' is:
- Get an administrative install of the old version. I hadn't known this
before: An administrative install effective just extracts the file payload
from an .msi into a given directory leaving a lighter .msi with just the
MSI database tables. AFAIK this is the same thing as if you had built an
"uncompressed MSI" -- i.e. one in which
<Package Compressed='no' .../>. make.py will put this in "1.0.1\build\before". - Get an administrative install of the new version. make.py will put this in "1.0.1\build\after".
- Write a WiX file that describes the patch.
- Compile to a Patch Creation Properties (.pcp) file with WiX.
- Compile to a '.msp' file with the "msimsp.exe" utility from the MSI SDK (Part of the Microsoft Platform SDK).
Here is a our WiX file describing the patch (1.0.1\upgrade-1.0.0.wxs) with comments inline:
<?xml version='1.0' encoding='windows-1252'?>
<Wix xmlns='http://schemas.microsoft.com/wix/2003/01/wi'>
<!-- TODO: Update PatchCreation Id for each new patch.
Can we just use WiX's '????????-????-????-????-????????????' ? -->
<PatchCreation Id='e8ee6400-7877-47e4-9519-ce17e3f1d59b'
CleanWorkingFolder='yes'
WholeFilesOnly='no'
AllowMajorVersionMismatches='yes'
AllowProductCodeMismatches='no'>
<PatchInformation Description="ActiveFoo 1.0.1 Patch"
Comments='blah blah'
Manufacturer='Acme'
Languages='1033'
Compressed='yes' />
<!-- TODO: Play with other values of 'Classification'. Does msiexec's
behaviour actually change for different values? -->
<PatchMetadata Description="ActiveFoo 1.0.1 Patch"
DisplayName="ActiveFoo 1.0.1 Patch"
TargetProductName='ActiveFoo 1.0'
ManufacturerName='Acme'
MoreInfoURL='http://www.example.com/products/activefoo'
Classification='Update'
AllowRemoval='yes' />
<!-- From <http://wix.sourceforge.net/manual-wix2/patch_building.htm>
"""
The SequenceStart value is influenced by the number of files that
the previous patch delivered, as well as the number of files that
this patch will deliver. This tells PatchWiz.dll to start assigning
File sequence numbers from this number. So if this patch ships 11
files, and the next patch uses a SequenceStart of 1020, it will step
on the 11th file's assigned sequence number. In this case the next
patch would use a SequenceStart of 1030, and 03 as the patch id to
avoid conflicts with this patch. This scheme helps prevent this by
coordinating the SequenceStart (file sequence numbers) with the
patch sequence number. Also, note that the SequenceStart of the
first patch must be greater than the number of files in the original
installation. If the original installation contained more than 1000
files(rare), then the SequenceStart for the first patch must be set
to a higher value (e.g 2010.)
"""
-->
<!-- Name is max 8 chars. *How* unique does this have to be? -->
<Family Name='Fam101' DiskId='2' MediaSrcProp='AFoo10_2_1_01'
SequenceStart='1010'>
<UpgradeImage Id='AFoo10Upgrade'
SourceFile='after\activefoo-1.0.1.msi'>
<TargetImage Id='AFoo10Target' Order='1' IgnoreMissingFiles='no'
SourceFile='before\activefoo-1.0.0.msi' />
</UpgradeImage>
</Family>
<TargetProductCode Id='cdc5e50f-b490-4a37-8ff6-22e3cb3d690e' />
</PatchCreation>
</Wix>
Use make.py to build the patch:
C:\tmp\wix_and_msp> python make.py -v 101_upgrade
INFO:make:build target '101_upgrade'
DEBUG:make:running 'msiexec /a activefoo-1.0.0.msi TARGETDIR=C:\tmp\wix_and_msp\1.0.1\build\before'
DEBUG:make:running 'msiexec /a activefoo-1.0.1.msi TARGETDIR=C:\tmp\wix_and_msp\1.0.1\build\after'
1 file(s) copied.
DEBUG:make:running 'candle -nologo upgrade.wxs' in 'C:\tmp\wix_and_msp\1.0.1\build'
upgrade.wxs
DEBUG:make:running 'light -nologo upgrade.wixobj' in 'C:\tmp\wix_and_msp\1.0.1\build'
DEBUG:make:running '"C:\Program Files\Microsoft Platform SDK\Samples\SysMgmt\Msi\Patching\MsiMsp.Exe" -s upgrade.pcp -p C:\tmp\wix_and_msp\activefoo-1.0.1-upgrade-1.0.0.msp -l upgrade.log' in 'C:\tmp\wix_and_msp\1.0.1\build'
INFO:make:'activefoo-1.0.1-upgrade-1.0.0.msp' created
INFO:make:To install the update, run:
msiexec /p activefoo-1.0.1-upgrade-1.0.0.msp REINSTALL=ALL REINSTALLMODE=omus
You should now be able to install "activefoo-1.0.1-upgrade-1.0.0.msp" over an ActiveFoo 1.0.0 installation to upgrade to ActiveFoo 1.0.1. Note that some docs out there mention an MSI bug preventing installation of a '.msp' by double-clicking on that. I've found that I am able to install by double-clicking on my WinXP box with Windows Installer V 3.01.4000.1823.
Notes/Limitations
- Having to explicitly put in RemoveFile elements to ensure upgrades remove them is a pain. It would be nice if WiX inferred that automatically. WiX v3 is slated to include "Patch creation support" and "ClickThrough". Perhaps these will go a long way to making all of this easier.
- There are many variables to tweak here that I haven't played with. I haven't deployed any .msp's built as describe here to users on any scale so I there may be gremlins lurking in this procedure.
I'd be happy to hear about others' experiences working with WiX and MSI patches.
1 comments
Posts
