2009-06-29

Stochasticity x Determinism in elementary math

Most of the problems I'm having in studying theoretical topics nowadays stem from the fact that I've had null measure content of Statistics as a math student. I've seen interesting topics as Analysis, and Differential Geometry in undergrad, and useful stuff like Linear Algebra and Computational Linea Algebra in the masters course, but absolutely no Probability and Inference.

Nothing related to Data (not the android) at all. And this is mostly my fault. I knew before the Masters Course I wouldn't be pursuing a PhD in Math, and I knew my Math teachers wouldn't deal with topics I would most likely need in the near future (the one I'm living now).

I'm chasing the lost time, with books in Bayesian Data Analysis and Inference, but I although most of the times I understand what I read, I never seem to grok it. I will, certainly, in time, but time is a commodity a grad student doesn't have - I need Statistics now. As well as Biology, Ecology, Evolutionary Biology, and (why not?) a little Computer Science. So it's fair to say I'm chasing the lost time, and losing.

Prior to leaving I've seen the creation of a new undergrad course, Applied Math, in my former University; it started with concentrations in Finances, Mathematical Biology and Scientific Computing, and it soon became obvious to the faculty that some basic knowledge in Probability and Inference were a must, so it's been introduced as obligatory courses for all concentrations. My former advisor there told me he thought it was overkill at first but soon (in the first year or so) realized how suitable it was.

This is why I think it's a terrific idea (definitely worth spreading) this nice Arthur Benjamin fella presented on this talk on TED.



Obviously I don't think you should rip out everything related to Calculus (I like it, after all :) ). If you are really to grok Probability, you need a strong base in Calculus (from integrals, to maximizing Likelihood functions). But a change in paradigm is definitely well deserved. Our modern western societies are still studying according to old rules. Rules that fit well to the time and reality where they were idealized, but probably are just outdated now. We are a new society on overdrive, with a new (still changing) set of moral rules, new problems and challenges, new perspectives, new age limits. Why stick with the 19th century education philosophies? At least let's realize it's about time to discuss if it's worth changing it. See also another TED talk on this subject.

This all also brings an old question I've always had: is it a good thing that education curricula should be centralized? It's good to know beforehand what people must (might? should?) have learned looking at their curriculum. I'ts useful for the teacher/professor to know what to expect the student to know, and this also applies to the student. I've been bitten before, when the pre-requisites for a course weren't clear. I also have first hand experience of how good and dynamic an improvised class can be, when given by a motivated (and skilled) professor. OTOH, I also have firsthand experience in improvised classes that sucked.

Which is the lesser evil?

2009-06-20

RFC: New paradigm for Ubuntu release cycle

Would it be possible to break Ubuntu into two independent sections: 'Interface' and 'Infrastructure'?

Think about Interface, as everything the ordinary user 'wants' to be updated in the 6 month cycle (GUI, apps, kernel, etc), and Infrastructure as everything else that keeps the engines running, i.e., glibc, and the like.

I think it would be nice if Ubuntu could make 6 month releases be binary compatible with each other, and in order to do that, you must compile everything with the same library versions. I can see why we might want a newer kernel every now and then, but why do we really need to to upgrade glibc every 6 months? Do the applications really need all the new features all the time? Can there be a separation of some sort so that some packages be upgraded, and others preserved? I think it would benefit all, since using the same packages for two or three (or so) releases would decrease the support burden, security- and stability- wise.

It would also have many other side benefits: easier to backport packages, reduce storage space in mirrors, and reduce upgrade issues (since there will be less packages being upgraded). It would definitely be a plus in the corporate side, where ISVs would have a much more stable infrastructure to develop on, and much closer to the two most popular closed OS approaches.

One drawback would be that bugfixes would be harder to get propagated downstream, and longer maintenance cycles sometimes aren't easy to do, specially in open source community, where development is often a continual burst of unordered creativity. It would also not be necessarilly easy to decide where to draw the line (e.g., would poppler/cairo be interface, or infrastructure? and XUL? GTK?).

Still, I think this would be an improvement, and this post is intended to draw healthy discussions in mailing lists and ubuntu related fora.

2009-06-19

My Ubuntu wishlist for the next LTS

I have quite high hopes for the next Ubuntu LTS release. Currently, I expect that:

• LTS releases will never ever be used as a platform for propagation of new technologies...

... such as the pulseaudio fiasco in Hardy. I still don't understand why pulseaudio is a Depends, not a Recommends. I hope this kind of thing will never happen again in an LTS. Seriously.

Ayatana be widespread used in most or all officially supported applications.

I hope it will be used in all default installed applications. I'd like to see this happening as soon as 10.04, the proposed target for the next LTS. This will leave LTS-only users with jaws dropped for sure.

Tracked here.

• Fully featured PIM suite.

Seriously, Evolution is well integrated with Gnome, but I guess (citation needed) Thunderbird has a more widespread user base. It would be good to also have Thunderbird as the center of a whole alternative full PIM suite, that could sync with Evolution, mobile devices and the cloud (Google calendar/contacts, MS Live, Apple, coff... Squirrelmail coff..., etc).

I'd like to sync my PIM applications with mobile devices, like my Palm PDA, and in the future, a cell phone. However gpilot is less than complete (in comparison with e.g. kpilot), and since PalmOS is basically dead upstream, I don't expect much development in that front. OpenSync looked like it would be the way to go, but now it looks like vaporware - it's stable release 0.22 is years old, and the 0.3x development branch progresses slowly. It worked in my tests in Ubuntu 9.04, didn't in 8.04, but it seriously lacks a proper interface. Granted, I'm geekly enough to use it, but it feels very arcane. Definitely not ready for Ubuntu widespread usage (Kubuntu users might disagree, since kitchensync seems interesting, YMMV).

Also, I'd like that my PIM was stored (or synced) in such a way that it would be accessible with either evolution, thunderbird, alpine, my Palm TX and my cellphone. (Akonadi, anyone?)

I know most of what I just mentioned is not Ubuntu specific, but upstream work, but if Canonical considers itself in such a position to drive development, it should use its weight to direct a clean PIM infrastructure, be it opensync finally releasing something new and usable (either 0.2x, or 0.4x), or something else. Please, do it for the next LTS and I'll stick with it for quite a few years.

• Firefox should time out the addons upgrade window

Every now and then, when I open Firefox, it helpfully informs me that there are newer versions of addons I have installed, and offers to upgrade them. But because I cultivate obscene amounts of tabs open, I don't often wait to see firefox opening since it usually takes quite some time. This is when this window makes me throw something out the window; it prevents firefox from opening until I take some measure, either to upgrade or dismiss. This should clearly have a time out and dismiss after a few seconds (I guess 15 would be more than enough for a someone to aim at the OK button, and click it. Even if the user misses the time out, there is a chance of doing it afterwards, so it's not really necessary to do it at the start.

• Home cloud

Sure, why not? Canonical is soon launching Ubuntu One, I sure would like them to bundle a similar sync suite that could sync my PIM data across my desktop and laptop (and my PDA), media storage available over the wire for my music and videos, and various other potential applications. Who has the time to sync their whole hard drive with the cloud? Let me sync my important (small) stuff with the cloud, and the more heavyweight stuff remain at home. Palm is doing this kind of PIM could sync with the new pré smartphone.

And please give me the option to not have to trust anyone with my PIM stuff, it's too important to be left at third party servers, thank you. Home cloud, I always loved you, even though I never met you.

Quickly be released quickly (get it? get it?)

This is something that I miss since I started programming for my project, it will really help me keep my test and workstations up to date. I already love it since the blueprint.

It's targeted to Karmic, AFAICT.

• Separation between 'Interface' and 'Infrastructure'

This is something worth a more in depth explanation, so warrants it its own post.

Upgrading Ubuntu from Hardy to Jaunty directly

I originally intended to upgrade only my laptop because of NM, and waited until Jaunty beta was released for the laptop upgrade, but decided the corrected procedure was safe enough for my purposes to do it also in my desktop.

The following is what I compiled after doing it twice, and correcting the errors from both these attempts. What this means is that this is what I think I should have done on both times, but is not exactly what I did, i.e. on both attempts I had issues, and was able to overcome them.

I decided to follow loosely the upgrade instructions from Debian's Release Notes, and it mostly worked. I found out, unfortunately, that some tweaks were necessary, and in the second time I tried, I could bypass most of the problems I had.

First, a brief review of the process, then a brief description of the problems I had, and how I solved them.

The instructions start with obvious recommendations on backup, notification of users, and contingency plans. I back up fairly often my mission critical files, and did a full backup of my home dir and /etc. Since it was just my personal laptop, I didn't have to bother much about a contingency plan or warning anyone else. Check, check, check.

If you're trying these instructions, you should definitely read the Debian Release Notes sections 4.1 to 4.4, skipping only 4.2.4 and 4.2.5, which are debian specific, before doing anything to your system. Instead of these, make sure you disable every unnoficial repositories (meaning, everything that doesn't seem like archive.ubuntu.com). I made an exception for the partner repo, archive.canonical.com, which is where I get adobe-flashplugin from. If in doubt, read the Repo Howto.

A nice tip from debian is to do the upgrade from inside a screen(1), and also to use script(1) in order to have a full log of everything. Just copy and paste the commands from section 4.5.1:


screen -S upgrade
script -t 2>~/upgrade-jaunty.time -a ~/upgrade-jaunty.script
sudo aptitude update


Then proceed to section 4.5.3 to make sure you won't face an epic fail because of disk space, specially if you have a separate /var partition.

From 4.5.4 onwards, it gets very Debian specific, and although it might be interesting to read, we have to stick with our Ubuntu specific issues. This brings us to the point in which I describe what can go wrong if you follow the Debian instructions.

All the problems I had stemmed from the fact that the packages python and python-minimal was never pulled as a dependency in this whole process, and was only upgraded in the final step, too late for most python apps, that expected a clean python environment, but the above entioned metapackages kept at their 2.5 versions. I was actually surprised that this turned out to be a problem, especially because I thought all these apps were also in Debian.

I say many errors in packages upgrading returning messages like:


Instalando language-selector-common (0.4.2.1) ...
INFO: using unknown version '/usr/bin/python2.6' (debian_defaults not up-to-date?)
Traceback (most recent call last):
File "/usr/bin/fontconfig-voodoo", line 101, in
main()
File "/usr/bin/fontconfig-voodoo", line 56, in main
fc = FontConfig.FontConfigHack()
File "/usr/lib/python2.5/site-packages/LanguageSelector/FontConfig.py", line 36, in __init__
self.li = LocaleInfo("%s/data/languagelist" % datadir)
File "/usr/lib/python2.5/site-packages/LanguageSelector/LocaleInfo.py", line 41, in __init__
et = ElementTree(file="/usr/share/xml/iso-codes/iso_639_3.xml")
File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 546, in __init__
self.parse(file)
File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 586, in parse
parser.feed(data)
File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 1245, in feed
self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 52, column 11


And a lot of other packages with less verbose errors, containing only the first INFO warning line. Crossing fingers wasn't enough, and when I saw most of the interface apps were hosed, I got this kind of errors when trying to run apps from the command line:


Traceback (most recent call last): File "/usr/bin/software-properties-gtk", line 44, in
from softwareproperties.gtk.SoftwarePropertiesGtk import SoftwarePropertiesGtk
ImportError: No module named softwareproperties.gtk.SoftwarePropertiesGtk
These messages made it somewhat clear that python modules were missing, but I made sure their packages were in fact installed. Reinstalling ( sudo aptitude reinstall [package] ) the package made the problem disappear for the application. I'm not knowledgeable in Python, but I think this has something to do with python byte compiling the scripts during package installation (please correct me if I'm wrong in the comments).

Luckilly I didn't have to reinstall the whole OS, just the python apps that were misbehaving. I had to go back and forth through dependency trees in order to find everything that depended on python and python-minimal, and reinstall them, and there were a lot of them. This was a good opportunity to use the script(1) log that was created in the process. It took me more than one hour, but I was able to fix everything without reformatting and reinstalling (kudos to aptitude interactive interface!).

I'm not sure why this issue affects Ubuntu, and not Debian, but I did run into this in my two upgrade adventures, so I thought it was worth to share the (probable) correct routine.

Now that the issue is covered, let's see what the upgrade routine would look like.

The first step in Debian instructions, is to upgrade apt, which will also pull libc and other essential stuff. However from the experience above, it would be useful to also upgrade python or python-minimal as soon as possible. It was only later that I found out I could probably have done both in a single step:

sudo aptitude install ubuntu-minimal

This is a meta-package that depends on apt, and python/python-minimal, so I think it would suffice. It will pull python2.6 and python-minimal2.6 as dependencies, so all python apps will already have their bed made, when it's required.

As a second step, let's make sure a second base meta-package kicks in, which will pull another batch of essential stuff:

sudo aptitude install ubuntu-standard

After that, you can safely issue the first step of the Debian release notes upgrade:

sudo aptitude safe-upgrade

Depending on which packages you you have installed, you might need to repeat this step until it upgrades no more packages. This is because although aptitude has a great dependency resolution algorithm, it's not perfect (but this is being worked on ;-) ).

When you come to the point that safe-upgrade can't upgrade any more packages, it's the time when most of the other stuff must be resolved, which is:

sudo aptitude dist-upgrade

This command should be the first time where packages are removed by conflict resolutions. Again, you might need to issue it more than once, but there's nothing to worry (unless you get errors, but they are being logged by script, and you can go over it whenever you want).

Now, as a final step, make sure the basic meta-package is upgraded, if it wasn't already:

sudo aptitude install ubuntu-desktop

Your kernel will be upgraded in this step, if you use the generic flavor, but your current version won't be removed. You might want to remove it manually afterwards. You can keep more than one, but it's mostly clutter, specially if you, like me, have a separate /boot partition.

Now you can logout of script, logout of screen and boot your new Jaunty. Computer Janitor (System>Administration>Computer Janitor, or whatever it is translated to for your language) will help you cleanup afterwards.

Note: Expect this mini howto to be updated.

2009-06-18

Why I upgraded my Hardy to Jaunty

l really tried to hold my upgrade mania and stick with the LTS releases of Ubuntu. I was convinced that LTS was the way to go for academic purposes, where one needs the stability of a non-chaning environment to work on his own projects, instead of working on his system, until I decided to upgrade my Ubuntu 8.04 installs to Ubuntu 9.04. A few reasons drove me to this decision, but I still think 8.04 was good enough for my purposes, and my petty problems were manageable.

I was constantly annoyed by a few usability bugs in evince, since viewing PDFs is a frequent thing in my daily routine. I managed to bypass completely any pseudo-need I had to upgrade to Intrepid 8.10 since my main motivation was a group of fixes for evince, and Perl 5.10.

Most of these needs were either not critical, or could be easily solved by a backport (which I used to scratch my poppler/evince itch). Other positive aspects of 9.04 that directly affect me were the new NetworkManager with improved wireless support as well as support to GSM modems, that makes a Huawei modem I borrowed work in a truly Plug n Play fashion.

But Perl 5.10, poppler/evince fixes, new NetworkManager were all the rationalizations I made to justify the hassle of a major, untested, undocumented dist-upgrade of this sort. Truth be told, I was mostly interested in checking out Notify-OSD Ayatana.

Mark Shuttleworth announced bold plans for a new notification system that could at the same time look great and clean up the clutter in the notification area. I gave some serious thought about why his original idea was a good thing, as well as a bad thing. I was not alone. Although I never expressed my opinions on the topic publicly, they are often partially mirrored by others, who can express in a better (clearer and more concise way) like Celeste Lyn Paul and Scott Kitterman.

Most important is the fact that the original proposal was revised and what I consider some of the most controversial sections were abandoned in the implementation that was released with 9.04 (notably the no history requisite).

The final version is good enough to provide a compelling case for an usability experiment - it's good enough to discuss about what's been around so far. I think everyone always thought about notifications in a basically equal way, as if there was a 'right' way to do it. Mark is trying to do it differently, and acks he may be wrong. I disagree with some points, but the major points are valid enough for a trial.

Not all is working, since the applications must be aware of this, and it appears that Canonical only developed the proper plugins for Evolution and Pidgin. I'd like to see similar integration for other applications, such as Mozilla Thunderbird, and hardware discovery notification (flash drives, camera, etc). I'm sure these will come soon, though. I also hope that a history GUI will be created in time. Hopefully something like the Log file Viewer app, that keeps logs separate by sources, and dates. The way it is now, it's very hard to even check out the log, since it's a dump of every notification from every application, including pidgin's messages, so it's growing very fast for me. Also, the notifications from Evolution are less than useful, since it only tells that I have new message, but doesn't point me to which account the new message is from, like pidgin does.

I've been using Jaunty for more than two months now (since beta), and I think I'm used to the new notification system by now. I switched my main IM app from amsn to pidgin, and get notifications of logins and messages. I get notifications of the bzr commits for my programming routine (which is not necessary, since I'm the only one developing it, but it's nice anyway), I get notification for hardware buttons, like sound volume, wifi connection and screen brightness. So far I like it, and I'm glad upgraded. It feels modern and cool.

Update: s/Notify-OSD/Ayatana/ and links.