I recently decided not to create a logging class of my own, since there must already be a lot out there. Not surprisingly this proved to be a correct decision.
I started looking for packages already available for Ubuntu, and found Log::Handler, Log::Log4perl, and Log::LogLite. I started drafting prototypes for them.
2009-07-14
2009-07-01
F: {alignments} -> {tree topologies} is continuous
If the title is not clear enough, it means that the function that maps alignments of genetic information to phylogenetic tree topologies is continuous. Well, kind of. It's a scientific proof, not a mathematical one, so it's based on evidence. But as far as evidence goes, it makes a compelling argument in favor of the thesis.
The paper is Wong, et al, 2008 (Science) "Alignment Uncertainty and Genomic Analysis". DOI: 10.1126/science.1151532 .
The relevance IMHO of this paper is not to estabilish that 'common sense' is usually correct (although it is important in its own matter to know when it's possible to infer things solely from ituition). The authors make the case for interpreting alignments themselves as random variables, and in doing so, they conclude that (in a very precise way) small variations in alignments produce small variations in tree topologies inferred from those alignments. More so, they indicate this result is robust in respect to methods of inferring phylogenetic trees.
For the mathematically inclined, they defined a metric in the set of alignments, a metric in tree topologies, generated alignments (via MCMC) that were near the reference alignment, and observed that the resulting trees all had topologies near the reference topology in the given metric.
This CPU intensive work was done for alignments of whole genomes, with several alignment techniques and tree inferring algorithms, with similar results for most cases. Definitely worth a read.
(UPDATE: fixed title typo)
The paper is Wong, et al, 2008 (Science) "Alignment Uncertainty and Genomic Analysis". DOI: 10.1126/science.1151532 .
The relevance IMHO of this paper is not to estabilish that 'common sense' is usually correct (although it is important in its own matter to know when it's possible to infer things solely from ituition). The authors make the case for interpreting alignments themselves as random variables, and in doing so, they conclude that (in a very precise way) small variations in alignments produce small variations in tree topologies inferred from those alignments. More so, they indicate this result is robust in respect to methods of inferring phylogenetic trees.
For the mathematically inclined, they defined a metric in the set of alignments, a metric in tree topologies, generated alignments (via MCMC) that were near the reference alignment, and observed that the resulting trees all had topologies near the reference topology in the given metric.
This CPU intensive work was done for alignments of whole genomes, with several alignment techniques and tree inferring algorithms, with similar results for most cases. Definitely worth a read.
(UPDATE: fixed title typo)
Migrating mail filters from MUA to MDA
I like freedom of choice. A lot. I even like to be able to choose different alternatives for common tasks during the course of the day. Call it a whim, but when it comes to mail programs, I can never decide, really, what's the best for me. Maybe this is because I never found one that suits all my needs.
Nevertheless, I used to use KDE before I became an Ubuntu user, and from that time I kept my whole PIM suite in the KDE stack from Dapper 6.06 to Hardy 8.04. Enough is enough, and if I'm not using KDE I wasting precious RAM and cycles keeping all those libraries loaded just for kmail and amarok. I will surely miss amarok, and kmail was great but it's time to move on.
I wanted my PIM to sync both from my Palm PDA and my home desktop, laptop, and workstation at work. I'm not a big fan of third party cloud services, if I can't be sure who has access to my data, so I decided to go back in time and migrate from a fully-featured-MUA centered setup to a home cloud-like setup. Provided I can access my PIM data from whatever MUA I choose, I can use pretty much anything that talks IMAP (which is anything these days). I'll be truly free, then, to use a great GUI app for the daily routine, a light GUI app if I need RAM to do more resource intensive work, and good old pine (ressurected as alpine) if I need to access my mail remotely, via ssh.
The project is basically to store everything in the desktop an imap server in my desktop machine, POP3 accounts fetched by fetchmail , and fed to an MDA (procmail or maildrop).
I already had kmail store my mails in maildir format, so installing an imapd server was a matter of installing the package. After a brief poll in ubuntu-users mailing list I installed dovecot, and kept using kmail for a year or so later to pull my gmail mail via POP3. Now, I'm only accesssing my mail via IMAP since the upgrade to Jaunty, so my home cloud project is kind of stalled.
I'm in the process of testing gnome-pilot and opensync to keep my PIM and Palm PDA synced.
All that remains is to migrate my mail filters from kmail to an MDA (be it procmail or maildrop), and then use fetchmail to get mail via POP. This is where I'm stomped. So far, the only think I've found that resembles a solution is this perl script, except it's the oposite of what I want. I'm not even sure I can use it to create the reverse solution I need.
I've drafted a little perl script, and I'll probably have to do it from scratch since my needs are pretty modest. I have 100+ filters (too many to convert manually), but most or all are the kind of 'match and move to a folder' simple filters. I'm just trying to scratch my own itch, not create a general purpose application, so it's probabl y a matter of simple Perl, once I decide what exactly will I output to.
Since time is always in short supply, if anyone knows a way to convert filters to either procmail or maildrop, I'll be glad to hear it. I also welcome suggestions on which of these MDAs to use.
Nevertheless, I used to use KDE before I became an Ubuntu user, and from that time I kept my whole PIM suite in the KDE stack from Dapper 6.06 to Hardy 8.04. Enough is enough, and if I'm not using KDE I wasting precious RAM and cycles keeping all those libraries loaded just for kmail and amarok. I will surely miss amarok, and kmail was great but it's time to move on.
I wanted my PIM to sync both from my Palm PDA and my home desktop, laptop, and workstation at work. I'm not a big fan of third party cloud services, if I can't be sure who has access to my data, so I decided to go back in time and migrate from a fully-featured-MUA centered setup to a home cloud-like setup. Provided I can access my PIM data from whatever MUA I choose, I can use pretty much anything that talks IMAP (which is anything these days). I'll be truly free, then, to use a great GUI app for the daily routine, a light GUI app if I need RAM to do more resource intensive work, and good old pine (ressurected as alpine) if I need to access my mail remotely, via ssh.
The project is basically to store everything in the desktop an imap server in my desktop machine, POP3 accounts fetched by fetchmail , and fed to an MDA (procmail or maildrop).
I already had kmail store my mails in maildir format, so installing an imapd server was a matter of installing the package. After a brief poll in ubuntu-users mailing list I installed dovecot, and kept using kmail for a year or so later to pull my gmail mail via POP3. Now, I'm only accesssing my mail via IMAP since the upgrade to Jaunty, so my home cloud project is kind of stalled.
I'm in the process of testing gnome-pilot and opensync to keep my PIM and Palm PDA synced.
All that remains is to migrate my mail filters from kmail to an MDA (be it procmail or maildrop), and then use fetchmail to get mail via POP. This is where I'm stomped. So far, the only think I've found that resembles a solution is this perl script, except it's the oposite of what I want. I'm not even sure I can use it to create the reverse solution I need.
I've drafted a little perl script, and I'll probably have to do it from scratch since my needs are pretty modest. I have 100+ filters (too many to convert manually), but most or all are the kind of 'match and move to a folder' simple filters. I'm just trying to scratch my own itch, not create a general purpose application, so it's probabl y a matter of simple Perl, once I decide what exactly will I output to.
Since time is always in short supply, if anyone knows a way to convert filters to either procmail or maildrop, I'll be glad to hear it. I also welcome suggestions on which of these MDAs to use.
2009-06-29
Stochasticity x Determinism in elementary math
Most of the problems I'm having in studying theoretical topics nowadays stem from the fact that I've had null measure content of Statistics as a math student. I've seen interesting topics as Analysis, and Differential Geometry in undergrad, and useful stuff like Linear Algebra and Computational Linea Algebra in the masters course, but absolutely no Probability and Inference.
Nothing related to Data (not the android) at all. And this is mostly my fault. I knew before the Masters Course I wouldn't be pursuing a PhD in Math, and I knew my Math teachers wouldn't deal with topics I would most likely need in the near future (the one I'm living now).
I'm chasing the lost time, with books in Bayesian Data Analysis and Inference, but I although most of the times I understand what I read, I never seem to grok it. I will, certainly, in time, but time is a commodity a grad student doesn't have - I need Statistics now. As well as Biology, Ecology, Evolutionary Biology, and (why not?) a little Computer Science. So it's fair to say I'm chasing the lost time, and losing.
Prior to leaving I've seen the creation of a new undergrad course, Applied Math, in my former University; it started with concentrations in Finances, Mathematical Biology and Scientific Computing, and it soon became obvious to the faculty that some basic knowledge in Probability and Inference were a must, so it's been introduced as obligatory courses for all concentrations. My former advisor there told me he thought it was overkill at first but soon (in the first year or so) realized how suitable it was.
This is why I think it's a terrific idea (definitely worth spreading) this nice Arthur Benjamin fella presented on this talk on TED.
Obviously I don't think you should rip out everything related to Calculus (I like it, after all :) ). If you are really to grok Probability, you need a strong base in Calculus (from integrals, to maximizing Likelihood functions). But a change in paradigm is definitely well deserved. Our modern western societies are still studying according to old rules. Rules that fit well to the time and reality where they were idealized, but probably are just outdated now. We are a new society on overdrive, with a new (still changing) set of moral rules, new problems and challenges, new perspectives, new age limits. Why stick with the 19th century education philosophies? At least let's realize it's about time to discuss if it's worth changing it. See also another TED talk on this subject.
This all also brings an old question I've always had: is it a good thing that education curricula should be centralized? It's good to know beforehand what people must (might? should?) have learned looking at their curriculum. I'ts useful for the teacher/professor to know what to expect the student to know, and this also applies to the student. I've been bitten before, when the pre-requisites for a course weren't clear. I also have first hand experience of how good and dynamic an improvised class can be, when given by a motivated (and skilled) professor. OTOH, I also have firsthand experience in improvised classes that sucked.
Which is the lesser evil?
Nothing related to Data (not the android) at all. And this is mostly my fault. I knew before the Masters Course I wouldn't be pursuing a PhD in Math, and I knew my Math teachers wouldn't deal with topics I would most likely need in the near future (the one I'm living now).
I'm chasing the lost time, with books in Bayesian Data Analysis and Inference, but I although most of the times I understand what I read, I never seem to grok it. I will, certainly, in time, but time is a commodity a grad student doesn't have - I need Statistics now. As well as Biology, Ecology, Evolutionary Biology, and (why not?) a little Computer Science. So it's fair to say I'm chasing the lost time, and losing.
Prior to leaving I've seen the creation of a new undergrad course, Applied Math, in my former University; it started with concentrations in Finances, Mathematical Biology and Scientific Computing, and it soon became obvious to the faculty that some basic knowledge in Probability and Inference were a must, so it's been introduced as obligatory courses for all concentrations. My former advisor there told me he thought it was overkill at first but soon (in the first year or so) realized how suitable it was.
This is why I think it's a terrific idea (definitely worth spreading) this nice Arthur Benjamin fella presented on this talk on TED.
Obviously I don't think you should rip out everything related to Calculus (I like it, after all :) ). If you are really to grok Probability, you need a strong base in Calculus (from integrals, to maximizing Likelihood functions). But a change in paradigm is definitely well deserved. Our modern western societies are still studying according to old rules. Rules that fit well to the time and reality where they were idealized, but probably are just outdated now. We are a new society on overdrive, with a new (still changing) set of moral rules, new problems and challenges, new perspectives, new age limits. Why stick with the 19th century education philosophies? At least let's realize it's about time to discuss if it's worth changing it. See also another TED talk on this subject.
This all also brings an old question I've always had: is it a good thing that education curricula should be centralized? It's good to know beforehand what people must (might? should?) have learned looking at their curriculum. I'ts useful for the teacher/professor to know what to expect the student to know, and this also applies to the student. I've been bitten before, when the pre-requisites for a course weren't clear. I also have first hand experience of how good and dynamic an improvised class can be, when given by a motivated (and skilled) professor. OTOH, I also have firsthand experience in improvised classes that sucked.
Which is the lesser evil?
2009-06-20
RFC: New paradigm for Ubuntu release cycle
Would it be possible to break Ubuntu into two independent sections: 'Interface' and 'Infrastructure'?
Think about Interface, as everything the ordinary user 'wants' to be updated in the 6 month cycle (GUI, apps, kernel, etc), and Infrastructure as everything else that keeps the engines running, i.e., glibc, and the like.
I think it would be nice if Ubuntu could make 6 month releases be binary compatible with each other, and in order to do that, you must compile everything with the same library versions. I can see why we might want a newer kernel every now and then, but why do we really need to to upgrade glibc every 6 months? Do the applications really need all the new features all the time? Can there be a separation of some sort so that some packages be upgraded, and others preserved? I think it would benefit all, since using the same packages for two or three (or so) releases would decrease the support burden, security- and stability- wise.
It would also have many other side benefits: easier to backport packages, reduce storage space in mirrors, and reduce upgrade issues (since there will be less packages being upgraded). It would definitely be a plus in the corporate side, where ISVs would have a much more stable infrastructure to develop on, and much closer to the two most popular closed OS approaches.
One drawback would be that bugfixes would be harder to get propagated downstream, and longer maintenance cycles sometimes aren't easy to do, specially in open source community, where development is often a continual burst of unordered creativity. It would also not be necessarilly easy to decide where to draw the line (e.g., would poppler/cairo be interface, or infrastructure? and XUL? GTK?).
Still, I think this would be an improvement, and this post is intended to draw healthy discussions in mailing lists and ubuntu related fora.
Think about Interface, as everything the ordinary user 'wants' to be updated in the 6 month cycle (GUI, apps, kernel, etc), and Infrastructure as everything else that keeps the engines running, i.e., glibc, and the like.
I think it would be nice if Ubuntu could make 6 month releases be binary compatible with each other, and in order to do that, you must compile everything with the same library versions. I can see why we might want a newer kernel every now and then, but why do we really need to to upgrade glibc every 6 months? Do the applications really need all the new features all the time? Can there be a separation of some sort so that some packages be upgraded, and others preserved? I think it would benefit all, since using the same packages for two or three (or so) releases would decrease the support burden, security- and stability- wise.
It would also have many other side benefits: easier to backport packages, reduce storage space in mirrors, and reduce upgrade issues (since there will be less packages being upgraded). It would definitely be a plus in the corporate side, where ISVs would have a much more stable infrastructure to develop on, and much closer to the two most popular closed OS approaches.
One drawback would be that bugfixes would be harder to get propagated downstream, and longer maintenance cycles sometimes aren't easy to do, specially in open source community, where development is often a continual burst of unordered creativity. It would also not be necessarilly easy to decide where to draw the line (e.g., would poppler/cairo be interface, or infrastructure? and XUL? GTK?).
Still, I think this would be an improvement, and this post is intended to draw healthy discussions in mailing lists and ubuntu related fora.
2009-06-19
My Ubuntu wishlist for the next LTS
I have quite high hopes for the next Ubuntu LTS release. Currently, I expect that:
• LTS releases will never ever be used as a platform for propagation of new technologies...
... such as the pulseaudio fiasco in Hardy. I still don't understand why pulseaudio is a Depends, not a Recommends. I hope this kind of thing will never happen again in an LTS. Seriously.
• Ayatana be widespread used in most or all officially supported applications.
I hope it will be used in all default installed applications. I'd like to see this happening as soon as 10.04, the proposed target for the next LTS. This will leave LTS-only users with jaws dropped for sure.
Tracked here.
• Fully featured PIM suite.
Seriously, Evolution is well integrated with Gnome, but I guess (citation needed) Thunderbird has a more widespread user base. It would be good to also have Thunderbird as the center of a whole alternative full PIM suite, that could sync with Evolution, mobile devices and the cloud (Google calendar/contacts, MS Live, Apple, coff... Squirrelmail coff..., etc).
I'd like to sync my PIM applications with mobile devices, like my Palm PDA, and in the future, a cell phone. However gpilot is less than complete (in comparison with e.g. kpilot), and since PalmOS is basically dead upstream, I don't expect much development in that front. OpenSync looked like it would be the way to go, but now it looks like vaporware - it's stable release 0.22 is years old, and the 0.3x development branch progresses slowly. It worked in my tests in Ubuntu 9.04, didn't in 8.04, but it seriously lacks a proper interface. Granted, I'm geekly enough to use it, but it feels very arcane. Definitely not ready for Ubuntu widespread usage (Kubuntu users might disagree, since kitchensync seems interesting, YMMV).
Also, I'd like that my PIM was stored (or synced) in such a way that it would be accessible with either evolution, thunderbird, alpine, my Palm TX and my cellphone. (Akonadi, anyone?)
I know most of what I just mentioned is not Ubuntu specific, but upstream work, but if Canonical considers itself in such a position to drive development, it should use its weight to direct a clean PIM infrastructure, be it opensync finally releasing something new and usable (either 0.2x, or 0.4x), or something else. Please, do it for the next LTS and I'll stick with it for quite a few years.
• Firefox should time out the addons upgrade window
Every now and then, when I open Firefox, it helpfully informs me that there are newer versions of addons I have installed, and offers to upgrade them. But because I cultivate obscene amounts of tabs open, I don't often wait to see firefox opening since it usually takes quite some time. This is when this window makes me throw something out the window; it prevents firefox from opening until I take some measure, either to upgrade or dismiss. This should clearly have a time out and dismiss after a few seconds (I guess 15 would be more than enough for a someone to aim at the OK button, and click it. Even if the user misses the time out, there is a chance of doing it afterwards, so it's not really necessary to do it at the start.
• Home cloud
Sure, why not? Canonical is soon launching Ubuntu One, I sure would like them to bundle a similar sync suite that could sync my PIM data across my desktop and laptop (and my PDA), media storage available over the wire for my music and videos, and various other potential applications. Who has the time to sync their whole hard drive with the cloud? Let me sync my important (small) stuff with the cloud, and the more heavyweight stuff remain at home. Palm is doing this kind of PIM could sync with the new pré smartphone.
And please give me the option to not have to trust anyone with my PIM stuff, it's too important to be left at third party servers, thank you. Home cloud, I always loved you, even though I never met you.
• Quickly be released quickly (get it? get it?)
This is something that I miss since I started programming for my project, it will really help me keep my test and workstations up to date. I already love it since the blueprint.
It's targeted to Karmic, AFAICT.
• Separation between 'Interface' and 'Infrastructure'
This is something worth a more in depth explanation, so warrants it its own post.
• LTS releases will never ever be used as a platform for propagation of new technologies...
... such as the pulseaudio fiasco in Hardy. I still don't understand why pulseaudio is a Depends, not a Recommends. I hope this kind of thing will never happen again in an LTS. Seriously.
• Ayatana be widespread used in most or all officially supported applications.
I hope it will be used in all default installed applications. I'd like to see this happening as soon as 10.04, the proposed target for the next LTS. This will leave LTS-only users with jaws dropped for sure.
Tracked here.
• Fully featured PIM suite.
Seriously, Evolution is well integrated with Gnome, but I guess (citation needed) Thunderbird has a more widespread user base. It would be good to also have Thunderbird as the center of a whole alternative full PIM suite, that could sync with Evolution, mobile devices and the cloud (Google calendar/contacts, MS Live, Apple, coff... Squirrelmail coff..., etc).
I'd like to sync my PIM applications with mobile devices, like my Palm PDA, and in the future, a cell phone. However gpilot is less than complete (in comparison with e.g. kpilot), and since PalmOS is basically dead upstream, I don't expect much development in that front. OpenSync looked like it would be the way to go, but now it looks like vaporware - it's stable release 0.22 is years old, and the 0.3x development branch progresses slowly. It worked in my tests in Ubuntu 9.04, didn't in 8.04, but it seriously lacks a proper interface. Granted, I'm geekly enough to use it, but it feels very arcane. Definitely not ready for Ubuntu widespread usage (Kubuntu users might disagree, since kitchensync seems interesting, YMMV).
Also, I'd like that my PIM was stored (or synced) in such a way that it would be accessible with either evolution, thunderbird, alpine, my Palm TX and my cellphone. (Akonadi, anyone?)
I know most of what I just mentioned is not Ubuntu specific, but upstream work, but if Canonical considers itself in such a position to drive development, it should use its weight to direct a clean PIM infrastructure, be it opensync finally releasing something new and usable (either 0.2x, or 0.4x), or something else. Please, do it for the next LTS and I'll stick with it for quite a few years.
• Firefox should time out the addons upgrade window
Every now and then, when I open Firefox, it helpfully informs me that there are newer versions of addons I have installed, and offers to upgrade them. But because I cultivate obscene amounts of tabs open, I don't often wait to see firefox opening since it usually takes quite some time. This is when this window makes me throw something out the window; it prevents firefox from opening until I take some measure, either to upgrade or dismiss. This should clearly have a time out and dismiss after a few seconds (I guess 15 would be more than enough for a someone to aim at the OK button, and click it. Even if the user misses the time out, there is a chance of doing it afterwards, so it's not really necessary to do it at the start.
• Home cloud
Sure, why not? Canonical is soon launching Ubuntu One, I sure would like them to bundle a similar sync suite that could sync my PIM data across my desktop and laptop (and my PDA), media storage available over the wire for my music and videos, and various other potential applications. Who has the time to sync their whole hard drive with the cloud? Let me sync my important (small) stuff with the cloud, and the more heavyweight stuff remain at home. Palm is doing this kind of PIM could sync with the new pré smartphone.
And please give me the option to not have to trust anyone with my PIM stuff, it's too important to be left at third party servers, thank you. Home cloud, I always loved you, even though I never met you.
• Quickly be released quickly (get it? get it?)
This is something that I miss since I started programming for my project, it will really help me keep my test and workstations up to date. I already love it since the blueprint.
It's targeted to Karmic, AFAICT.
• Separation between 'Interface' and 'Infrastructure'
This is something worth a more in depth explanation, so warrants it its own post.
Upgrading Ubuntu from Hardy to Jaunty directly
I originally intended to upgrade only my laptop because of NM, and waited until Jaunty beta was released for the laptop upgrade, but decided the corrected procedure was safe enough for my purposes to do it also in my desktop.
The following is what I compiled after doing it twice, and correcting the errors from both these attempts. What this means is that this is what I think I should have done on both times, but is not exactly what I did, i.e. on both attempts I had issues, and was able to overcome them.
I decided to follow loosely the upgrade instructions from Debian's Release Notes, and it mostly worked. I found out, unfortunately, that some tweaks were necessary, and in the second time I tried, I could bypass most of the problems I had.
First, a brief review of the process, then a brief description of the problems I had, and how I solved them.
The instructions start with obvious recommendations on backup, notification of users, and contingency plans. I back up fairly often my mission critical files, and did a full backup of my home dir and /etc. Since it was just my personal laptop, I didn't have to bother much about a contingency plan or warning anyone else. Check, check, check.
If you're trying these instructions, you should definitely read the Debian Release Notes sections 4.1 to 4.4, skipping only 4.2.4 and 4.2.5, which are debian specific, before doing anything to your system. Instead of these, make sure you disable every unnoficial repositories (meaning, everything that doesn't seem like archive.ubuntu.com). I made an exception for the partner repo, archive.canonical.com, which is where I get adobe-flashplugin from. If in doubt, read the Repo Howto.
A nice tip from debian is to do the upgrade from inside a screen(1), and also to use script(1) in order to have a full log of everything. Just copy and paste the commands from section 4.5.1:
Then proceed to section 4.5.3 to make sure you won't face an epic fail because of disk space, specially if you have a separate /var partition.
From 4.5.4 onwards, it gets very Debian specific, and although it might be interesting to read, we have to stick with our Ubuntu specific issues. This brings us to the point in which I describe what can go wrong if you follow the Debian instructions.
All the problems I had stemmed from the fact that the packages python and python-minimal was never pulled as a dependency in this whole process, and was only upgraded in the final step, too late for most python apps, that expected a clean python environment, but the above entioned metapackages kept at their 2.5 versions. I was actually surprised that this turned out to be a problem, especially because I thought all these apps were also in Debian.
I say many errors in packages upgrading returning messages like:
And a lot of other packages with less verbose errors, containing only the first INFO warning line. Crossing fingers wasn't enough, and when I saw most of the interface apps were hosed, I got this kind of errors when trying to run apps from the command line:
Luckilly I didn't have to reinstall the whole OS, just the python apps that were misbehaving. I had to go back and forth through dependency trees in order to find everything that depended on python and python-minimal, and reinstall them, and there were a lot of them. This was a good opportunity to use the script(1) log that was created in the process. It took me more than one hour, but I was able to fix everything without reformatting and reinstalling (kudos to aptitude interactive interface!).
I'm not sure why this issue affects Ubuntu, and not Debian, but I did run into this in my two upgrade adventures, so I thought it was worth to share the (probable) correct routine.
Now that the issue is covered, let's see what the upgrade routine would look like.
The first step in Debian instructions, is to upgrade apt, which will also pull libc and other essential stuff. However from the experience above, it would be useful to also upgrade python or python-minimal as soon as possible. It was only later that I found out I could probably have done both in a single step:
This is a meta-package that depends on apt, and python/python-minimal, so I think it would suffice. It will pull python2.6 and python-minimal2.6 as dependencies, so all python apps will already have their bed made, when it's required.
As a second step, let's make sure a second base meta-package kicks in, which will pull another batch of essential stuff:
After that, you can safely issue the first step of the Debian release notes upgrade:
Depending on which packages you you have installed, you might need to repeat this step until it upgrades no more packages. This is because although aptitude has a great dependency resolution algorithm, it's not perfect (but this is being worked on ;-) ).
When you come to the point that safe-upgrade can't upgrade any more packages, it's the time when most of the other stuff must be resolved, which is:
This command should be the first time where packages are removed by conflict resolutions. Again, you might need to issue it more than once, but there's nothing to worry (unless you get errors, but they are being logged by script, and you can go over it whenever you want).
Now, as a final step, make sure the basic meta-package is upgraded, if it wasn't already:
Your kernel will be upgraded in this step, if you use the generic flavor, but your current version won't be removed. You might want to remove it manually afterwards. You can keep more than one, but it's mostly clutter, specially if you, like me, have a separate /boot partition.
Now you can logout of script, logout of screen and boot your new Jaunty. Computer Janitor (System>Administration>Computer Janitor, or whatever it is translated to for your language) will help you cleanup afterwards.
Note: Expect this mini howto to be updated.
The following is what I compiled after doing it twice, and correcting the errors from both these attempts. What this means is that this is what I think I should have done on both times, but is not exactly what I did, i.e. on both attempts I had issues, and was able to overcome them.
I decided to follow loosely the upgrade instructions from Debian's Release Notes, and it mostly worked. I found out, unfortunately, that some tweaks were necessary, and in the second time I tried, I could bypass most of the problems I had.
First, a brief review of the process, then a brief description of the problems I had, and how I solved them.
The instructions start with obvious recommendations on backup, notification of users, and contingency plans. I back up fairly often my mission critical files, and did a full backup of my home dir and /etc. Since it was just my personal laptop, I didn't have to bother much about a contingency plan or warning anyone else. Check, check, check.
If you're trying these instructions, you should definitely read the Debian Release Notes sections 4.1 to 4.4, skipping only 4.2.4 and 4.2.5, which are debian specific, before doing anything to your system. Instead of these, make sure you disable every unnoficial repositories (meaning, everything that doesn't seem like archive.ubuntu.com). I made an exception for the partner repo, archive.canonical.com, which is where I get adobe-flashplugin from. If in doubt, read the Repo Howto.
A nice tip from debian is to do the upgrade from inside a screen(1), and also to use script(1) in order to have a full log of everything. Just copy and paste the commands from section 4.5.1:
screen -S upgrade
script -t 2>~/upgrade-jaunty.time -a ~/upgrade-jaunty.script
sudo aptitude update
Then proceed to section 4.5.3 to make sure you won't face an epic fail because of disk space, specially if you have a separate /var partition.
From 4.5.4 onwards, it gets very Debian specific, and although it might be interesting to read, we have to stick with our Ubuntu specific issues. This brings us to the point in which I describe what can go wrong if you follow the Debian instructions.
All the problems I had stemmed from the fact that the packages python and python-minimal was never pulled as a dependency in this whole process, and was only upgraded in the final step, too late for most python apps, that expected a clean python environment, but the above entioned metapackages kept at their 2.5 versions. I was actually surprised that this turned out to be a problem, especially because I thought all these apps were also in Debian.
I say many errors in packages upgrading returning messages like:
Instalando language-selector-common (0.4.2.1) ...
INFO: using unknown version '/usr/bin/python2.6' (debian_defaults not up-to-date?)
Traceback (most recent call last):
File "/usr/bin/fontconfig-voodoo", line 101, in
main()
File "/usr/bin/fontconfig-voodoo", line 56, in main
fc = FontConfig.FontConfigHack()
File "/usr/lib/python2.5/site-packages/LanguageSelector/FontConfig.py", line 36, in __init__
self.li = LocaleInfo("%s/data/languagelist" % datadir)
File "/usr/lib/python2.5/site-packages/LanguageSelector/LocaleInfo.py", line 41, in __init__
et = ElementTree(file="/usr/share/xml/iso-codes/iso_639_3.xml")
File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 546, in __init__
self.parse(file)
File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 586, in parse
parser.feed(data)
File "/usr/lib/python2.5/xml/etree/ElementTree.py", line 1245, in feed
self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 52, column 11
And a lot of other packages with less verbose errors, containing only the first INFO warning line. Crossing fingers wasn't enough, and when I saw most of the interface apps were hosed, I got this kind of errors when trying to run apps from the command line:
These messages made it somewhat clear that python modules were missing, but I made sure their packages were in fact installed. Reinstalling ( sudo aptitude reinstall [package] ) the package made the problem disappear for the application. I'm not knowledgeable in Python, but I think this has something to do with python byte compiling the scripts during package installation (please correct me if I'm wrong in the comments).
Traceback (most recent call last): File "/usr/bin/software-properties-gtk", line 44, in
from softwareproperties.gtk.SoftwarePropertiesGtk import SoftwarePropertiesGtk
ImportError: No module named softwareproperties.gtk.SoftwarePropertiesGtk
Luckilly I didn't have to reinstall the whole OS, just the python apps that were misbehaving. I had to go back and forth through dependency trees in order to find everything that depended on python and python-minimal, and reinstall them, and there were a lot of them. This was a good opportunity to use the script(1) log that was created in the process. It took me more than one hour, but I was able to fix everything without reformatting and reinstalling (kudos to aptitude interactive interface!).
I'm not sure why this issue affects Ubuntu, and not Debian, but I did run into this in my two upgrade adventures, so I thought it was worth to share the (probable) correct routine.
Now that the issue is covered, let's see what the upgrade routine would look like.
The first step in Debian instructions, is to upgrade apt, which will also pull libc and other essential stuff. However from the experience above, it would be useful to also upgrade python or python-minimal as soon as possible. It was only later that I found out I could probably have done both in a single step:
sudo aptitude install ubuntu-minimalThis is a meta-package that depends on apt, and python/python-minimal, so I think it would suffice. It will pull python2.6 and python-minimal2.6 as dependencies, so all python apps will already have their bed made, when it's required.
As a second step, let's make sure a second base meta-package kicks in, which will pull another batch of essential stuff:
sudo aptitude install ubuntu-standardAfter that, you can safely issue the first step of the Debian release notes upgrade:
sudo aptitude safe-upgradeDepending on which packages you you have installed, you might need to repeat this step until it upgrades no more packages. This is because although aptitude has a great dependency resolution algorithm, it's not perfect (but this is being worked on ;-) ).
When you come to the point that safe-upgrade can't upgrade any more packages, it's the time when most of the other stuff must be resolved, which is:
sudo aptitude dist-upgradeThis command should be the first time where packages are removed by conflict resolutions. Again, you might need to issue it more than once, but there's nothing to worry (unless you get errors, but they are being logged by script, and you can go over it whenever you want).
Now, as a final step, make sure the basic meta-package is upgraded, if it wasn't already:
sudo aptitude install ubuntu-desktopYour kernel will be upgraded in this step, if you use the generic flavor, but your current version won't be removed. You might want to remove it manually afterwards. You can keep more than one, but it's mostly clutter, specially if you, like me, have a separate /boot partition.
Now you can logout of script, logout of screen and boot your new Jaunty. Computer Janitor (System>Administration>Computer Janitor, or whatever it is translated to for your language) will help you cleanup afterwards.
Note: Expect this mini howto to be updated.
Subscribe to:
Posts (Atom)