Yesterday, I was installing the latest KDE 4.10 beta, which is built from source.
The Gentoo KDE overlay already has ebuilds, which use KDE’s SVN as the source, rather than a tarball.
While installing all the packages, there was a failure to install kdeartwork-wallpapers:
* Package: kde-base/kdeartwork-wallpapers-9999
* Repository: kde
* Maintainer: kde@gentoo.org
* USE: amd64 elibc_glibc kernel_linux userland_GNU
* FEATURES: sandbox splitdebug userpriv usersandbox
>>> Unpacking source...
* Fetching disabled since 1 hours has not passed since last update.
* Using existing repository copy at revision 1326016.
* working copy: /usr/portage/distfiles/svn-src/kdeartwork/kdeartwork
* Exporting parts of working copy to /var/tmp/portage/kde-base/kdeartwork-wallpapers-9999/work/kdeartwork-wallpapers-9999
rsync: link_stat "/usr/portage/distfiles/svn-src/kdeartwork/kdeartwork/wallpapers" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
* ERROR: kde-base/kdeartwork-wallpapers-9999 failed (unpack phase):
* {ESVN}: can't export subdirectory 'wallpapers' to '/var/tmp/portage/kde-base/kdeartwork-wallpapers-9999/work/kdeartwork-wallpapers-9999/'.
** Call stack:
* ebuild.sh, line 93: Called src_unpack
* environment, line 4224: Called kde4-meta_src_unpack
* environment, line 3403: Called kde4-meta_src_extract
* environment, line 3309: Called die
* The specific snippet of code:
* rsync --recursive ${rsync_options} "${wc_path}/${subdir%/}" "${S}/${targetdir}" || die "${escm}: can't export subdirectory '${subdir}' to '${S}/${targetdir}'.";
** If you need support, post the output of `emerge --info '=kde-base/kdeartwork-wallpapers-9999'`,
* the complete build log and the output of `emerge -pqv '=kde-base/kdeartwork-wallpapers-9999'`.
* This ebuild used the following eclasses from overlays:
* /var/lib/layman/kde/eclass/kde4-meta.eclass
* /var/lib/layman/kde/eclass/kde4-base.eclass
* /var/lib/layman/kde/eclass/kde4-functions.eclass
* /var/lib/layman/kde/eclass/cmake-utils.eclass
* This ebuild is from an overlay named 'kde': '/var/lib/layman/kde/'
* The complete build log is located at '/var/tmp/portage/kde-base/kdeartwork-wallpapers-9999/temp/build.log'.
* The ebuild environment file is located at '/var/tmp/portage/kde-base/kdeartwork-wallpapers-9999/temp/environment'.
* Working directory: '/var/tmp/portage/kde-base/kdeartwork-wallpapers-9999/work/kdeartwork-wallpapers-9999'
* S: '/var/tmp/portage/kde-base/kdeartwork-wallpapers-9999/work/kdeartwork-wallpapers-9999'
This is a simple package, and since it is a live ebuild, I assumed a directory had been moved and set about debugging the problem.
I spent a good amount of time looking into the KDE eclasses, trying to figure out how to figure out what was being built.
Only after a while of this, during another attempt at compiling the package, did I realize that SVN could be the culprit.
A little digging revealed that indeed SVN, for reasons unknown, had done only a partial checkout of the repository.
Indeed trying manually to checkout the entire part of the repository required was never successful in one go: it took multiple manual resumes to checkout the complete repository.
Somehow, this error was never detected by Portage.
Having not even touched SVN for 8 months, after several years of begrudingly using it, I had totally forgotten the sort of pains it allowed.
Today I decided to resurrect an old piece of hardware I’ve had sitting useless for a while: an Asus Eee PC 1005PE.
As an experiment, and to pave the way for future experimentation I decided to use GRUB2.
I’m also trying out the no-multilib profile, so grub 0.xx is not supported.
I’m pleased to report that using GRUB2 on a normal, BIOS booting system was extremely easy.
Here’s some information about how I did it.
Partition Table
I’m using a single EXT4 partition, the simplest possible setup.
pebble ~ # cat /etc/fstab
# /etc/fstab: static file system information.
#
# noatime turns off atimes for increased performance (atimes normally aren't
# needed); notail increases performance of ReiserFS (at the expense of storage
# efficiency). It's safe to drop the noatime options if you want and to
# switch between notail / tail freely.
#
# The root filesystem should have a pass number of either 0 or 1.
# All other filesystems should have a pass number of 0 or greater than 1.
#
# See the manpage fstab(5) for more information.
#
# <fs> <mountpoint> <type> <opts> <dump/pass>
# NOTE: If your BOOT partition is ReiserFS, add the notail option to opts.
/dev/sda1 / ext4 noatime 0 1
On October 22nd, EBS issues in a single availability zone brought down all sorts of websites, big and small.
EBS is an important and impressive (read: complicated) part of AWS, but this isn’t the first time we’ve seen it underperform, and I was honestly not expecting so many well known websites with extremely talented teams to be bitten again.
Having worked with EC2 at scale for almost three years, I’ve seen several similar incidents and know a bit about how to weather the storm (since a hurricane has previously caused AWS issues, this is more than a figure of speech).
So, what does it take to survive an AWS outage? More specifically: how can we build a system with data stored on EBS while gracefully handling its more ugly moments?
The Setup
EBS has a lot going for it: it’s pretty cheap (storing 100GB will only cost you $10/month), you can create snapshots at any time for backup, and when an EC2 instance fails, any EBS volumes can easily be remounted on a new instance.
EBS very rarely loses data, and it’s common to use RAID 1 (or 10) for even more safety.
Lots of people have their data on EBS and still sleep well at night.
But when EBS acts up, your data can essentially go missing.
Without good preparation, there’s not much to do other than wait for EBS to recover.
What can be done to prepare?
Two pieces are particularly important: replication across availability zones and quick (preferably automated) failover.
Using Multiple Availability Zones
Due to the nature of EBS, it has been particularly vulnerable to cascading failures.
A small trigger can cause outages that affect nearly an entire availability zone.
Without having data and instances ready to go in multiple zones, there’s no way to quickly recover.
Most serious deployments in EC2 will replicate data to one or more slave databases. But it seems it’s less common to ensure that the master and slave are in different availability zones.
How much will it cost?
While transfers within the same availability zone are free, Amazon charges charges a whopping 1 cent per GB sent to another availability zone in the same region.
What’s a reasonable expected cost for replicating from one database instance?
According to Scalyr’s EC2 benchmarks, we can expect a maximum of about 2000 4KB writes per second to a single EBS volume on a small EC2 instance.
That’s only 8MB/sec, or a maximum of 675GB per day, for which Amazon will charge $6.75.
Meanwhile, those two small instances have an on-demand rate of $3.84 per day.
Keep in mind 8MB/sec is an absolute maximum. In practice, even with large EC2 instances (which Scalyr showed to have double the EBS throughput, but cost four times as much), a more sustainable throughput is at best half of that.
It’s reasonable to expect that the cost of replicating to another availability zone won’t exceed the cost of the database instances.
What about sending data back to non-database instances in another zone?
Scalyr found that EBS read performance is actually about an order of magnitude worse than write performance.
Even assuming maximum read AND write performance can be obtained simultaneously, it isn’t significantly more expensive than just maxing out on writes.
A little bit extra safety
Amazon describes availability zones as “distinct locations that are engineered to be insulated from failures in other Availability Zones”.
However, there has been at least one incident where an entire region was affected.
If the possibility of downtime from this scenario is unacceptable, you have to consider replicating across regions instead of availability zones.
Latency will be higher between regions, and bandwidth costs will follow the public data transfer pricing, which starts at 12 cents/GB.
However, if surviving this level of catastrophe is important, the costs are surely worth it.
How to manage failover
While it’s replication across zones that allows for any chance to survive major EBS outages, quickly reacting to the outage is what will really minimize downtime.
Fortunately, in the last few years this has gotten quite a bit easier.
There are many databases supporting master-master replication, automatic leader election, and all sorts of other ways to switch over without any manual action.
MongoDB has replica sets, CouchDB replication supports multi-master configuration and was specifically designed to handle extended downtime (there are people running CouchDB on their cell phones!).
Even MySQL has good support for master-master replication and automatic failover now.
If the particular database you’re using doesn’t support automatic failover, at the very least build a small tool that lets you manually fail over to another set of databases.
Test it heavily ahead of time and make it easy to use: there’s nothing worse than scrambling to fix a problem only to mistakenly make it worse, and any time spent fiddling is time where your site is down.
Simulating behavior during an EBS outage
Simulating EBS failures is hard.
The EBS volume doesn’t disappear, it just stops responding to requests.
The instance mounting the EBS volume is generally unaffected overall.
The closet thing I can come up with to the behavior of a stuck EBS volume is to mount an NFS volume, and then shut down the NFS daemon.
Any process reading or writing to the NFS volume will hang until the volume is reconnected.
Fortunately, there is a very important and dangerous type of issue to demonstrate how this works.
Handling EBS issues on the client side
Most out of the box automatic failover setups work great when one of the databases fails.
But as mentioned above, EBS issues don’t cause database to completely fail.
There will be very high iowait, and any processes using EBS-backed data may hang, but the instance in general continues running just fine.
Importantly, connecting to a database instance with a ‘stuck’ EBS volume will generally succeed.
However, remote requests for writing, (and reading, depending on the specifics of the database’s caching), will generally not return any data.
Most libraries for working with databases will specify both a connection timeout and a “general” timeout, but general timeout often defaults to unlimited.
This is reasonable, since any specific timeout would limit long-running jobs, but not setting this to a relatively small value can sabotage any failover mechanisms.
To see what happens, lets use CouchDB and run a little test using the methodology described above.
CouchDB uses HTTP as an interface, so its dead simple to work with.
Here’s the test procedure:
Set up two machines on the same network.
Machine A exports a directory via NFS, which will simulate our EBS volume.
Machine B mounts the NFS directory, and uses it to store CouchDB’s data.
After starting CouchDB, the NFS daemon is stopped.
curl is used to simulate database traffic
And here’s the result:
12345678910111213
#everything works initially
user@machineB ~ $ curl http://127.0.0.1:5001/sampledb/doc1{"_id":"doc1","_rev":"1-15f65339921e497348be384867bb940f","hello":"world"}#NFSd is stopped on machine A
#now requests hang forever
user@machineB ~ $ curl http://127.0.0.1:5001/sampledb/doc1^C#this hangs forever too: the connection succeeded
user@machineB ~ $ curl http://127.0.0.1:5001/sampledb/doc1 --connect-timeout 5^C#this correctly times out
user@machineB ~ $ curl http://127.0.0.1:5001/sampledb/doc1 -m 1curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received
Any mechanism expecting the --connect-timeout option to protect against misbehaving instances will be completely defeated when EBS starts acting up.
Be sure to look into specifically what timeout options every DB client library gives.
The names may differ, but generally there is an option for connection timeouts, and another timeout for the length of the entire request.
Unfortunately, the connection timeout is often the only one prominintely mentioned in documentation.
Be sure to look into specifically what timeout options every DB client library gives.
The names may differ, but generally there is an option for connection timeouts, and another timeout for the length of the entire request.
Unfortunately, the connection timeout is often the only one prominintely mentioned in documentation.
Handling everything else
I’ve covered one particularly important aspect of handling outages, but of course there’s more to it.
As Amazon mentioned in their summary, EBS is one of the building blocks of several other components of AWS.
Having a service completely immune to EBS-backed database issues is no good if you aren’t also prepared for ELB issues.
And of course any other instance in your infrastructure can flat out fail at any time.
We can’t prepare for everything, but handling single-AZ EBS outages is one thing we should all be able to handle.
UPDATE: The issue mentioned below has been fixed and will be in Kernel version 3.5!
As I mentioned in past posts, I’ve been working on getting my new 15” MacbookPro working completely with Linux. Good progress has been made on a lot of fronts, but there is a new regression in the i915 drivers that is holding me (and likely others) back. I added a post to the long MacbookPro thread on ubuntuforums. Here’s what I posted for anyone who would like to help.
Hi all,
Recently, the team of developers working on improvements to the i915 linux kernel drivers that are needed for many recent Macbook Pro models have been adding several enhancements to the drivers that benefit Macbook Pro users. The most notable one is that in their most recent git kernel tree, the number of LVDS channels is automatically set properly for the Macbook Pro, eliminating the need to apply a patch and pass the i915.lvds_channels=2 option as many on this thread (including myself) have done. Another change is the option to disable the intel_backlight controls, allowing the gmux_backlight control to take over which (for me at least) is the only working backlight control for a Macbook Pro.
However recently they inadvertently introduced a regression that is also causing a similar black screen. I reported a bug and the developers are taking a look but I suspect they do not have access to any Macbooks to test with. If any readers of this post who have a modern (say, 6,x or above) Macbook pro running linux in EFI mode, and are comfortable building their own kernel from git could help out with the bug I’m sure it would be greatly appreciated and would help get these improvements into the mainline kernel faster.
Here’s what you can do:
1.) Read the bug report at https://bugs.freedesktop.org/show_bug.cgi?id=49518 2.) Get the development git sources from git://people.freedesktop.org/~danvet/drm-intel 3.) Confirm that commit e646d57 from the git sources introduce a black screen 4.) Post to the bug report the result of any patches by kernel developers. include all the information a kernel developer would need to diagnose the issue: the system name of your Macbook (eg MacbookPro8,1), the full dmesg output with drm.debug=0xe added to your kernel parameters, and the output of the intel_reg_dumper command that is part of the intel-gpu-tools package located at http://cgit.freedesktop.org/xorg/app/intel-gpu-tools/ (version 1.1 works for me). 5.) Otherwise help out however you can
Thanks to Matěj Laitl, Amarok now supports transcoding music when copying to iPods and iPhones.
However, while attempting to copy some FLAC encoded music to my iPod while transcoding to ALAC, I discovered it did not work, with only an unhelpful error message displayed.
Launching amarok with the –debug flag, I was able to see a little more detail:
So the transcoding appears to be failing. Fortunately we have a big help to figure out why: we have the exact command parameters used for the transcoding.
On the surface of course, the command and parameters look completely legitimate.
To see what was broken, I just had to run the transcode command myself. The following output was the result:
It looks like it’s the map_meta_data option causing issues.
After looking through the ffmpeg repository a bit, it seems map_meta_data was removed in version 0.10 after being depricated in version 0.7.
There is a new option called map_metadata that replaces it, but with different syntax.
However, for simply transcoding an audio file, neither of these options are needed:
the only time they should have to be specified is when doing something such as transcoding a movie with multiple audio tracks.
This means the fix is as simple as ensuring that the map_meta_data option is removed. Amarok has a large codebase but it is actually very friendly to new developers, so understanding where and how to fix this turned out to be straightforward. Here, the relevant code is in src/transcoding/TranscodingJob.cpp.
This class represents a single job to do one transcode, which will be run in the background.
Different transcoders, such as the one found in src/core/transcoding/formats/TranscodingAlacFormat.cpp can set options to the transcoder (in this case it sets the “-acodec alac” option as you would expect), but there are several common options set in the Transcoding::Job::init() method; one of these is the offending map_meta_data option.
The relevant code is included below. It uses a KProcess object from the KDE libraries to run the transcode job in the background.
The command and parameters are set, and then the slots for progress and completion are connected to the proper signals.
voidJob::init(){DEBUG_BLOCKm_transcoder=newKProcess(this);m_transcoder->setOutputChannelMode(KProcess::MergedChannels);//First the executable...m_transcoder->setProgram("ffmpeg");//... then we'd have the infile configuration followed by "-i" and the infile path...*m_transcoder<<QString("-i")<<m_src.path();//... and finally, outfile configuration followed by the outfile path.constTranscoding::Format*format=Amarok::Components::transcodingController()->format(m_configuration.encoder());*m_transcoder<<format->ffmpegParameters(m_configuration)<<QString("-map_meta_data")<<QString(m_dest.path()+":"+m_src.path())<<m_dest.path();//debug spam followsdebug()<<"foo";debug()<<format->ffmpegParameters(m_configuration);debug()<<QString("FFMPEG call is ")<<m_transcoder->program();connect(m_transcoder,SIGNAL(readyRead()),this,SLOT(processOutput()));connect(m_transcoder,SIGNAL(finished(int,QProcess::ExitStatus)),this,SLOT(transcoderDone(int,QProcess::ExitStatus)));}
So the exact fix is simply to remove lines 17 and 18, so that the map_meta_data option is no longer passed to ffmpeg.
It does in fact work: after removing those lines and recompiling I can happily copy all my FLAC encoded music to my iPod in ALAC format!
I submitted the code for review on the Amarok reviewboard.
In addition to the transcoding fix, I also included a tiny fix to make debugging easier in the future: in the debug lines above where a QStringList is sent to debug output, use the .join() function to print the command exactly as it would be run, instead of a comma separated list of individual parameters.
The full set of changes can be found on Github.
During this whole process, I’ve noticed a couple other pain points that would be great to improve. If possible I’ll work on this in the near future:
When multiple tracks are picked to be transcoded at once, each track is handled in series. I painfully watched htop show me one core working dilligently while the others, and the iPod’s disk, sit mostly idle. Ideally multiple tracks could be transcoded at once.
Amarok kept telling me about stale tracks on my iPod and telling me I can do something about them. Either I can’t find this option or it doesn’t exist. Either way the UX at least probably has to be improved.
When selecting a transcoding option, Amarok currently performs this transcode whether or not it makes sense or not based on the source filetypes. For example, if I ask Amarok to transcode to ALAC, any lossy filetype would be converted, which would simply waste space and time. It would be great if Amarok could detect and prevent such “upconverting”.
