Wednesday 13 April 2011

DjView4 plug-in in Mac OS X Firefox? Only under X11.

It was easy to find open tools to make decent-quality e-books from my scans of a classic text. Scan Tailor and djvubind are a killer combination for producing an e-book with positional, searchable OCR in no time. The URI and the web browser are central to what I'm doing at the moment, and DjVu is no use unless a single click on a URI will render it directly in the browser, jumping straight to the fragment of interest.

I'm deeply invested in Firefox and OS X and while the LizardTech plug-in lives on at Celartem and works on Firefox 3.6.x, it's closed-source, its toolbar buttons don't work and I'm not sure it'll survive for long. DjView 4 includes an open NPAPI plug-in which works on Linux but isn't available for OS X, so I thought I'd have a go at getting the DjView 4 plug-in working on OS X in case it was something trivial. It wasn't, and I thought I'd document why in case anyone else wasted their time wondering/speculating before (or after) the resulting patch gets accepted.

My first misdirected effort was to simply build the DjView 4.7 tree outside MacPorts and copy the resulting nsdejavu.so into the ~/Library/Internet Plug-Ins/ directory, hoping simple shared objects still worked like the good ol' Linux days. They don't; on OS X, plug-ins must be "loadable bundles" (well, technically a "loadable module" bundle within a loadable bundle, which also happens to be a package — and let's not even get started on flat packages).

My second misdirected effort was to simply copy the built nsdejavu.so into an existing plug-in bundle (updating CFBundleExecutable). The plug-in ceased to appear in about:plugins. I guess if Firefox cared to check, the stuff returned by NP_GetValue() would be inconsistent with the Info.plist, but that seemed unlikely.

My third effort was to build the plug-in as a bundle, using the OS X template plug-in from the new NPAPI SDK. After altering the HEADER_SEARCH_PATHS to include config.h, the MacPorts glib headers and the NPAPI headers and fixing a bunch of trivial compile errors, I hit a wall at:

DjVuPlugin.c: In function 'NPP_SetWindow':
DjVuPlugin.c:2484: error: 'NPWindow' has no member named 'ws_info'


NPWindow only has ws_info if you're compiling for Mozilla on non-OSX UNIX. Reading a little more closely, it turns out the plug-in runs DjView in a separate process, relying on Firefox support for either Xt or XEmbed to render the document, neither of which are supported on the normal Cocoa-based Firefox most OS X users use. (While it's possible to run X11-based Firefox on OS X and probably possible to build an Qt-on-X11 variant of DjView that would support the -netscape flag, it's not worth it.)

So we're stuck with Celartem's plug-in for now.

Saturday 5 March 2011

Predictable path for single result in SpotlightFS

As part of a slightly bigger project, I'm putting together a simple system to look up select files or directories by a unique ID I've assigned them previously which I can use to reliably refer to them forever, without depending on their contents (which rules out checksums) or their paths (which rules out symlinks), and which survives (careful) moves across filesystems/disks (which rules out inode numbers), or even of changing my UNIX flavour (which rules out hard dependence on OS X). It'll be rather like URL shorteners (except some have disappeared!) or Carbon's Alias Manager.

I'm using Spotlight to index the UID and SpotlightFS to make access a simple matter of accessing a URL such as file:///byUID/o9e8 (making Spotlight an implementation detail that can change later). I can formulate a Spotlight query that'll return exactly one result — the one with the corresponding UID — but SpotlightFS will always give you a directory of symlinks whose names encode the true paths to results (link targets), meaning they vary based on the path. An example query (not using my UID stuff) with one result:


me$ ls -la /Volumes/SpotlightFS/SmarterFolder/exec_separate_vm/
.
..
:Users:me:JavaSnoop.properties -> /Users/me/JavaSnoop.properties


What I want is a predictable symlink that works when there's exactly one result, linking to that single result. The only way I can see to do this and keep file:///byUID/ URLs working is to patch SpotlightFS. The resulting patch (to version 2.0.3,2, ready to be dropped into /opt/local/var/macports/sources/rsync.macports.org/release/ports/fuse/spotlightfs/files/) allows me to do this:


me$ ls -la /Volumes/SpotlightFS/SmarterFolder/exec_separate_vm/the-only-result
/Volumes/SpotlightFS/SmarterFolder/exec_separate_vm/the-only-result
->
/Users/me/JavaSnoop.properties


It doesn't show up in the directory listing of a SmarterFolder query — if you don't stat() it, you'll never know it's there (which could be considered a feature). If there are zero results or multiple results, it'll provide the same symlink with an explanatory message as the target:


me$ ls -la /Volumes/SpotlightFS/SmarterFolder/arloegu098pgm/the-only-result
/Volumes/SpotlightFS/SmarterFolder/arloegu098pgm/the-only-result
->
error-there-must-be-exactly-one-result

Saturday 19 February 2011

Make OS X forget incorrect Wi-Fi details

Every time I connect to one particular wireless network I get prompted with an "Authenticating to network" dialog box:



The "Please enter your name" always contains a username I once entered by mistake; even though I now enter the correct one every time, the mistake gets remembered. "Please enter your password" is always blank, which is annoying because I'd expect it to be stored in the Keychain. Deleting all related Keychain entries for the network made no difference.

A quick ferret through ~/Library/ turns up a likely suspect:


me$ grep -r 'incorrect-username@domain' ~/Library/
Binary file Library/Preferences/com.apple.eap.profiles.plist matches


Opening that file up with Property List Editor and correcting all instances of incorrect-username@domain fixed the problems (including prompting for a password).

Sunday 13 February 2011

Root shell in Mac OS X Terminal.app using system dialog

I run OS X as a standard user and often need to do admin-y things. I like OS X's use of sudo but feel uneasy about typing an administrator password into a terminal even with "Secure Keyboard Entry" on. Supposing somebody exploited some browser hole and got access as me, it seems to me it's just a little easier to alias sudo to something evil than it is to fake the system authentication dialog (the one provided by SecurityAgent). Using the SecurityAgent dialog has its problems (I'd really like to see the show-picture-chosen-by-user approach that online banking uses) but as I always trigger it explicitly it'd require a human to poke around. So, using a variant of the technique from Mac OS X Hints:

osascript -e \
"do shell script \
\"/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal\" \
with administrator privileges" &>/dev/null &

This'll prompt for admin login details using the system authentication dialog (SecurityAgent) then start a new Terminal.app with a root shell inside. Conveniently enough it also turns the menu bar blue restores the default menu bar transparency whenever the terminal has focus, making it easy to tell which you're using even if you don't set a special background colour.

UPDATE : two minor disadvantages to this approach if you (constantly) have a Terminal running as a normal user:

  • if you quit the normal user Terminal, you can't restart it from the dock; seems the dock sees the root one and assumes your click is redundant

  • if you use OpenTerminalHere then it opens the new terminal in the root Terminal — which I guess means "Secure Keyboard Entry" prevents only eavesdropping and not stuffing commands/keys into it

Template Toolkit, tpage, redirect and OUTPUT_PATH

It seems you can't use [% FILTER redirect(...) %] from tpage:

me$ cat foo.tt
[% FILTER redirect("foo.html") %]
blah
[% END %]
me$ tpage foo.tt
redirect error - OUTPUT_PATH is not set
me$

Somebody submitted a very simple patch against tpage, which is one way to solve it. Another is to write a very short Perl script that uses the programmatic interface and use it to replace tpage. In case you can't or don't want to do either of those, though, here's a third, quick-and-dirty recipe which you can just cut and paste and still use with tpage:

me$ cat foo2.tt
[% PERL %]
$context->CONFIG->{OUTPUT_PATH} = ".";
[% END # perl %]
[% FILTER redirect("foo.html") %]
blah
[% END %]
me$ tpage --eval_perl foo.tt

me$ cat foo.html

blah
me$

Thursday 10 February 2011

MacPorts MacFUSE/bindfs and "Operation not permitted"

I couldn't get MacPorts MacFUSE + bindfs to work out of the box; it worked fine as an admin user but not as a normal user:


$ mkdir foo bar
$ bindfs foo bar
mount_fusefs: failed to mount /Users/me/foo /Users/me/bar@/dev/fuse0:
Operation not permitted


I'm not the only one of course, but I don't see any solution described. The device has lenient permissions:


$ ls -la /dev/fuse*
crw-rw-rw- 1 root operator 31, 0 2011-02-10 12:06 /dev/fuse0


Quick fix: tell MacFUSE to treat everyone as a (MacFUSE) administrator:


$ id
uid=501(me) gid=20(staff) groups=20(staff),
[...]
$ sudo sysctl -w macfuse.tunables.admin_group=20
macfuse.tunables.admin_group: 80 -> 20


I'm sure this isn't meant to be necessary and it has security implications (I've not thought it through properly but the docs say if someone could already run code as a normal user and used allow_other then they'd at least be able to hang some system processes that walked the mount) but it gets the job done.

GnuPlot "all points y value undefined" with "set timefmt %s"

Try as I might, I couldn't get MacPorts GnuPlot ("4.2 patchlevel 5") to restrict the range of some time data I was plotting (as usual, I'm not the only one):

gnuplot> set xdata time
gnuplot> set timefmt "%s"
gnuplot> set xrange ["1297212300":"1297256400"]
gnuplot> plot "battery_graph_Feb09-002305_to_Feb09-130805.csv" \
using 2:3 with lines
^
all points y value undefined!

gnuplot> show xrange

set xdata time
set xrange [ "" : "" ] noreverse nowriteback

gnuplot> set xrange ["1297212300":"1297256400"]
gnuplot> show xrange

set xdata time
set xrange [ "" : "" ] noreverse nowriteback



Solution: roll the latest version from source (Version 4.4 patchlevel 2; I couldn't work out what version fixed it from the ChangeLog and don't care to binary-chop it). Downside: can't quite work out how to enable AquaTerm support.

Wednesday 9 February 2011

Sainsbury's Basics Android splash screen

Wifi battery drain on CM6.1.1 on my HTC Desire

I (only) recently upgraded from CyanogenMod 6.0.0-RC1 to 6.1.1, and started noticing much higher battery drain, just like others did as soon as it came out. Aside from a subjective feeling that the battery drains fast, the proportion of time spent running "Android OS" under "Battery use" and the proportion of "Time spent without sleeping" under "Spare Parts -> Battery history -> Running" seemed high:







There's been some discussion by other Desire owners on CyanogenMod forums and on XDA Developers over whether the "Wi-Fi sleep policy" is better set to "Never" or "When screen turns off"; it certainly seems intuitive that the latter would save more power. Being the scientific sort I experimented, running on "When screen turns off" overnight and on "Never" from when I woke up (always within range of my router):





"Never" is clearly much kinder to the battery on my phone, at least on CM 6.1.1 and within range of my router. Another subjective observation: I'm not sure whether it's just a change in power accounting, but with "Never" I now find "Wi-Fi" accounts for a much larger proportion of "Battery use" than it used to even though the rate of power drain seems to be back to normal.



I generated the graph from data captured and exported by Battery Graph, using GnuPlot (though perhaps just screenshotting the one Battery Graph made would be easier!):



set term png
set xlabel "Clock time"
set ylabel "Charge (%)"
set timefmt x "%s"
set xdata time
set xrange ["`gdate -d "2011-02-09 00:45" +%s`":"`gdate -d "2011-02-09 15:00" +%s`"]
set arrow \
from "`gdate -d "2011-02-09 08:01" +%s`", graph 0 \
to "`gdate -d "2011-02-09 08:01" +%s`", graph 1 \
nohead \
linetype 0
set label '"Wi-Fi sleep policy" set to "Never"' \
at "`gdate -d "2011-02-09 08:01" +%s`", graph 0.5 \
center \
rotate
set datafile separator ","
set output "battery.png"
plot "battery_graph_Feb09-002305_to_Feb09-150905.csv" \
using 2:3 \
notitle \
with lines \
smooth bezier

Monday 7 February 2011

HTC Desire S-OFF / NAND unlock with VirtualBox on Mac OS X

I had a couple of brief moments of discomfort while doing AlphaRev's NAND unlock (a.k.a. S-OFF, aka @secuflag) via Mac OS X / VirtualBox. While I'm sure the vast majority of people who'll do this have already done it I'll explain them quickly just in case it helps anyone.

I followed these instructions. (I'd post this there but apparently I'm too new to be trusted.)

Firstly, if you find that your handset is listed under the USB devices list but greyed out, make sure adb isn't connected (run adb disconnect).

Secondly, expect to have to re-establish the phone's virtual USB connection several times during the process -- just click the USB icon at the bottom of VirtualBox and select the phone. I spent a good few minutes waiting for progress when the flasher was actually waiting for me.

Aside: I did this to allow repartitioning of the NAND (if you're a CyanogenMod 6 / HTC Desire user you'll know just how little space is left on the /data partition of the already-tiny internal NAND for downloaded apps), but I don't really understand the locking mechanism. Clearly custom firmware could already be flashed to /system, but apparently /system was only writable while FASTBOOT was running, i.e. not while Android was running. Fine, but why can't you just repartition the NAND under FASTBOOT? Why does it require a patch to the recovery?

Android smartphone/handset locking politics

I don't follow the custom Android distro or handset hacking worlds very closely, and once you've rooted and flashed a custom distro like the excellent CyanogenMod on your phone you've wrested back about as much control over your own hardware as most people need, so I only just noticed that the HTC Desire's NAND protection (a.k.a. the @secuflag or S-ON) got beaten by the AlphaRev team a while back. Even though I don't fully understand the implications, this makes me happy; any increase in the control customers have over the devices they paid for is great. I had a quick FWSE but couldn't find any substantial discussion of the practice of locking down the hardware (I guess it could be termed weak Tivoisation?). Paul @ MoDaCo said "Shame on you HTC for going to such unnecessary lengths (but that's another story for another day)." Cory Doctorow said "what a misery it is that the mobile phone companies continue to spend good money to frustrate the legitimate activities of their customers". Also, HTC clearly has a cynical view of the GPL, following it in letter but not in spirit.

The ability to unlock/root/whatever was a significant factor in my choice of smartphone. Back in July '10 when I was choosing a smartphone I shortlisted the Desire and the Motorola Droid 2; the Desire was already rooted, but I would almost certainly have held out for a Motorola Droid 2 if they'd said it would be rootable. Instead they said they had no intention (I'm sure I remember a much less friendly statement of refusal from Motorola but I can't find it now). At least they were open about it! Several months on I'm very happy with the Desire, so perhaps it's for the best. The Droid 2 turned out to be rootable, as did the T-Mobile HTC G2 more recently. I really can't tell whether we should worry about the future. All handset manufacturers could go the TiVo route if they wanted to, so it seems current devices are only rootable because they want to make it difficult/risky, but not impossible.

Handset manufacturers are driven by profits, so there must be an advantage to the manufacturer in doing this (or a disadvantage in not doing it). Hackers like the CyanogenMod team will bring out the best in a handset and a tiny minority of early adopters and geeks will test it out; the manufacturer gets protection against having to honour warranty returns if we brick our phones in the process, plus free testing/stabilisation of features they can consider (and indeed copy) for later stock firmware releases and future phones. They also get "plausible deniability" -- they can claim to networks that they've done their best to hand control to networks over features like wireless tethering and removal of garbage bundled apps, which networks don't want end-users to be able to do (higher support burden, less income from "value-added" services they've bundled, more load on the data network). I guess it also frustrates attempts by competitors to copy whatever innovative features the handset has, but my understanding is that competitors are more than capable of completely reverse-engineering a phone, and that preventative measures have a negligible effect.

When I buy the unbranded version of a phone I expect to pay more and I expect full control in return. Sadly those of us who expect that are a tiny minority of the market and have almost no sway. Increasingly, I recognise this as the trade-off you get for living in a free society. Shareholders want the maximum return on investment, at any moral cost. If there's a morally indefensible but slightly less expensive way to do something, that's how it'll get done -- capitalism rewards greed above all else. In a sense, though, it's Rawls' "First Principle of Justice", so I guess I'm at least free to be just as morally obnoxious.