Posts | Tags | Archive

Recovering data off a bootlooping Nexus 5

The situation

The Nexus 5 that I've had for almost 5 years started bootlooping repeatedly.

I've replaced/repaired multiple parts of it over the years but lately it had been more flaky than usual. For example:

  • Sometimes when booting it would show a "Firmware update in progress, don't disconnect your computer" message, even though there was nothing connected to the USB port.
  • It would randomly freeze and soft reboot when using it about once a week.
  • When propping it up on a book or something to angle the screen towards me, the screen would sometimes turn off and on (as it turns out, this was a warning sign)

I've been ready to replace it for a while, but couldn't justify it while this one was perfectly (ok, mostly) functional. That changed today when I tried to turn it on and it started bootlooping.

So buy a new phone and move on, right? Right, except I committed the cardinal sin of living in the digital age - not having complete backups. Yeah, yeah, I know.

I got most of it. I'd been using Syncthing to automatically back up photos, videos, and other files to my home server every night so they were all good. The problem was that I had been using FreeOTP-export to back up my 2FA secrets manually and the latest backup was missing a bunch of logins.

So I needed to somehow get into my non-booting phone and pull the data off it. At least because I was replacing it, all destructive options were on the table.

Diagnosis

The first step was seeing if other people had this problem. Some searching revealed that it was most likely the power button gone bad and being stuck in the pressed position.

To confirm this, I rapidly hammered the power button on a hard surface while booting, hoping that the jarring would un-stick it and let it boot. This sort of worked in that it would boot, but stopping at any point would result it it powering off again. It was also extremely difficult to tap it consistently enough to keep the phone on for long enough to pull any data off it.

The issue I mentioned earlier where propping the phone up made the screen turn on and off now made sense. I'm guessing this was causing a slight flexing of the power button switch, causing it to close and trigger. Over time this must've permanently bent something so it stayed closed, causing the bootloop.

Now that I had pretty much confirmed that it was something physically wrong with the power button circuitry, I messaged my friend Mike. When it comes to electronics, he's definitely more experienced than me. Plus, he lets me borrow his tools :)

The "fix"

The next day Mike came over with his soldering iron and we started brainstorming.

The plan of attack was to just take the power button off the board entirely. It can't always be pressed down if it's not on the board right?

In preparation for this, I wanted the phone to automatically boot when it was plugged into a charger so I wouldn't have to manually short the pins on the board. Fortunately this can be done with a fastboot command: fastboot oem off-mode-charge 0 (basically: don't allow charging while off, therefore turn on when charging).

After desoldering the power button from the main board, it was still bootlooping. Not good. I went back and cleaned up the solder to make absolutely sure that the power button signal pins weren't connected in any way. Still bootlooping when powered on. It would get to the bootloader and stay there, meaning the power button wasn't being pressed anymore, but when launching to either recovery or the main OS, it would immediately restart.

Looked into dumping data from the bootloader - impossible.

Reflashed the boot and recovery partitions - same thing.

Tried booting into recovery using fastboot boot <recovery-image> - bootloop.

sounds like it's time to cry

-- Someone on Freenode's #lineageos-dev channel after I explained the situation

Thankfully, it was not. Right after that, someone else asked if I had reconnected the battery. I had not.

Turns out that a working battery is required to boot the phone, even when it's plugged in. I had taken the battery out to desolder the power button and never put it back in since the phone seemed to boot fine without it. Whoops.

After plugging the battery into the board and booting the phone, it launched the main OS without any issues. Huge relief.

Recovering the data

With the phone booted up normally I used adb (running as root) to pull the 2FA codes and other important things that I knew I needed over to my laptop.

I really don't trust myself to remember everything so I booted the phone into recovery mode and pulled a complete backup of /data/data, /data/app, and /storage/sdcard (not actually an sdcard) to my laptop as well.

To make absolutely sure I didn't miss anything, I also pulled a raw image of the entire flash memory (disk-based encryption was not enabled) using adb pull /dev/block/mmcblk0 mmcblk0.img. Everything I need should be in the normal backups, but if not, I can always go spelunking through that image.

Lessons learned

  • If it's important, back it up automatically. If it's not automatic, at some point it will be missed.
  • Make sure you have the ability to get root access on every device you own before you need it - in this case I wouldn't've been able to pull the 2FA codes or do full backups without it.

Future plans

For the Nexus 5, I'm planning on either buying a replacement power button or maybe just soldering some wires to the exposed pads and snaking them out of the phone to an external button if I can't find one. The phone can then continue to be used as an app development testbed, a Chromecast remote, or as a basic emulation console until it dies for real. In theory I could keep using it as a phone too, but at this point I just don't trust it enough.

For my next phone, I've decided on the Xperia XZ1 Compact. It's a smaller phone that's mostly waterproof (IP68), has a fast SoC (Snapdragon 835), a headphone jack, SD card support, and great battery life. I have high hopes for it. Also, as evidenced by this post, having root access to the OS is pretty critical at times so I'll be flashing LineageOS on it ASAP.


Password recovery from an Alcatel-Lucent Cellpipe 7130

Background

In my current rental situation, the landlord provides internet included with the rental. Since it's fast enough and has enough bandwidth this isn't an issue at all. The only problem is that the modem provided by the telco is an ancient Alcatel-Lucent Cellpipe 7130 5VzA2001 modem/router combo. It's running firmware version 1.0.4.4R8-wh, released on 2012-05-08 16:30 (there are no updates, I checked).

Since the Cellpipe only has 4 x 10/100 LAN ports and terrible WiFi performance, I would much prefer to use my own equipment to perform the routing and just use the Cellpipe 7130 as a WAN gateway. Unfortunately, for some reason, there is no way to turn off the routing aspect of the Cellpipe and just use it as a modem (aka bridge mode). There is also no way to turn off DHCP functionality on it.

My current hacky solution to these problems is to let my router get an IP from the modem/router via DHCP, then set that IP as the DMZ host on the Cellpipe. This effectively forwards all TCP and UDP ports to the router, making it seem like it's the boundary device for the network. Luckily, my router always seems to request the same IP no matter what so this continues to work even if both devices are power cycled.

Since this is not an ideal situation, I'd like to replace the Cellpipe with a more performant modem without any of the routing overhead. To do this, I just need to port the configuration details from the Cellpipe (like the username and password for the upstream PPPoE connection) over to a new modem. Unfortunately, there's nothing in the web interface that will allow me to see the password and there's no option to do a configuration backup. I also can't just call up the telco since it's the landlord's account.

This means we're stuck, right?

Digging into the Cellpipe

Doing a quick CVE search for the Cellpipe only turns up CVE-2015-4586 (a CSRF vulnerability) and CVE-2015-4587 (an XSS vulnerability), both of which aren't useful in this case. Oh well.

Seeing as the Cellpipe 7130 manual(PDF) mentions various functionality that my device doesn't seem have, my hypothesis is that there might be a configuration backup/restore function somewhere, just not exposed in the UI. With that in mind, I went trawling through the HTML and JavaScript files the web UI served up. I found:

The color 'red' is commented out and redefined as white. This suggests that they could be hiding error messages by just making them the same color as the background:

1
2
//var red    ='#FF0000';
var red    ='#FFFFFF';

Debugging in production builds with commented out alert statements:

1
2
//alert(isRouter);
//alert(isAPmode);

Various comments indicating lack of source control:

1
2
3
4
5
6
7
8
//Jamie hide 20111109
/*if(menuItem=='HPNA')
{
    printMenuItem('hpna.html', 'HPNA', white, darkBlue);
}else{
    printMenuItem('hpna.html', 'HPNA', black, blue);
}
*/
1
2
//fanny add 2011/02/22
printMenuItem('wifi_statistics.html', 'WLAN Statistics ', black, white);
1
2
3
4
5
<!-- jonathan 2004.04.07    Begin  -->
<form name="logoutForm" method="post">
    <input type="hidden" name="logoutMsg">
</form>
<!-- jonathan 2004.04.07    End  -->

These all seem to allude to shoddy release processes, meaning that it's highly likely that per-model customizations were rushed UI-level hack jobs on top of the current release and not maintained branches where actual functionality was changed.

Since a configuration backup will include the PPPoE username and password, let's do a search for "config". Sure enough:

1
2
3
4
5
6
7
8
printMenuSection('util_main.html', 'Utilities', white);
printMenuItem('lang_set.html', 'Language Setting', black,white);
printMenuItem('util_reboot.html', 'Reboot Gateway', black,white);
//printMenuItem('util_reservice.html', 'Restart Service', black,white);
printMenuItem('util_factory.html', 'Restore Factory Defaults',black, white);
//printMenuItem('util_cfgstore.html', 'Configuration Store', black,white);
//printMenuItem('util_cfgrestore.html', 'Configuration Restore', black,white);
printMenuItem('util_webfirmware.html', 'Web Firmware Upload', black,white);//Jamie add back 20111109

Visiting http://<router_ip>/util_cfgstore.html shows a nice "Store" button that downloads a text file containing 771(!) key/value pairs, including the username and password for the router. Success!

Future plans

There are a ton of other options in the downloaded config file, some of which look like they enable bridge mode. However, flipping those settings on and applying the new config using http://<router_ip>/util_cfgrestore.html didn't seem to change anything. I'm going to keep messing with it while I look into sourcing a better modem to use my newfound credentials with.

Longer-term, the plan is to transition to a dedicated modem that provides WAN access to a low-power computer running pfSense for firewall and routing duties. From there, an unmanaged switch (possibly injecting PoE) can provide access for enough dumb wireless APs to bathe the house in WiFi, as well as wired hookups for the devices that don't move around too much or need the throughput. For now though, I'll settle for replacing the modem.


Forwarding spam with Gmail

With multiple Gmail accounts, it's often easier to forward all emails to a single main account that you actually check. This can be accomplished by setting up a forwarding account in Gmail's settings.

However, something that isn't immediately obvious is that messages that are considered spam are never forwarded. While Gmail's spam detection is extremely good, it's not perfect. There have been times where emails that I actually wanted have ended up in the spam folder and weren't forwarded to my main account.

To fix this, we want to tell Gmail to not flag anything as spam so it will be forwarded along.

Note

This will not cause spam to appear in your inbox. It bypasses the spam filters so the email is forwarded properly, but the account it's forwarded to will also flag the email as spam. The end result is that that the email will end up in the spam folder of your main account (where you can actually look at it) instead of not being forwarded from the original account at all.

We can do this with a filter that matches is:spam and applies the option "Never send it to Spam". Unfortunately, Gmail's interface makes this task much harder than it needs to be.

  1. Open Gmail and do a search for is:spam. Notice that Gmail autocorrects it to in:spam.
  2. Click the dropdown arrow on the right side of the search box to bring down the advanced options and click "Create filter with this search". Hit "OK" on the warning box that pops up. Notice that the filter has been autocorrected again to label:spam.
  3. In the URL you should see something like #create-filter/has=label%3Aspam. Change label to is in the URL and hit enter. It should modify the text in the box without changing anything else.
  4. Check the "Never send it to Spam" checkbox and hit "Create filter". You'll see the text get autocorrected to in:spam again, but if you check in Settings -> Filters and Blocked Addresses you should see the correct is:spam filter.

© Carey Metcalfe. Built using Pelican. Theme is subtle by Carey Metcalfe. Based on svbhack by Giulio Fidente.