So, I just botched a firmware update on one of my PowerLine couplers. The model I had trouble with is a TP-Link TL-PA4020P. However, this quick guide should work with pretty much any Atheros-based PLC device. It was confirmed working on at least a TL-WPA4220 as well.

Steps to Reproduce

• Download manufacturer’s setup tool
• Download firmware files (“nvm” and “pib” files)
• Refrain from directly connecting the PLC to the computer, instead leave it hooked up to a switch and other hardware.
• Start firmware update using the aforementioned setup tool
• Have a bit of bad luck, or a power outage during the update.

Result

• “Firmware Upgrade failed” error message.
• Next up, “Local device not connected” error message.
• After unplugging and re-plugging the PLC, no lights light up.
• Faint hissing from the device, in normal operation it is silent.

Diagnosis

OK, keep calm. This is a modern piece of hardware, surely it wouldn’t need to be disassembled to flash a firmware. Right? Or so my hopes went as I started panicking. Looks like I’m not getting the manufacturer tool to retry the update on the (hopefully just) soft-bricked device. That piece of software only tells me that it can’t find the local device. A quick web search (“TP-Link PowerLine failed firmware fix” and similar) didn’t come up with anything good right away.

Well then, I thought, let’s see if it gives off any signs of life. I directly connected the PLC to my trusty MacBook and fired up WireShark. When the first packets started appearing I breathed a sigh of relief. The PLC still manages to get an ethernet link up. Amidst the stuff the Mac fires off when detecting a link (DHCP, MDNS, etc.) I finally found what I was looking for: Broadcast packets, “HomePlug AV” protocol, “Atheros_something” MAC, “Action Required Notification (Bootloader)”. Awesome! This thing is even politely asking me to remote-boot it. Let’s figure out how.

Armed with the right keywords to feed to my preferred search engine, I finally found “Qualcomm Atheros Open Powerline Toolkit” aka “open-plc-utils“. BSD-Licensed tools to set up Atheros-based PLC equipment. And, not really surprisingly, that includes a “plcboot” tool, which does just that – feed the PLC a firmware such that it can proceed to boot.

The Fix

• Connect the PLC’s network port directly to your computer’s.
• Rename the .nvm and .pib files from the manufacturer firmware package to nvm and pib (The atheros utilities are picky when it comes to file names, something I only found out after head-scratchingly reading the code.)
• Download and compile the tools, and boot the PLC:

  git clone https://github.com/qca/open-plc-utils.git
  cd open-plc-utils
  make
  cd ..
  open-plc-utils/plc/plcboot -i eth0 -N nvm -P pib  # replace eth0 if you use another ethernet

• Since the pib file shipped with the firmware only contains a default mac address, the adapter will have been set to that, too. I presume the firmware utility reads it beforehand and updates the pib before uploading it. Anyway, to get the correct mac address back onto your PLC, have a look at the label on your plc, and then:

  open-plc-utils/pib/modpib -M <your mac address> pib
  open-plc-utils/plc/plctool -P pib -R local

• Finally, flash the firmware again, using the manufacturer tool. plcboot only performs a one-time boot when given the options above. To make the firmware permanent again, the flash needs to be rewritten. (Allegedly, plcboot can do that too, but it needs a “softloader” file, which I couldn’t be bothered to extract from the TP-Link software.)

Hints and issues

• The plcboot and plctool utility may need root permissions, since they try to write raw packets to the ethernet interface. You may need to put a sudo in front.
• It may not work running the tools inside a VM, depending on your setup. This may be due to the raw packets not being properly forwarded to your physical interface. At any rate, you would need to put the VMs network interface into bridge mode.
• That said, I recommend doing all of these steps on bare hardware (it should be possible using a Linux live CD/stick).
• Depending on how broken the firmware on your device is, it may still try to boot. Therefore you may only have a short time window in which to run plcboot. Try plugging in the device to the power socket, and then as soon as possible press enter on the command line.
• If in doubt, have a look at the tool documentation, which you can view with

  man open-plc-utils/plc/plcboot.1

• There is a “PLC Recovery.exe” tool for Windows floating around the internet, which seems to do the same things as my tutorial. However, since its origins are not verifiable to me, and it doesn’t seem to come from TP-Link or Qualcomm, I won’t link to it. You’re free to search the web for it yourself, although ultimately it seems the method presented here is more successful than the exe.

Conclusions

I love the fact that chip manufacturers are building in sensible bootloaders, and that there is open source software available to access these. This is for example also the case with the Atmel ARM processor families of Arduino Due fame. I, for one, welcome this trend, making it increasingly hard to turn your hardware into a paperweight. On the downside, OEMs like TP-Link try to hide these as best as they can: the manual just says to return the device to the distributor for service when experiencing the symptoms I’ve encountered.

Bottom line: When a firmware update goes bad, don’t panic. It’s just a matter of finding the right tools. Also, it helps a lot having a general grasp of how things work on the inside to actually know what may or may not be possible.

Revisions

Nov. 2017: Added a “cd ..” to the command example and changed the path to plcboot to make the example more consistent. The open-plc-utils folder also has an “nvm” an “plc” subfolder, which may have been confusing.

Dec. 2017: Added bullet point on resetting the MAC address, thanks to reader Manfred Lischka for notifying me of the fact the MAC gets changed. Since I only had one broken adapter, I didn’t care too much. But he had two, which after recovery didn’t work since they had the same MAC.

Jan. 2018: I actually got the MAC reset commands wrong, since I didn’t do it myself. Should be better now.

June 2018: Added -i flag to plcboot command line, since the default of eth1 probably won’t work for most people. Added “Hints and issues” section. Thanks to Paolo Fiore for sharing his story, as well as Michael Stornes. Their experiences have inspired some of these changes.

With DynDNS almost inevitably deciding to abandon their free offerings, it was time for me to be off to pastures new to adress my few dynamic hosts. DynDNS, having long served as the go-to address for many system admins wishing to give a DNS name to their machines on dialup, has over the last few years slowly but steadily tried to convert its userbase from free to paid customers. This culminated in an eMail today, announcing that they would be discontinuing their free service entirely.

I initially toyed with the idea of hosting a dynamic DNS myself, but ended up with several reasons against it, to wit:

• I wanted to have port 53 (DNS) on my virtual machine unoccupied, because I intend to run a “stealthy VPN” disguised as DNS traffic at some point.
• Every solution I found pretty much boiled down to tearing a hole somwhere into my server to write zone files from a CGI, or worse, PHP script. I do not want the outside world to have an attack vector on my DNS zone.
• My hosters’ DNS admin interface is simple and powerful enough for me to not want to go back to writing zone files myself.
• I actually found a free offering that does what I want, and even better than my current setup!

 What my wishlist looked like:

• I have two dial-up endpoints that due to various reasons (primarily remote location backup, and remote administration) need to be able to talk to each other. Therefore I’m content with a small number of (free) hosts.
• Ideally, I would be able to incorporate these dynamic hosts into my domain hierarchy. Therefore, the service should allow me use my own custom domain instead of only allowing me to choose from a predefined list.
• The service should provide an https:// update URL – one would be surprised how rarely this is the case.
• My routers (AVM Fritz!Box) should be able to automatically update the names. Almost all dynamic DNS hosters comply with this requirement, since the URL configurable on the router is pretty free-form.
• The service should be free as in beer.

What I ended up using:

My router came preconfigured with several other services, but none of them had the option to use custom domains. afraid.org has custom domains, but for their free service you need to offer your custom domain to all other users. Since I did not want to allow random strangers to post hostnames under my hierarchy, their service was also out.

Through random chance I ended up at Zonomi. Their free offer has a limit of 10 DNS entries, and otherwise all the features I want! Since they count the 2 NS entries and the SOA record towards that limit of 10, you only really get 7 custom entries, but other than that it fit my profile perfectly. And they have a nicely minimalistic, modern interface to boot, which I liked a lot.

I have now setup a zone dyn.example.org containing my two hosts, and have linked that to my primary domain via NS entries (dyn.example.org IN NS ns1.zonomi.com).

Works totally as expected and I got rid of those ugly mycustomname.dyndns.org names in the process. Thank you DynDNS, but good riddance!

It’s been a while since I moved my main email to a new server and now the website is following suit. If you’re reading this than it means everything has gone over finally.

I switched over to WordPress from SPIP, because I wanted to buy into the app ecosystem that surrounds the former, in the hopes of getting more content on the site more easily. That means I’m still picking some nits here and there to get the template to do what I actually want. This may result in some minor design changes along the way.

Most of my effort has so far gone into beating WordPress to use schemaless (or protocol-relative) URLs, as the site should be transparently available over both http and https, without the broser complaining. So far I think I’ve been able to keep the upper hand, but the endeavor does by far not go as smoothly as I would have liked.

You’ll also see that I have switched to a very minimalistic theme. It’s not quite at the level of this no-design website, or Fefe’s Blog, because I’m a font geek and I love Bootstrap and my medieval numerals. I’ll try to keep the site as JavaScript free as possible.

You’ll also note that as opposed to your run-of-the-mill blog I haven’t enabled comments anywhere, because I (still) think a website is not a discussion forum. I do however enjoy getting suggestions, questions and fan-mail. So please do not hesitate to contact me.

Well, I think that’s it for the moment. I hope I can bring some interesting stuff to this website, if only for my personal future reference. I already have some ideas forming about things I would like to publically address. Stay tuned!