My last post went over an ESP8266-based wifi gateway for Milight/LimitlessLED bulbs. This supports a few kinds of bulbs that have been around for a couple of years (e.g., this one).
About a year ago, newer bulbs and controllers started showing up that used a different 2.4 GHz protocol. This introduced some scrambling that made it difficult to emulate many devices. This was presumably done intentionally to prevent exactly the sort of thing that my last project accomplished (boo!).
The new bulbs actually have some really cool features that none of the old ones do, so there’s some incentive to figure this out. In particular, they support saturation, which allows for ~65k (2**16) colors with variable brightness instead of the 256 colors that the old one does. They also combine RGB and CCT (adjustable white temperature) in one bulb, which is super cool.
A few others have dug into this a little, but as far as I’ve been able to tell, no one has figured out (or at least shared) how to de-scramble the protocol. I think I’ve managed to do so. I should mention that I don’t have much experience doing this kind of thing, so it’s entirely possible the structure I’m imposing is a lot more complicated than what’s actually going on. But as far as I’ve been able to tell, it does work. I’ve tested with five devices – four remotes and one wifi box.
I’m going to start by detailing the structure, and I’ll follow up with some of the methodology I used to reverse the protocol.
Differences from old protocol
From a quick glance, there are a few superficial differences between the new and old protocols:
- Listens on a different channelset (this was true of different bulb types among the old bulbs too). The new bulbs use channels 8, 39, and 70.
- Different PL1167 syncword. It uses 0x7236 , 0x1809 .
- Packets have 9 bytes instead of 7.
- Packets are scrambled. The same command can look completely different.
The scrambling is the tricky part. As others who have stared at packet captures noticed, when the first byte of packets for the same command is held fixed, most of the other bytes stay fixed too. This suggests that the first byte is some kind of scramble key. Turns out this is the case.
Example packets for turning group 1 on with one of my remotes:
|
1 2 3 4 5 |
A7 45 E9 6F BB 99 9E CF 4C E1 D6 4F 79 99 78 16 9A A7 23 D1 55 03 47 25 2E 5B E8 92 18 18 33 42 15 60 02 CE CA E0 E0 EB 0A DD A1 CA 9F |
Notation
I’ll for a packet p, I’ll use pi to refer to the 0-indexed ith byte in the packet. For example, p0 refers to the 0th byte.
I’ll use p’ to refer to the scrambled packet for a packet p.
Structure
The 9 bytes of the packet are:
| p0 | p1 | p2 | p3 | p4 | p5 | p6 | p7 | p8 |
|---|---|---|---|---|---|---|---|---|
| Key | 0x20 | ID1 | ID2 | Command | Argument | Sequence | Group | Checksum |
Packet scrambling
The designer of this protocol added in quite a few things to complicate reversing it. None of them are particularly hard on their own, but with them all added together it makes it pretty tough.
The scrambling algorithm is basically:
- A scramble key k is computed from p0
- Each byte position i has a different set of four 1-byte integers A[i]. Integer A[i][j] is used when p0 ≡ j mod 4.
- A[i][j] is up-shifted by 0x80 when p0 is in the range [0x54, 0xD3]. This does not apply to the checksum byte.
- p’i = ((pi ⊕ k) + A[i][p0 mod 4]) mod 256, where ⊕ is a bitwise exclusive or.
The algorithm to compute k is as follows (in ruby):
|
1 2 3 4 5 6 7 8 9 10 11 |
def xor_key(p0) # Generate most significant nibble shift = (p0 & 0x0F) < 0x04 ? 0 : 1 x = (((p0 & 0xF0) >> 4) + shift + 6) % 8 msn = (((4 + x) ^ 1) & 0x0F) << 4 # Generate least significant nibble lsn = ((((p0 & 0xF) + 4)^2) & 0x0F) msn | lsn end |
A values:
| Position | 0 | 1 | 2 | 3 |
| p1 | 0x45 | 0x1F | 0x14 | 0x5C |
| p2 | 0x2B | 0xC9 | 0xE3 | 0x11 |
| p3 | 0x6D | 0x5F | 0x8A | 0x2B |
| p4 | 0xAF | 0x03 | 0x1D | 0xF3 |
| p5 | 0x5A | 0x22 | 0x30 | 0x11 |
| p6 | 0x04 | 0xD8 | 0x71 | 0x42 |
| p7 | 0xAF | 0x04 | 0xDD | 0x07 |
| p8 | 0x61 | 0x13 | 0x38 | 0x64 |
There are probably actually several possible values for some of these. It really only matters that they line up in a particular way because of the checksum.
In addition to all of this, command arguments have different offsets from 0, and some commands (i.e., saturation and brightness) have the same p4 value with arguments spanning different ranges. For example, arguments for brightness start at 0x4F. Arguments for color start at 0x15.
Checksum
The checksum byte is calculated by summing all bytes in the unscrambled packet except for the first (scramble key) and last (checksum), and k + 2.
Code
Further detail is probably easier to communicate in code, so here is a ruby library that can encode/decode packets. The project on github also has a ton of packets I scraped from my devices.
Methodology
I scraped a ton of packets with this script.
Figuring this out was mostly making assumptions, pattern recognition, and trial and error. The most helpful assumption was that sequential values for arguments in the UDP protocol had sequential values in the 2.4 GHz protocol (this turned out to be true).
I noticed that packets that had p0 values with the same remainder mod 4 followed a nearly sequential pattern. It often looked something like 0xC, 0xD, 0xA, 0xB, etc. A sequence follows this pattern when it’s xored with a constant. I wrote some cruddy ruby methods to brute-force search for constants that yielded the sequence [0, 1, …, N]. This also allowed me to find the values for the As.
Roughly the same process was repeated for each byte. Bytes that are constants were trickier because they didn’t follow a sequence. I instead brute-forced values for A given a sequence of xor keys.
Next Steps
I’ll be porting over the scrambling code to my ESP8266 milight hub to add support for the new bulbs.
UPDATE 2017-03-28: A few kind volunteers sent me packet captures from their devices, and the ID bytes were not staying fixed under decoding. Assuming my methodology is right, these should be the right values for all parameters, with the possible exception of p1 and the checksum byte.
UPDATE 2017-03-20: I found that the wifi box I was testing with supported older protocols, which transmits the unscrambled device ID. There were several possible values for the ID byte offsets, and I chose a few of them arbitrarily. The decoded ID in the scrambled protocol was not matching the ID in the unscrambled protocol. Updating the additive offset values fixed this.
Don’t suppose they would have included a Key to switch the encryption right out. (possibly for old remotes and globs) it would make life easier if they did.
Hi Chris
I think the work you have put into this project is a credit to you and what you have done so far is great!
I have bean playing with the code for this and some of the other linked projects and have not bean able to get anything happening. So I thought I would try the pre-compiled file.
I have downloaded your hex bin file for ESP8266 node MCU and flashed it. I can connect to the WiFi and does not ask for a pass word. It comes up as FaryLink_CEAA52 the IP is 92.168.4.2 if enter into a browser I get a cannot be found message. If I open the serial monitor in Arduino and rest the ESP I get
NodeMCU 0.9.5 build 20150318 powered by Lua 5.1.4
lua: cannot open init.lua
I thought my ESP was a V2 however is has the elongated chip so maybe it is a V3 Do you have any idea what I should try to get it going?
There is something else you may be able to tel me, with disco mode is the sequencing in the remote or in the bulb?
Thanks.
I want to have a pay with the code for the MiLleght hub what I am trying to do replace the remote with a ESP8266 and sequence color changes of the MiLight globe directly from ESP 8266 I chose the FUT104 globe because it is 12 volts and safe to play with. Not worry about extra banks of globes or anything like that at the moment. However I figure that are good price to start all day with the hub code. I have been trying to set up the project with Arduino IDE
I have downloaded the project file from GitHub and unzipped it. Then created a project with Arduino IDE named ESP8266-MyLight-Hub. In the unziped GitHub files in the SRC folder there is a the main file, I have copied the code from mane to my Arduino ESP8266-MyLight-Hub file. In the lib folder there are folders complaining library files .cpp and .h I have copied these folders and their contents to documents/arduino/libraries directory, along with a few other files that Arduino was asking for.
When I verify/Compile I get Errors, Arduino keeps asking for more libraries it seems there are libraries calling libraries. I cannot find all these libraries and the ones that I have found I am uncertain if I have the best alright ones. Somehow I don’t think I am setting the project up right, the structure does not look like something for Arduino. The structure looks more like something for code blocks no some other ide.
Can anyone help with how to set up the project on Arduino ide. Or possibly a cut down version, just ESP8266 remote, something really simple able to just turn on 4 or 5 colours flat out.
Amzing work you’ve done Chris!
Thanks for sharing this with us!
Hi Chris
Are you interested in looking at the MiLight SYS cable protocol? its 3 wires (0v, 24vDC, data), its likely a cutdown version of DMX for a single fitting as its part of there DMX range, each fitting has 5 colors (RGBCW, WW), there controller will only receive a propriety WiFI (not the same as the MiLight wireless) and i would like to drive there lights with just a Arduino, which is possible if the data wire can be decoded, it might even be a standard 5 color existing LED driver, i really have no idea, i can program, not decode.
Many Thanks
Ashley – contact@ashleygriffin.ca
Hey maybe you could tell us what hardware you used to find out on which channels the remotes were transmitting or how you sniffed the data
Kind regards
Any digital logic analyzer will do. Just need to sniff the SPI bus.
Would it be possible to port this awesome project to Raspbian so one doesn’t need to use an extra ESP8266?
Hi Felix,
Theoretically yes, but the code in my project is very tied to the ESP8266 platform.
Ah, that’s interesting.
I guess as-is, even if it is dropping packets, it’s not enough to matter for what receive is getting used for. As I fiddle with the remote, it seems like it’s capable of printing several packets a second. Really you only need one packet for what sniffing in my project gets used for. It’s only useful to nab the ID of a device.
My understanding was that the slow part was changing syncwords, etc. But I didn’t really dig in.
I’d love for sniffing to work without the need for the user to select a RF config, so definitely worth looking into it again. Thanks for the info.
Yes i saw that your code can switch RF configs dynamically.
I wanted to do it automatically by changing keyword, channels and packet length depending on incoming packets. You do not need to check all channels, just those that are used by the milight remotes you want to read. In the Henryk code only one channel was being read per remote anyhow. By reading all three channels i found that you actually drop less packets. Maybe thats why i do not have so much problems switching automatically. Anyhow i think this concept can still be optimized by switching the channels smarter.
https://github.com/sidoh/esp8266_milight_hub/blob/master/lib/MiLight/MiLightRadio.cppL72
In your code i think this also is happening see above line.
You will find you drop less packets by changing this line to the following even if you dont change anything else:
if ((_pl1167.receive(CHANNELS[0]) > 0) || (_pl1167.receive(CHANNELS[1]) > 0) || (_pl1167.receive(CHANNELS[2]) > 0)){
Hi,
Dont know if you are interested in this but it is possible to decode several different protocols simultaneously. I need this because of my wish to clone a milight bulb and be able to control it with any remote or setting on your hub (emulator).
You can find the fork here https://github.com/krulkip/openmili/tree/multiremote
You can find some writeups on this here.
http://arduino-projects4u.com/milight-rf-control/
http://arduino-projects4u.com/milight-new-protocol/
Kind regards
Hi Bas,
By simultaneously, do you mean that the scanning on all possible channels? Or do you mean with the ability to dynamically instruct the ESP to switch between channels?
I had really bad luck with the former — it seemed to cause listens to be really unreliable. I think because it takes enough time to switch RF configs that the NRF was missing packets.
My esp8266_milight_hub code is capable of switching RF configs dynamically as well. It happens here:
https://github.com/sidoh/esp8266_milight_hub/blob/master/lib/MiLight/MiLightClient.cpp#L5
Here’s where it gets used:
https://github.com/sidoh/esp8266_milight_hub/blob/master/lib/WebServer/MiLightHttpServer.cpp#L271
To Chris,
Fantastic piece of work. Just to confirm it also works for my two RGBCWWW CCT remotes.
I have done some work on previous generation RF protocol and made a milight bulb from scratch. http://arduino-projects4u.com/milight-rf-control/
I noticed that p0 = j mod 4 should read j = p0mod 4 just above the compute k calculation on this page.
Are you able to switch between the different types of protocol on the fly. I tried to do that at the very bottom of the above page but it is not yet reliable.
Have you considered making a Philips hue bridge emulator that talks to milight bulbs via RF.
https://github.com/probonopd/ESP8266HueEmulator
There is some support (see eaample below) for events and colour change but i do find that rather cumbersome to install. Why can Philips not implement on a skill.
https://github.com/sarkonovich/Alexa-Hue
There is also a skill released which has events which can also cover colour change.
https://www.amazon.co.uk/Philips-Hue/dp/B01LYQA2RR
Finally there are some workarounds eg IFTTT or Yonomi.
http://blogs.msmvps.com/connectedhome/2016/05/05/alexa-change-colors-on-my-hue-stuff-thanks-yonomi/
Bas
Nice, glad to hear it works for you!
Ah, yeah. It’s sort of confusing notation. “≡” in this context means “congruent to”. a ≡ b (mod n) means that a and b have the same remainder when divided by n. So if a ≡ b (mod n), b ≡ a (mod n) is also true. More on that congruence notation here.
Yeah. I had to make some fairly non-trivial modifications to Henryk’s code to do this. I pulled out the bits of MiLightRadio that configure the NRF24L01 and the PL1167 wrapper. The magic happens here. There were some other miscellaneous changes like supporting packets of different lengths, but that was the bulk of it.
I’ve not considered that, but it’s an interesting idea. Do you think something like amazon-echo-ha-bridge would cover some of the usecases you want? Basically emulates a hue bridge and allows you to configure custom commands. You could have it hit the ESP’s HTTP API, for example.
Dear friend! I was also try to investigate new MIlight B2 like this – https://www.aliexpress.com/item/Milight-B2-4-Zone-CCT-Adjust-Smart-Panel-Remote-Controller-for-led-strip-light-lamp-or/32790035244.html?ws_ab_test=searchweb0_0,searchweb201602_6_10065_10068_433_434_10136_10137_10138_10060_10062_10141_10056_10140_10055_10054_10059_9911_10099_10103_10102_10096_120_10052_10053_10050_10107_10142_10051_10106_10084_10083_10119_10080_10082_10081_10110_10111_10112_10113_10114_10078_10079_10073_10070_10122_10123_10120_10124-9911_10120,searchweb201603_13,afswitch_1_afChannel,ppcSwitch_5,single_sort_0_default&btsid=857396d0-41cf-4863-80c5-29187dd5beda&algo_expid=d3e93619-973d-411e-bb60-417f9524e141-1&algo_pvid=d3e93619-973d-411e-bb60-417f9524e141
But i was beginner in C++ (Arduino or Plaformino), can you rewrite you algo with unscramble on c+??