Map Projections

I first learned about cartography and mapping technologies back in 2007 or so when I was working on Danger's bluetooth stack.  In order to test my work on the serial port protocol (SPP), I connected to a bluetooth GPS "puck".  This led to finding interesting things to do with that data, namely a mapping and location tracking application.

Back to the present... As part of trying to find a really interesting project to really drive my web development goal in my spare time, I've been doing a lot of reading on map projections again, looking into what's required (technically) to convert things like park maps or trail maps into something that could be overlaid on a map.

Here's the 30 second background summary: Since the earth is a sphere, but your screen is a rectangle, some magic is required in order to display maps of the earth on your screen.  This magic is actually mathematics, and is called projection.  You can liken the problem of projection to the problem of flattening out an orange peel on a table top.  You can do it, but there are trade-offs involved.

There are some really interesting projections out there, but popular maps on the web use the Mercator projection.  This projection is accurate around the equator, but more and more distorted the further north and south you travel.  This projection is popular because it flattens out longitude, so that they orthogonal to latitude, making for a nice x, y coordinate system that is intuitive and square, among other things.

It may blow your mind to realize that what you think you know about the earth is actually wrong... or distorted, but what does that mean for trying to display a park map as an overlay on one of these Mercator maps?  It depends on the projection of the map of the park, but generally it means that the park map will have to be distorted to fit the faux reality of Mercator.

Figuring out and applying that distortion is what I'm interested in, and has led me down all sorts of interesting paths over the past few days.

Scala + Play

Many moons ago, I was a web developer.  I created my own web framework, developed a bunch of libraries (charting, scraping, etc.), wrote articles and everything.  I even managed to created a website that had close to a million views in a month, and that was mentioned in a PopSci article.  I've tried to keep up with web technology over the years, but haven't really had the need to do much web development and am far behind the curve.

Every once in a while I get the urge to start a project involving the web.  I end up spending a bunch of time reading about some new language or technology, and don't really get much actual coding done before I lose interest or catch a glint of something shiny in another corner.  (This is a recurring story for me that is definitely not limited to web initiatives.)

Anyway, I'm back to thinking about the web this time because I really want a good tool that can automate (more of) the ice scheduling that I do for my daughter's hockey association.

I have written some code towards this purpose already, but it's just a C# command-line app that performs some web scraping to pull in all team schedules into an aggregated list for a given week (impossible using the current navigation on the site).  I've also made some false starts toward a more powerful desktop form-based app to automate things, but that approach never sat well with me.

1. I want to be able to access this from anywhere, including my phone or tablet.
2. I want to be able to let other associations use the tool, if it works well.
3. I don't really want to tie it to Windows, and would prefer to push my skill-set

So here I am, reading about Scala.  This is a hybrid object-oriented/functional language that is definitely cross-platform (runs on the JVM), and has web frameworks like Play, which is less like a Java servlet environment, and more like PHP.

Let's see how far I get this time.

Fitbit Zip Teardown

Just saw this (pretty light) teardown of the Zip:

http://www.designnews.com/author.asp?section_id=1386&doc_id=256756

Since they came out in the same year, I expect the Zip and the One (and maybe even the Flex?) have fairly similar hardware characteristics.

Fitbit Dongle Enumeration

I just checked in my notes on how the USB dongle responds to enumeration.  You can see them here:

https://github.com/hiptopjones/fitbit/blob/master/UsbDongle/usb_enumeration.txt

I learned a lot about the workings of USB devices, but can now see that it publishes two HID interfaces, each of which produces reports of a generic 32 bytes at a time.  While this is not a HID device in the traditional (mouse, keyboard, joystick) sense, and using a HID interface means quite low data rates, I expect this is done (at least partly) to avoid needing special drivers on most common operating systems.

I haven't yet figured out the format of these packets, but I can see normal sync data (matching the blocks in the sync log), as well as a number of strings like "StartDiscovery", etc.

I had been wondering if the strings represented some sort of command-based protocol with the dongle, instructing it to perform bluetooth operations, but I see that most of these strings are on IN endpoints, which means they're probably just there for logging, and not instructions.  This is supported by the fact that we see lines like this in the log:

    <Notice>: IN<CTRL> trace: StartDiscovery

I can't yet rule out that one of the HID interfaces could be being used for the transport layer of the Bluetooth stack, but I don't see bluetooth stack-like strings in the binaries on Mac or Windows.

Continuing to investigate.

Fitbit USB Dongle Sniffing

Sync packets are all well and good, and decoding them continues to be a priority, but I decided to give it another go with the USB filter driver today.

Using busdog, I'm now seeing chatter text in a USB trace!  (Does that mean the link is not encrypted?)

11 In  (USB URB Function: 9) 0.007991 32 48 45 59 20 48 45 59 21 20 20 48 49 47 48 46 49 56 45 20 20 00 00 00 00 00 00 00 00 00 00 00 14 HEY HEY!  HIGHFIVE  ............

I expected that the filter driver would just pick up bluetooth transport (HCI?) bytes, but I also see strings in the log that make me wonder if the dongle uses something like a serial protocol to communicate with the host PC.  That would be weird, though, or at least not like most other bluetooth dongles.

13  In  (USB URB Function: 9)   0.000002    32  20 01 43 61 6e 63 65 6c 44 69 73 63 6f 76 65 72 79 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  .CancelDiscovery...............
. . .
13  In  (USB URB Function: 9)   0.000003    32  20 01 47 41 50 5f 4c 49 4e 4b 5f 45 53 54 41 42 4c 49 53 48 45 44 5f 45 56 45 4e 54 00 00 00 00  .GAP_LINK_ESTABLISHED_EVENT....

Anyway... we'll see.  I don't have protocol docs for Bluetooth LE, but perhaps by subtracting the bytes in the sync log from this output and doing some more reading, I'll be able to deduce something more about the actual device communication.

Fitbit Sync Decode - Part 3

Slowly but steadily piecing together parts of the puzzle from the sync logs / fiddler traces.

Discovered that there is a so-called "mega-dump" and "micro-dump".  The mega-dump is used for general syncs, but the micro-dump was used when I tried a firmware update.

I've setup a github repo for my parsing code at https://github.com/hiptopjones/fitbit.  I'll let that code stand as documentation for now.

Fitbit Sync Decode - Part 2

If we think about what information would need to be passed back and forth, that might help when looking at the sync data, and provide ideas about what knobs to turn.  A good source for inspiration here is the user manual.

This summary data would probably go in both directions:

• Calories
• Steps (walking + running)
• Elevation (floors, distance)
• Distance (native units)
• Active score (# plant leaves, score)

These would be passed from device to server:

• Timer information (start/stop)
• Steps data series (per minute)
• Elevation data series (per minute)
• Battery level (charging status?)
• Firmware version
• Beginner mode flag (?)

These would be passed from server to device:

• Enable/Disable flag for all of the displayed metrics
• Chatter text
• Current date/time
• Stride length (walking, running)
• Body stats / BMR (for calorie estimations)
• Timezone
• Units (distance)
• Time format (ie. 12/24 hour clock)
• Left-hand/Right-hand mode
• Silent alarm settings (including repetitions on specific days)
• Sleep tracking mode (normal vs. sensitive)

Some additional inputs:

• There are 10 leaves on a fully-grown plant
• The device can store 14 days of minute by minute data, and summary data for 30 days
• The device can have 8 separate alarms

Fitbit Sync Decode - Part 1

There are two sync blobs available to us for inspection.  One blob is from the POST to the server, and the other is the server's response.

An example request:

POST http://client.fitbit.com/tracker/client/message?p_lcl=en_US HTTP/1.1

Content-Type: text/xml

User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)

Host: client.fitbit.com

Content-Length: 3328

Pragma: no-cache

Cookie: JSESSIONID=...

<?xml version="1.0"?>

<galileo-client version="2.0">

<client-info>

<client-id>...guid...</client-id>

<client-version>1.0.0.2292</client-version>

<client-mode>force-sync</client-mode>

<dongle-version major="1" minor="1" />

</client-info>

<tracker tracker-id="FFFFFFFFFFFF">

<data>...base64-encoded data...</data>

</tracker>

</galileo-client>

An example response:

HTTP/1.1 200 OK

Server: Apache-Coyote/1.1

Expires: Thu, 01 Jan 1970 00:00:00 GMT

Cache-control: no-store, no-cache, must-revalidate

Pragma: no-cache

Content-Type: text/xml;charset=UTF-8

Content-Language: en-US

Vary: Accept-Encoding

Date: Sun, 06 Jan 2013 20:25:11 GMT

Content-Length: 17077

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><galileo-server version="2.0"><server-version>URL: https://wush.net/svn/fitbit/weightsite/branches/Release_20130104

Last Changed Rev: 44872

</server-version><ui-request action="done"><client-display height="450" width="650" minDisplayTimeMs="20000" containsForm="true"> ...html...</client-display></ui-request><commands><connect-to-tracker connection="disconnect" tracker-id="FFFFFFFFFFFF"/></commands><tracker type="megadumpresponse" tracker-id="FFFFFFFFFFFF"><data>...base64-encoded data...</data></tracker></galileo-server>

Comments:

• This is a trace from a forced sync, where I clicked the Sync Now button in the FitbitConnect UI, so it also contains an HTML page for display.
• The blobs are base64-encoded since they are part of an XML body.
• The blob data is also written (not base64 encoded) to the sync log on my desktop

Next up... start digging into these blobs.

Fitbit Text

Just for fun, keeping a list of the chatter text the Fitbit throws at me:

• HI THERE
• HOWDY
• I LIKE U
• GO
• STEPGEEK
• HELLO
• CLIMB IT
• WOOT!
• ITS ON!
• FASTER
• IM READY
• WALK ME
• SMOOCHES
• STEPITUP
• HOLD ME
• HEY HEY!
• BON JOUR
• MOVE IT
• CHEERS
• UCANDOIT
• LETS GO
• HUG ME
• READY?
• GOOOOAL
• VAMOS
• YOU ROCK
• BURN IT
• HIGHFIVE
• LOVE YA
• WHATS UP

FileScanner

Started working on a file scanner today.  This project is a result of my cluttered hard drives.  I have music and photos and videos all over the place, with duplication, etc.  I need to consolidate them, index them, and make sure they are all stored safely on one set of RAID-1 drives.

This is also an attempt to get out of the C# trap.  Since I use C# a lot at work, it's become my go-to language.  This sucks in a non-MSFT world, like the Mac I use at home, so I'm writing the scanner in Java.  I used to have mad Java chops, but have mostly lost them over the past 4 years.  Hopefully this will bring some of that back.

Ideally, I'd like to write this in Objective C or C++, just to really push myself, but I don't want to spend a ton of time on this particular project.  Java is a lot like C#, so I'm hopeful this will be wrapped up fairly quickly.

Fitbit Sync Experiment

Managed to screw up my Win7 VM by installing the USB filter driver.  Had to go back to a system restore point.

On another note, I tried a little experiment today.  I didn't let my Fitbit sync all day so I could see if there would be a difference in the size of the data payload when I eventually did sync it.  There was.

My Fitbit sent 2380 bytes to the service.  My wife's Fitbit was home with her all day, syncing regularly, and her Fitbit only sent 784 bytes.

Need to decode the sync packets.

Fitbit One Sync Logs

My first order investigation of the Fitbit One involved the following:

• Installing the FitbitConnect software on my Win7 VM
• Setting up the device with my account
• Forcing a sync
• Finding for the sync logs

The FitbitConnect options panel allows the specification of a proxy server, so I obliged and used Fiddler to capture network traffic.  

I figured the sync logs had to be stored somewhere on the Win7 filesystem, since the FitbitConnect UI mentions them (provides a checkbox to keep them for 5 days), and so does this guy.

On my install, the logs were found under C:\ProgramData\FitbitConnect\Logs.  An example filename is log-20130104004939.txt

UPDATE: On my iMac, the log is here: /private/var/run/com.fitbit.galileod.log

Scanning the logs I see scattered bits of interesting things.

1. The server is basically in control of the FitbitConnect UI.  On almost every action, the server returns an HTML page embedded in the XML response.

2. Among other things, the service also appears to provide the display strings to be used on the device.  For example, here is part of the data blob that the service provided in one of its responses:

    0000020: 0b00 5045 5445 5220 2020 2020 4855 4720  ..PETER     HUG 

    0000030: 4d45 2020 2020 5741 4c4b 204d 4520 2020  ME    WALK ME   

    0000040: 4845 5920 4845 5921 2020 0000 0000 0000  HEY HEY!  ......

3. The server classifies me as "obese".  While I can't disagree with this, it caught me by surprise that this is part of my account profile.

            var props = {

                'Android App Version': '',

                'iPhone App Version': '',

                'Has Android App': false,

                'Environment': 'prod',

                'Has iPhone App': false,

                'User Age Range': '35-44',

                'Body Type': 'obese',

                'Gender': 'male',

                'Locale': 'en_US',

                'Premium': false,

                'Premium Expired': false,

                'Platform': 'Fitbit Connect',

                'Facebook Linked': false,

                'Height': '76.02362',

                'GoalPrimary': 'steps',

                'GoalWeight': '',

                'Operating System Version': osVersion,

                'Paired Aria': false,

                'Paired One': true,

                'Paired Zip': false,

                'Paired Flex': false,

                'Paired Ultra': false,

                'Paired Classic': false

            };

More later.  Planning to add a USB filter driver to capture traffic at the dongle.

Fitbit One

Gave the wife a Fitbit One for Christmas.  She's really enjoying it, and today (yesterday?) I bought my own.

Unfortunately, my interest is less about fitness and more about learning how it works... though I am interested in the fitness side of it too.  (For example, I'm sure it will be frowning at me tomorrow for the amount of sleep I'm not going to get tonight.)