Map Projections

I first learned about cartography and mapping technologies back in 2007 or so when I was working on Danger's bluetooth stack.  In order to test my work on the serial port protocol (SPP), I connected to a bluetooth GPS "puck".  This led to finding interesting things to do with that data, namely a mapping and location tracking application.

Back to the present... As part of trying to find a really interesting project to really drive my web development goal in my spare time, I've been doing a lot of reading on map projections again, looking into what's required (technically) to convert things like park maps or trail maps into something that could be overlaid on a map.

Here's the 30 second background summary: Since the earth is a sphere, but your screen is a rectangle, some magic is required in order to display maps of the earth on your screen.  This magic is actually mathematics, and is called projection.  You can liken the problem of projection to the problem of flattening out an orange peel on a table top.  You can do it, but there are trade-offs involved.

There are some really interesting projections out there, but popular maps on the web use the Mercator projection.  This projection is accurate around the equator, but more and more distorted the further north and south you travel.  This projection is popular because it flattens out longitude, so that they orthogonal to latitude, making for a nice x, y coordinate system that is intuitive and square, among other things.

It may blow your mind to realize that what you think you know about the earth is actually wrong... or distorted, but what does that mean for trying to display a park map as an overlay on one of these Mercator maps?  It depends on the projection of the map of the park, but generally it means that the park map will have to be distorted to fit the faux reality of Mercator.

Figuring out and applying that distortion is what I'm interested in, and has led me down all sorts of interesting paths over the past few days.

Scala + Play

Many moons ago, I was a web developer.  I created my own web framework, developed a bunch of libraries (charting, scraping, etc.), wrote articles and everything.  I even managed to created a website that had close to a million views in a month, and that was mentioned in a PopSci article.  I've tried to keep up with web technology over the years, but haven't really had the need to do much web development and am far behind the curve.

Every once in a while I get the urge to start a project involving the web.  I end up spending a bunch of time reading about some new language or technology, and don't really get much actual coding done before I lose interest or catch a glint of something shiny in another corner.  (This is a recurring story for me that is definitely not limited to web initiatives.)

Anyway, I'm back to thinking about the web this time because I really want a good tool that can automate (more of) the ice scheduling that I do for my daughter's hockey association.

I have written some code towards this purpose already, but it's just a C# command-line app that performs some web scraping to pull in all team schedules into an aggregated list for a given week (impossible using the current navigation on the site).  I've also made some false starts toward a more powerful desktop form-based app to automate things, but that approach never sat well with me.

1. I want to be able to access this from anywhere, including my phone or tablet.
2. I want to be able to let other associations use the tool, if it works well.
3. I don't really want to tie it to Windows, and would prefer to push my skill-set

So here I am, reading about Scala.  This is a hybrid object-oriented/functional language that is definitely cross-platform (runs on the JVM), and has web frameworks like Play, which is less like a Java servlet environment, and more like PHP.

Let's see how far I get this time.

Fitbit Zip Teardown

Just saw this (pretty light) teardown of the Zip:

http://www.designnews.com/author.asp?section_id=1386&doc_id=256756

Since they came out in the same year, I expect the Zip and the One (and maybe even the Flex?) have fairly similar hardware characteristics.

Fitbit Dongle Enumeration

I just checked in my notes on how the USB dongle responds to enumeration.  You can see them here:

https://github.com/hiptopjones/fitbit/blob/master/UsbDongle/usb_enumeration.txt

I learned a lot about the workings of USB devices, but can now see that it publishes two HID interfaces, each of which produces reports of a generic 32 bytes at a time.  While this is not a HID device in the traditional (mouse, keyboard, joystick) sense, and using a HID interface means quite low data rates, I expect this is done (at least partly) to avoid needing special drivers on most common operating systems.

I haven't yet figured out the format of these packets, but I can see normal sync data (matching the blocks in the sync log), as well as a number of strings like "StartDiscovery", etc.

I had been wondering if the strings represented some sort of command-based protocol with the dongle, instructing it to perform bluetooth operations, but I see that most of these strings are on IN endpoints, which means they're probably just there for logging, and not instructions.  This is supported by the fact that we see lines like this in the log:

    <Notice>: IN<CTRL> trace: StartDiscovery

I can't yet rule out that one of the HID interfaces could be being used for the transport layer of the Bluetooth stack, but I don't see bluetooth stack-like strings in the binaries on Mac or Windows.

Continuing to investigate.

Fitbit USB Dongle Sniffing

Sync packets are all well and good, and decoding them continues to be a priority, but I decided to give it another go with the USB filter driver today.

Using busdog, I'm now seeing chatter text in a USB trace!  (Does that mean the link is not encrypted?)

11 In  (USB URB Function: 9) 0.007991 32 48 45 59 20 48 45 59 21 20 20 48 49 47 48 46 49 56 45 20 20 00 00 00 00 00 00 00 00 00 00 00 14 HEY HEY!  HIGHFIVE  ............

I expected that the filter driver would just pick up bluetooth transport (HCI?) bytes, but I also see strings in the log that make me wonder if the dongle uses something like a serial protocol to communicate with the host PC.  That would be weird, though, or at least not like most other bluetooth dongles.

13  In  (USB URB Function: 9)   0.000002    32  20 01 43 61 6e 63 65 6c 44 69 73 63 6f 76 65 72 79 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  .CancelDiscovery...............
. . .
13  In  (USB URB Function: 9)   0.000003    32  20 01 47 41 50 5f 4c 49 4e 4b 5f 45 53 54 41 42 4c 49 53 48 45 44 5f 45 56 45 4e 54 00 00 00 00  .GAP_LINK_ESTABLISHED_EVENT....

Anyway... we'll see.  I don't have protocol docs for Bluetooth LE, but perhaps by subtracting the bytes in the sync log from this output and doing some more reading, I'll be able to deduce something more about the actual device communication.

Fitbit Sync Decode - Part 3

Slowly but steadily piecing together parts of the puzzle from the sync logs / fiddler traces.

Discovered that there is a so-called "mega-dump" and "micro-dump".  The mega-dump is used for general syncs, but the micro-dump was used when I tried a firmware update.

I've setup a github repo for my parsing code at https://github.com/hiptopjones/fitbit.  I'll let that code stand as documentation for now.

Fitbit Sync Decode - Part 2

If we think about what information would need to be passed back and forth, that might help when looking at the sync data, and provide ideas about what knobs to turn.  A good source for inspiration here is the user manual.

This summary data would probably go in both directions:

• Calories
• Steps (walking + running)
• Elevation (floors, distance)
• Distance (native units)
• Active score (# plant leaves, score)

These would be passed from device to server:

• Timer information (start/stop)
• Steps data series (per minute)
• Elevation data series (per minute)
• Battery level (charging status?)
• Firmware version
• Beginner mode flag (?)

These would be passed from server to device:

• Enable/Disable flag for all of the displayed metrics
• Chatter text
• Current date/time
• Stride length (walking, running)
• Body stats / BMR (for calorie estimations)
• Timezone
• Units (distance)
• Time format (ie. 12/24 hour clock)
• Left-hand/Right-hand mode
• Silent alarm settings (including repetitions on specific days)
• Sleep tracking mode (normal vs. sensitive)

Some additional inputs:

• There are 10 leaves on a fully-grown plant
• The device can store 14 days of minute by minute data, and summary data for 30 days
• The device can have 8 separate alarms

Fitbit Sync Decode - Part 1

There are two sync blobs available to us for inspection.  One blob is from the POST to the server, and the other is the server's response.

An example request:

POST http://client.fitbit.com/tracker/client/message?p_lcl=en_US HTTP/1.1

Content-Type: text/xml

User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)

Host: client.fitbit.com

Content-Length: 3328

Pragma: no-cache

Cookie: JSESSIONID=...

<?xml version="1.0"?>

<galileo-client version="2.0">

<client-info>

<client-id>...guid...</client-id>

<client-version>1.0.0.2292</client-version>

<client-mode>force-sync</client-mode>

<dongle-version major="1" minor="1" />

</client-info>

<tracker tracker-id="FFFFFFFFFFFF">

<data>...base64-encoded data...</data>

</tracker>

</galileo-client>

An example response:

HTTP/1.1 200 OK

Server: Apache-Coyote/1.1

Expires: Thu, 01 Jan 1970 00:00:00 GMT

Cache-control: no-store, no-cache, must-revalidate

Pragma: no-cache

Content-Type: text/xml;charset=UTF-8

Content-Language: en-US

Vary: Accept-Encoding

Date: Sun, 06 Jan 2013 20:25:11 GMT

Content-Length: 17077

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><galileo-server version="2.0"><server-version>URL: https://wush.net/svn/fitbit/weightsite/branches/Release_20130104

Last Changed Rev: 44872

</server-version><ui-request action="done"><client-display height="450" width="650" minDisplayTimeMs="20000" containsForm="true"> ...html...</client-display></ui-request><commands><connect-to-tracker connection="disconnect" tracker-id="FFFFFFFFFFFF"/></commands><tracker type="megadumpresponse" tracker-id="FFFFFFFFFFFF"><data>...base64-encoded data...</data></tracker></galileo-server>

Comments:

• This is a trace from a forced sync, where I clicked the Sync Now button in the FitbitConnect UI, so it also contains an HTML page for display.
• The blobs are base64-encoded since they are part of an XML body.
• The blob data is also written (not base64 encoded) to the sync log on my desktop

Next up... start digging into these blobs.

Fitbit Text

Just for fun, keeping a list of the chatter text the Fitbit throws at me:

• HI THERE
• HOWDY
• I LIKE U
• GO
• STEPGEEK
• HELLO
• CLIMB IT
• WOOT!
• ITS ON!
• FASTER
• IM READY
• WALK ME
• SMOOCHES
• STEPITUP
• HOLD ME
• HEY HEY!
• BON JOUR
• MOVE IT
• CHEERS
• UCANDOIT
• LETS GO
• HUG ME
• READY?
• GOOOOAL
• VAMOS
• YOU ROCK
• BURN IT
• HIGHFIVE
• LOVE YA
• WHATS UP

FileScanner

Started working on a file scanner today.  This project is a result of my cluttered hard drives.  I have music and photos and videos all over the place, with duplication, etc.  I need to consolidate them, index them, and make sure they are all stored safely on one set of RAID-1 drives.

This is also an attempt to get out of the C# trap.  Since I use C# a lot at work, it's become my go-to language.  This sucks in a non-MSFT world, like the Mac I use at home, so I'm writing the scanner in Java.  I used to have mad Java chops, but have mostly lost them over the past 4 years.  Hopefully this will bring some of that back.

Ideally, I'd like to write this in Objective C or C++, just to really push myself, but I don't want to spend a ton of time on this particular project.  Java is a lot like C#, so I'm hopeful this will be wrapped up fairly quickly.

Fitbit Sync Experiment

Managed to screw up my Win7 VM by installing the USB filter driver.  Had to go back to a system restore point.

On another note, I tried a little experiment today.  I didn't let my Fitbit sync all day so I could see if there would be a difference in the size of the data payload when I eventually did sync it.  There was.

My Fitbit sent 2380 bytes to the service.  My wife's Fitbit was home with her all day, syncing regularly, and her Fitbit only sent 784 bytes.

Need to decode the sync packets.

Fitbit One Sync Logs

My first order investigation of the Fitbit One involved the following:

• Installing the FitbitConnect software on my Win7 VM
• Setting up the device with my account
• Forcing a sync
• Finding for the sync logs

The FitbitConnect options panel allows the specification of a proxy server, so I obliged and used Fiddler to capture network traffic.  

I figured the sync logs had to be stored somewhere on the Win7 filesystem, since the FitbitConnect UI mentions them (provides a checkbox to keep them for 5 days), and so does this guy.

On my install, the logs were found under C:\ProgramData\FitbitConnect\Logs.  An example filename is log-20130104004939.txt

UPDATE: On my iMac, the log is here: /private/var/run/com.fitbit.galileod.log

Scanning the logs I see scattered bits of interesting things.

1. The server is basically in control of the FitbitConnect UI.  On almost every action, the server returns an HTML page embedded in the XML response.

2. Among other things, the service also appears to provide the display strings to be used on the device.  For example, here is part of the data blob that the service provided in one of its responses:

    0000020: 0b00 5045 5445 5220 2020 2020 4855 4720  ..PETER     HUG 

    0000030: 4d45 2020 2020 5741 4c4b 204d 4520 2020  ME    WALK ME   

    0000040: 4845 5920 4845 5921 2020 0000 0000 0000  HEY HEY!  ......

3. The server classifies me as "obese".  While I can't disagree with this, it caught me by surprise that this is part of my account profile.

            var props = {

                'Android App Version': '',

                'iPhone App Version': '',

                'Has Android App': false,

                'Environment': 'prod',

                'Has iPhone App': false,

                'User Age Range': '35-44',

                'Body Type': 'obese',

                'Gender': 'male',

                'Locale': 'en_US',

                'Premium': false,

                'Premium Expired': false,

                'Platform': 'Fitbit Connect',

                'Facebook Linked': false,

                'Height': '76.02362',

                'GoalPrimary': 'steps',

                'GoalWeight': '',

                'Operating System Version': osVersion,

                'Paired Aria': false,

                'Paired One': true,

                'Paired Zip': false,

                'Paired Flex': false,

                'Paired Ultra': false,

                'Paired Classic': false

            };

More later.  Planning to add a USB filter driver to capture traffic at the dongle.

Fitbit One

Gave the wife a Fitbit One for Christmas.  She's really enjoying it, and today (yesterday?) I bought my own.

Unfortunately, my interest is less about fitness and more about learning how it works... though I am interested in the fitness side of it too.  (For example, I'm sure it will be frowning at me tomorrow for the amount of sleep I'm not going to get tonight.)

WTF! Moment

Just had a funny WTF! moment.

As mentioned in my previous post, I've been working on things related to audio output.  Tonight I breadboarded a simple amplifier using an LM386.  I decided to run a smoke test, and connected it to the 5V output on the LM3S1968 board with the input open.

When I turned it on, I got light buzzing.  I decided to see what happened when I moved the 10k pot... suddenly, my speaker was spewing forth a synthesized voice.  I was shocked and moved my hand... the sound stopped.  Touching the pot again caused the voice to return.

I soon realized that I was picking up some local AM radio station (I could also hear other stations in the background), but it is quite the conicidence for that station to be running an interview with Roger Ebert, who uses a synthesized voice??!

I momentarily wondered if I had gone mad. :-)

Back to Electronics

Wow, it's been a while since I wrote anything here.

These days I'm back to piddling about with electronics/robotics.  Sure, I can't seem to find my book on netting... but I don't think that's the reason I've swung around. :-)

I've pulled all my crap out of boxes, and set up what wifey calls a "cave" in the spare room.  That's a pretty accurate description--I'm packed in with other stuff at a little desk in a small room.

My focus these days is on some sort of smart, extensible toy for kids (and hackers), which has got me doing a lot of reading about how Text-to-Speech technology works (since you can't have a smart toy that doesn't TALK!).

I'm all for making things difficult on myself, so my current project is trying to use the little LM3S1968 board (used this earlier in the year to make Lucas a game) to do some formant speech synthesis.  I don't think this will be super hard, but there's enough tech/math involved to make it an interesting challenge.

We'll see how far this renewed interest takes me.

Ravenna Ultra Low Altitude Vehicle

This is pretty sweet.  A treehouse rocketship.  One of the most interesting bits here is the use of a matrix of seven-segment LED displays to create a larger display...

Lucas continues to ask for a treehouse, which of course our townhouse does not afford.  But there are some great ideas here that could be applied to other projects.  Very cool.

Lucas, Bob, Logo, EK-LM3S1968

Lucas told me last week that he wanted to make a Bob game.  He's referring to a game that Steve Hales created for the Sidekick many moons back, and that both kids still love playing.  Lucas has some ideas about new levels, and he demanded that I should show him what to do to make this game.

I love his interest, I just wish it was that easy.

After a few days of thinking about how exactly I was going to help bring my 7 year old's dream to life, I decided that he would probably be happy learning how to "walk" first, and I need not get all stressed out about helping him "run"... at least not yet.

This past weekend, I set him up on my computer with ACSLogo.  Of course he was thrilled, and had lots of fun making the turtle move around the screen, and say funny things.  He has repeatedly asked to play some more, which is awesome.

This got me thinking about options for Lucas' game... I have a bag of old Sidekicks, so I looked into downloading the hiptop SDK, but that didn't go so well.  I have an iPad, but don't yet have the right version of Mac OS X to develop for it.  I have a Windows Phone, but that's a development device for work.  I have various other electronic gadgets, but nothing that could really pass as a game platform...

Then I remembered I have an EK-LM3S1968 evaluation kit from Luminary Micro (now TI).  It has a 128x96 OLED screen on it, a few buttons (including 4 in the shape of a d-pad!), and a speaker.  And it comes with a built-in maze game in the quickstart application, so there is example code to work from.

Now I'm working on a dinky little vertical scroller/shooter.  Lucas hasn't mentioned his Bob game since I sat him in front of ACSLogo, but now my interest is piqued. :-)

Sidekick

Lol... Found out from Wikipedia yesterday that the Sidekick is now being (re)sold in Canada as the mobiflip through Mobilicity.  It of course has none of the service benefits that made it awesome, but still fun to see it eek out a little more life.

The other thing I found out yesterday was that the Danger Developer Zone is offline, and has been for months.  This is not wholly unexpected with the Microsoft acquisition, but was sad because I was hoping to use the hiptop SDK to help my son write a game that he could take with him and show his friends.  I guess I'll have to use a more modern development environment...

eZ430-Chronos... Again.

I took a small detour and briefly considered working on an application for the iPad, but have come back to the consensus that active RFID is something I really should put some time into.

The first order of business is to get these two eZ430 kits to talk to each other.

When I got distracted the other day I was attempting to compile the MSP-GCC tools for my Mac... this time I'm just going to do it the easy way and install the Code Composer tools that come with the kit.

Came across this write-up the other day.  Similar goal to mine actually, so I'm curious to see how my results at 433MHz compare to his at 2.4GHz.  He led me to OpenBeacon as well, which I hadn't seen previously.

Decisions

As my recent entries show I've been dreaming about active RFID and the scenarios I could chase down in that arena.  This would build my hardware/firmware skills, which are in need of some development.

But then yesterday and today I've been thinking about working on an iPad application that my daughter inspired.  I've never ventured into Apple development, let alone iOS, and that's something I feel I would benefit from.

This isn't the first time I've been down this road.  I'm almost certain I suffer from some form of attention deficit.  What I'm interested in doing during the off-hours changes so often that I find it difficult to make much forward progress on anything...