SamSpeak - Development

Amazing Things happening at Kynetx - Come to Impact

Wed, 17 Mar 2010 08:36:00-0700

Over sushi a few nights ago, I had the opportunity to bring Drummond Reid up to speed with what Kynetx has been doing. After several months in the trenches, it was a revealing experience to climb a tower and see just how far we've come. As I unfolded item after item, I was surpised to find another thing to describe. At our last Impact Conference, we unfolded the Kynetx Engine, and demonstrated some ways to use Kynetx in your quest to do amazing things. At this next Impact Conference, we will deliver an update on the progress with the engine, the improvement of our tools, and all of the things that have kept us busy over the last few months. If you want to hear about some of the stuff I downloaded to Drummond, you'll want to attend the conference. If you came to the last Impact, this one will be better. If you didn't, then it's time for you to understand what we are doing and what it means for the world. Oh, and the food will be worth the price of admission alone. Seriously. Sign up for Impact. Kynetx Impact Spring 2010
April 27-28 2010
Miller Free Enterprise Center (MFEC)
at Salt Lake Community College
9750 South 300 West
Sandy, UT 84070
Use Code FOK2010 for a 33% discount on the conference price.

Changing the World at Kynetx

Tue, 20 Oct 2009 16:00:00-0700

I've been pretty quiet on my blog lately, and I have a really good excuse. Now that I've graduated and have time to get involved in some serious endeavors, I've joined the folks at Kynetx. I've been contributing in their efforts to change the world. Indeed, this does change everything. At Kynetx, we believe that experiences can be made better through better use and understanding of context. (Gartner agrees...) Context is data in time and space: who we are, what we are doing, and what our purpose is. Kynetx has what we call a Context Automation Engine, which does the heavy lifting required to produce intelligent applications. Using our engine, you can create complex applications easily, and deploy them fast. I'll take a break now and again from my work to post more info, including examples and more information about what we are doing. If you want to learn more, attend the Kynetx Impact developer conference Nov 18-19. We will cover our technology and our vision. If you are interested in using our platform to add contextual intelligence to your applications, go sign up. Use code Windley50 for a 50% discount, and I'll see you there!

Sam Rides 1000: Augmenting the Web

Tue, 30 Jun 2009 09:19:00-0700

In my previous two posts, I introduced my project and described data collection using my G1 and Google Spreadsheets. Today, I'm going to show you how I used Kynetx Network Services to add my ride stats to my personal blog and to the Google homepage.

Dataset Conversion

Google Spreadsheets, where my stats are calculated, can publish data in a variety of formats. It cannot publish JSON data, so I use Yahoo's YQL to convert the data from CSV to json, with the following statement:


select * from csv where url='http://spreadsheets.google.com/pub?key=rxzHBMZyj1S-HVLy9lFEU7A&single=true&gid=1&range=A12%3AC16&output=csv' and columns='period,miles,hours' and period != ""

(See the raw JSON results)

Building the App

I then build my Kynetx App in AppBuilder, defining the following datasource in the Global block:


dataset ridestats <- "http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20csv%20where%20url%3D'http%3A%2F%2Fspreadsheets.google.com%2Fpub%3Fkey%3DrxzHBMZyj1S-HVLy9lFEU7A%26single%3Dtrue%26gid%3D1%26range%3DA12%253AC16%26output%3Dcsv'%20and%20columns%3D'period%2Cmiles%2Chours'%20and%20period%20!%3D%20%22%22&format=json&callback=" cachable for 2 hours

Since I ride in the morning and the evening, I cache the dataset for 2 hours. This keeps the data fairly current, but still keeps the service fast.

Annotating My Blog

I have two rules, the first of which adds stats to my personal blog:


select using "http://sam.curren.ws/" setting ()

pre {
    daymiles = ridestats.pick("$..results.row[0].miles");
    weekmiles = ridestats.pick("$..results.row[1].miles");
    monthmiles = ridestats.pick("$..results.row[2].miles");
    totalmiles = ridestats.pick("$..results.row[3].miles");
milesmessage = <<
Sam is riding 1,000 miles. Progress:
#{(daymiles > 0 ? daymiles + " Today, " : "")}
#{(weekmiles > 0 && weekmiles != daymiles ? weekmiles + " This Week, " : "")}
#{(monthmiles > 0 ? monthmiles + " This Month, " : "")}
#{totalmiles} Total.
>>

}

replace_html("#logo h2", milesmessage);

I set the rule to fire on my blog's domain, and then use the pick() method to extract different totals from the json dataset declared in the Global block. I construct a message string that varies depending on the different stat values. Finally, I replace the text at the top of my blog page with the message.

To run the Kynetx application on my blog, I plant Kynetx tags on my blog. This enables everyone to see the Kynetx Application with no installs or Action Cards. The html tags are available within AppBuilder, and I simply copied them into the template for my blog.

Annotating Google's Homepage

My second rule is activated by an Action Card installed on the user's computer (instructions for installing this are in my first post). My second rule is very similar to the first rule, with some minor difference in inserted HTML and appending the message to the existing site, instead of replacing anything on the page.


select using "http://www.google.com/" setting ()

pre {
    daymiles = ridestats.pick("$..results.row[0].miles");
    weekmiles = ridestats.pick("$..results.row[1].miles");
    monthmiles = ridestats.pick("$..results.row[2].miles");
    totalmiles = ridestats.pick("$..results.row[3].miles");
milesmessage = <<
Sam is riding 1,000 miles.
 Progress:
#{(daymiles > 0 ? daymiles + " Today, " : "")}
#{(weekmiles > 0 && weekmiles != daymiles ? weekmiles + " This Week, " : "")}
#{(monthmiles > 0 ? monthmiles + " This Month, " : "")}
#{totalmiles} Total.
>>

}

append("#body>center", milesmessage);

Activating Kynetx Rules with an Action Card also requires an update to the Dispatch block of the rule, adding this line:


domain "www.google.com"

I also generate the card inside AppBuilder, providing a custom image that I created using Pixlr

And there you have it. Sam rides 1000 miles, with automated stats provided by Android MyTracks, Google Spreadsheets, YQL, and Kynetx Network Services.

Shameless Plug

Kynetx is a cloud based automation engine, capable of doing the things I've demonstrated and much, much more. If you'd like to use Kynetx Network Services, sign up for an account, and start using AppBuilder.

Sam Rides 1000: Collecting Ride Data using the Android Powered G1

Fri, 26 Jun 2009 20:06:00-0700

On my Android powered T-Mobile G1, I'm using the free My Tracks application to record my rides. I start recording just before I start, then throw it in my pocket or bag. I stop recording at the end of my ride, then use the Upload to Google option in the map menu. You can upload the track to My Maps within Google Maps, but my rides are very similar, so I usually only upload to Google Spreadsheets.

Uploading to Google Spreadsheets creates a new spreadsheet in Google Docs, with one page for ride data, and another for stats. The program creates a new spreadsheet for each activity type, so I make sure and select Cycling when I stop recording at the end of my ride.

Additional Stats

It's important not to manually change too much on the Log sheet, as the program will get confused, but everything else in the spreadsheet is open to tinkering. In addition to the total miles and total time stats, I wanted to calculate daily, weekly, and monthly totals.

Before I computed those stats, I had to handle the date field uploaded by the My Tracks application. Google Spreadsheets cannot parse the provided date as a date value, so I had to help it along. I created a Dates sheet to accomplish that task. Rather then try and explain what I did, see this spreadsheet, which contains my modifications and formulas:

Sample Spreadsheet with modifications.

I then added some additional stats, first by calculating the start of the date range I wanted to sum, then using SumIf() to only add the mileage and time from the period I wanted.

Finally, I published the Stats sheet of my spreadsheet, so I could use it as a dataset for my Kynetx Application, which I'll explain in a future post.

Sam Rides 1000: An exercise in collecting data and web augmentation

Fri, 26 Jun 2009 15:42:00-0700

May 15, 2009 was National Ride your Bike to Work day, and I pulled out my bike and rode the 6 or so miles between my house and the Kynetx offices at Thanksgiving Point. I enjoyed it, and rode a few more times in the next week. I was musing how many miles I could ride this summer, and mentioned it to my wife. She promptly challenged me to ride 1,000 miles this summer.

I better say here that I'm not a cyclist. I haven't ridden 1,000 miles in the past 5 years, all combined together. While 1,000 miles might not be much for a cyclist, it is quite the challenge for me.

Tracking My Progress

In addition to making a few bike repairs and buying some commuting tires for my mountain bike, I immediately cooked up some geeky ways to track my progress and share my results with my family and friends.

I'm a (BIG) Android fan, and so I'm using my T-Mobile G1 as a cyclometer. I'm using the My Tracks application, which records both the route of my ride and my ride stats. After my ride, the app uploads my stats to a Google Docs Spreadsheet, where stats are calculated. I'll share more about that in a future post.

To share my progress, I'm using Kynetx Network Services (KNS) to augment my personal blog with my stats. If you are reading this post on my blog, look at just under the title for my updated stats. KNS pulls my ride stats from the Google Spreadsheet and annotates my website. All I had to do was plant some javascript tags in my blog's template to activate the Kynetx Application that makes the change.

My friends and family don't visit my blog EVERY day, so I've also produced an Action Card that displays my ride stats on Google's home page. KNS allows me to augment websites for anyone that has my Action Card installed. I'll explain more about how I wrote my Kynetx Application in a future blog post, but for now, you can install my "Sam Rides 1000" card to track my progress on Google's home page.

Installing The Action Card

An Action Card is a type of Information Card that allows your web experience to be augmented with a Kynetx Application. You activate the application by installing an Action Card Selector if you don't already have one, and then installing the card. You can disable or remove the card if and when you don't want to use the application.

Verify you have a supported browser: IE/FF on Win, FF on OSX
Install Azigo
Install the Sam Rides 1000 Action Card

Then, browse to Google's main page to see my stats: http://www.google.com

When I finish a ride and update my stats, you'll see the new numbers. If I'm falling behind, be sure and give me a nudge!

Progressive Spatial Networks

Sat, 30 May 2009 00:00:39-0700

Iâ�"ve been pretty silent the last few months here on my blog. Iâ�"ve been pretty busy with things like settling into our new house and starting full-time at Kynetx. A major sap on my free time both recently and for the last several years has been my Masterâ�"s Thesis. Iâ�"ve graduated now, and finally carved out some time to update my blog.

In my efforts in resuming regular blogging, I find it appropriate to post my Thesis for all the world. First, a little backstory.

For my thesis work, I worked on an algorithm to combine GPS tracklogs into what I call a spatial network. I chose this work because of my experience building ActiveTrails.com. As any excited graduate student (pre-thesis student, that is) I had grand ideas about what I was going to accomplish with my thesis work. Luckily for me, my graduate advisor guided me properly through the process, and I finally completed my work.

I do find it strange that only a written Thesis is required for an MS in Computer Science. Iâ�"ve decided that it only makes sense to post my code, that others might be able to experiment with my work without having to rewrite it from scratch. Now, Iâ�"m sure Iâ�"ve made plenty of mistakes in my code, and I hope that others can produce much better results then I, and not fall into the same lines of thinking that perhaps restricted my results.

I originally had plans to organize my code, clean it up, flush it full of comments, and organize my result files. And then I realized it might never happen. Iâ�"ve packaged my code, source files, and results into a zip file, and though it isnâ�"t perfectly clean, I hope itâ�"s useful for those who want to use it.

Progressive Spatial Networks: Learning from GPS Tracklogs (pdf link)

Source data, python source code, and result files (zip file).

Trouble with Random Long-Running request in ColdFusion 8

Tue, 10 Feb 2009 08:01:00-0700

I've been experiencing some run-time weirdness with ColdFusion for the past year or so, and I've finally decided to post my observations and see if anyone else has been having similar troubles. The trouble shows up on any number of scripts, but is most likely to appear with scripts that are called frequently. The screenshot I've included shows a Slow Request report from the ColdFusion Server Monitor. You can see that this request took 110 seconds to complete. The VERY strange part is that the runtime of the Application.cfc onRequest method (the outermost piece of code to run on any request) took only 468ms to complete. So what happened to the 109 seconds that were not spent executing my request? How can I prevent this from happening? Also observe the Min/Max/Avg response times for the script. At the time of this screenshot, this script had been called 250 times. Multiplying the average response time (.505 seconds) by the request count (250) tells us that this script has occupied 126.25 seconds of server time. Removing this one long running request (250-1, 126.25-110.234) from the statistics, we find the script has an average running time of 0.064 seconds, which is a more reasonable run-time, considering the reported min response time of 0.015 seconds. Eliminating these random long-running requests will do great things for the usability of the website, as well as free up server resources.

Misc Details

ColdFusion Enterprise 8,0,1,195765
Monitoring and Profiling Enabled
Windows 2003
Java 1.6.0_04

Summary

Scripts will randomly take MUCH longer then they usually do. There seems to be a massive discrepancy between the Response Time and the Time Taken by the onRequest method. Why is this happening, and what can be done about it?

Sorting SimpleDB queries on Multiple Attributes

Mon, 12 Jan 2009 16:20:10-0700

Image via CrunchBase

I was recently working with a dataset in Amazonâ�"s SimpleDB, and I needed to be able to sort query results on multiple attributes. SimpleDB currently only allows sorting by a single attribute, so I was stuck. The solution is rather simple: combine the columns that I need to sort on into a single new attribute, and use this new composite attribute to perform my sorting. I kept the existing attributes in their current form, but added a new attribute that is the concatenation of the two attributes I wished to sort.

SimpleDB sorts lexicographically, so concatenating two fields produces exactly the expected results provided you handle a few situations properly. There are a few things to keep in mind as you produce the multi-sort attribute.

Properly pad your fields

Sorting numbers in SimpleDB requires some encoding to provide the expected results. The SimpleDB documentation provides examples of this. If you have properly encoded your numeric fields, then the length of each attribute will be the same as all the rest. Each values of an attribute must be the same length to provide consistent results.

Align the sort order of each field

You can only sort your one field one direction; ascending or descending. If you wish to sort column 1 ascending and column 2 descending, then you must reverse the encoding of field 2 prior to concatenation, then sort the combined column ascending. You could also do the reverse: reverse the encoding of field 1, then sort the combined attribute descending.

The SimpleDB documentation describes encoding numeric values, but this concept applies to both numeric an alphanumeric fields, as long as the length of the attribute value is either the same, or can be padded to be the same.

Reversed encoding is nothing more then changing the data to be the reverse of what it really is. A single digit number attribute value could be subtracted from a base value, such that 0 becomes 9, 1 becomes 8, and so on. Characters can be reversed as well, turning a into z, b into y, etc. The exact encoding used will depend on the properties of the data being stored.

Keep in mind that the value of the multi-sort attribute never needs to be parsed or read: it is only used for sorting results. Even if the translation is an ugly one, it only needs to be done on an update.

Concatenate attributes in order of sorting preference

If you wish to sort on attribute A, then attribute B, followed by C, your combined attribute must be combined in that order, left to right.

Flexibility

This approach is fairly flexible, including creating a sorting attribute from any number of attributes and creating multiple sorting attributes, as long as the attribute length and number of attributes fall within the restrictions of SimpleDB.

While this technique is not new, I wanted to explain it here to provide hints to others with no idea how to solve the problem. Credit for revealing this concept to me go to Phil Windley, who told me that the only reason I donâ�"t know it already is that Iâ�"m â�Stoo youngâ�? to remember a time when most database platforms carried the same restriction. Iâ�"m just glad to have experienced old guys to learn from. :)

Cloud Computing - The 5th Utility

Mon, 22 Dec 2008 20:19:21-0700

Image by stan via Flickr

This post is part of a series of posts relating to distributed system design that I'm completing as part of my Computer Science MS program at BYU.

My 6th paper was written by some good folks at The University of Melbourne, Australia. They discuss the emerging cloud computing paradigm as the 5th Utility, and compare it with both clusters and grids. The paper (PDF) argues that clusters and grids cannot be considered a utility by themselves, but cloud computing fits the necessary requirements.

Clusters are groups of machines that work together to accomplish a single task, such as serve web content. Each machine is the same, and they all perform the same task. Clusters can scale in size to handle varying loads. Grids support multiple jobs of different characteristics, typically within a required framework. Clouds can scale (like clusters), and support a wide variety of jobs simultaneously. Clouds take scaling to such an extreme that they can scale to nothing, which neither clusters or grids support. This minimal commitment, without minimum usage levels, is what makes clouds so useful. Just as water, electricity, gas, and telephony (the first 4 utilities) can scale from nothing to very high usage, clouds can scale to any reasonable load.

As Iâ�"ve mentioned in several of my other paper reviews, it is very clear that cloud computing exists in a layer underneath clusters and grids. Clusters and grids can be built on top of cloud computing systems, as cloud systems utilize virtual machines as a hardware abstraction. The unique piece that makes it possible is the dynamic provisioning made available through the API. While some hosting providers can provision servers in only a few hours, cloud providers provision their resources within minutes, and sometimes within seconds.

At this point, there are few cloud providers, and each has their own API, terms of use, and types of services. As more providers enter the market, a consistent interface will be needed in order to tame the API chaos. While some services may end up sharing an API, consistency can also be provided via a meta-interface that can translate the users commands into whatever syntax is required by the particular provider and service being utilized. This layer can either be constructed in the cloud itself, in client based toolkits, or as a combination of the two. The concept of a metalayer is demonstrated in the paper through the creation of a meta-storage service, capable of storing data in several cloud services through a single API.

Just as interfaces are not likely to completely converge, the properties of each service are also not likely to be identical between providers. Speed, price, reliability, and other factors will vary, allowing users to select the proper service to fit their particular need. Some services will provide an SLA, providing service guarantees.

It is still very early in the development of cloud services, and Iâ�"m sure that we will be seeing new entries for years to come. Amazon has hinted at some of the services that will be made available in this next calendar year, including load balancing and monitoring and automation management. As we see more entries in the space, it will become easier to understand the strengths and weaknesses of cloud computing, as well as define itâ�"s limits.

Iâ�"m excited for the expansion of cloud computing, and I look forward to more studies that can help us understand better.

Really Bad reasons not to auto-scale cloud based systems

Sat, 06 Dec 2008 19:26:57-0700

O'Reilly writer George Reese posted today what I consider to be a poor evaluation of the perils of auto-scaling in the cloud.

He does mention the concept of using a governor to limit the power of the auto-scale agent to spin up servers (and spend money), but his insight ends there. Anyone following cloudy issues will have read Don MacAskill's excellent post this past June, where he explains their auto-scale operation, and the need to set limits.

George also makes a few arguments against auto-scaling, which I'll address briefly:

1. Amazon and other clouds cannot respond fast enough to increased capacity needs.

George claims that a 10 minute instance spin up time cannot respond fast enough to help. This is only true if you start to spin up your service when the existing is already (or nearly) toast. Common strategies involve already having some extra capacity running, so as to not immediately fold under an increase. Solving this problem is just tuning the thresholds.

2. Got any disgruntled employees, unhappy customers, or malicious competitors?

George claims that auto-scaling will waste your money in the event of a denial-of-service attack. What he doesn't mention is that a DoS on a non-auto-scaled system will likely take it down. At the very least, it will artificially inflate your usage anyway, and you will still have to spin up more resources to handle the load. I'd rather spend a few extra bucks and STAY UP.

3. So you think you'll stick some governors in place...

George's main claim here is that your governor is likely to be set at the wrong value. Although he doesn't explicity say, he seems to be implying that a governor can only be used to limit the total number of machines. SmugMug (in the aforelinked post) indicates that their governor limits the rate at which new machines can be started. Using this strategy, only the rate of traffic growth.

4. So what about getting slashdotted?

The main complaint here is that an auto-scale agent cannot tell the difference between true traffic growth and a random spike. Clearly, George has never worked with noise filters, which smooth data to reveal real trends. Evaluating load data from the past few minutes will allow agents to ignore spikes easily. Again, this is reduced to tuning thresholds.

5. Don't you lose a key value of the cloud without auto-scaling?

Despite George's claims that no value is lost, there are clear cases where auto-scaling can save your bacon. He claims that 'capacity planning' is the clear answer. I agree with him on the importance of capacity planning, but disagree that proper capacity planning eliminates the need to auto-scale. A good auto-scaling system can save quite a lot of money in cloud processing expenses, which will do wonders for the bottom line.

Summary

I'm not bashing capacity planning here. I believe that capacity planning concepts work very well with auto-scaling, that that proper user of governors and properly set thresholds are the right way to go.

I rarely respond to lousily written posts and dumb opinions, but this one irked me for some reason. At this point, I have nothing but logic and the experiences of others to rely upon. Over the next few years, I plan on gaining some extensive experience in auto-scaling cloud based systems, and perhaps then I'll be in a better position to dish a proper smack-down.

Amazon's Dynamo - Highly Available Key Store

Tue, 28 Oct 2008 20:41:25-0700

This post is part of a series of posts relating to distributed system design that I'm completing as part of my Computer Science MS program at BYU.

Amazon's Dynamo is a key-value storage system used internally at Amazon. It provides high-speed simple storage at extremely high levels of availability. The paper is available online.

In addition to describing the Dynamo system, this paper is an excellent resource for issues relevant to system design. I've selected a few of the topics mentioned in the paper, with some comments on each.

Symmetric Nodes

Dynamo clusters have two basic duties. Each read or write has a coordinating server, and each read and write uses several storage servers. Rather then form Dynamo into a 2 layer system with a layer of coordinators and a layer of storage servers, they formed the servers to perform both the coordination services and storage services. They cite simplified system provisioning and maintenance as their reasons for this choice.

This design does make scaling a little easier, as deploying a node adds resources to both services, but I think it applies so well here because of the predictable levels of work and the constant ratio between each of the services. An increase in requests will increase work for both the coordination services and the storage services, and adding additional servers will support both services to approximately the same degree.

In a system where the processing at each layer is highly variable, this strategy could result in an unbalanced situation, where launching more servers will only support one of the services. As always, considering the characteristics of the service will guide the proper solution.

Dynamically Scaling Under Load

Throughout the paper, they discuss the requirement that the system be capable of adding and removing nodes under load without impacting the external performance of the system. They noted that their first design required significant background processing, and they had to develop systems to carefully monitor and control the processing power consumed by the background services.

This issue is a sneaky one, because serving requests must continue during the process of re-distributing data and processing assignments. They mentioned some changes they made to reduce the background processing required, which allowed them to devote more processing time to serving requests.

Nearly every service will have background processing of some kind, and setting these services to be 'kind' to the performance of requests will help improve performance when the system comes under load.

Eventual Consistency

Dynamo (and several of the public Amazon Web Services) follow an eventually consistent model. By relaxing consistency, they can offer greater levels of availability and speed. Near the end of the paper, they reveal that 99.94% of the read requests had no artifacts of this relaxed requirement.

Their design leaves the process of conflict resolution to the client, which allows a resolution process specific to the nature of the data. Allowing the most recent write to win will work in many situations, and is easy to resolve. Under more specific requirements, the client may want to combine the conflicting data. Fortunately, passing the resolution buck to the client allows each application to resolve this issue in whatever way is appropriate. (CouchDB follows this same principle.)

Eventual consistency forces the consideration of failure; when it fails, how hard is it to recover? Understanding the consequences of failure can allow proper tuning of the system, as well as guide resolution processes.

Smart Clients

The paper describes two methods for distributing requests to the nodes in the cluster: load balancers and smart clients.

Load balancers are a common solution, and are considered standard fare in most systems. They forward inbound requests evenly among nodes, and then return the node's response to the client. Most large websites use load balancers to serve traffic, enabling many servers to act like one.

Smart clients are a powerful tool that can provide the same levels of fail over at increased speeds. If the client is programmable, then it can choose the server to ask itself, and eliminate the need for a load balancer. It chooses it's node according to embedded logic, and will also fail over to a different node if the first one does not respond. The clients must be 'smart' enough to discover cluster nodes and handle failure gracefully. In a load balanced configuration, these tasks are performed by the load balancer.

Smart Dynamo clients poll a random cluster node every 10 seconds to retrieve a list of nodes. The clients will also retrieve a new list if they detect a failure situation.

In many applications of smart clients, an update frequency of 10 seconds is far too fast and can be reduced to match the speed of adding or removing nodes. Smart clients can also choose cluster nodes based on network speed, allowing easy geographic load balancing.

The speed gains measured through the user of smart clients is remarkable. response times were more then 50% shorter as a result of removing the load balancer from the picture.

Thoughts

Each of these concepts, and others in the paper, can be applied to a wide variety of systems. I'm grateful to the Amazon team that published this paper, and I'm grateful for the insight it provides.

smart clients - improved performance by half by removing the load balancer. they polled frequently 10s for server lists, could be much longer. smart polling on failure.

My Android App Wishlist

Mon, 22 Sep 2008 01:31:02-0700

As I've been impatiently waiting for the impending announcement of T-Mobile's G1 Android phone, I've been thinking about which apps I will find (or build) and install. While current iPhone users might scoff at my lack of apps, there are a few here that are not currently possible with the current iPhone SDK. My list of desired phone abilities, with commentary:

Automagically publish my location to an online service. An obvious service would be Yahoo's FireEagle. By Automagic, I mean that no user interaction is required. This will enable me to write programs that respond to my change in location, such as automatically forwarding my work extension to my cell when I'm out of the office. It will also make it easier to find my phone when I've left it somewhere. This background functionality is not possible with the current iPhone SDK.
Find cheap gas. The first thing this app should do is find my location and then show me gas prices, with price and distance to station. Simple enough.

Maintain a car maintenance log. I'd like to record oil changes, repairs, tire rotations and similar info, for multiple cars. The ability to log tank info to calculate mileage is a plus.

Record business mileage logs. This would help me keep track of business miles without that pesky paper log book in the glove box.

Allow updating a personal health log. Weight is required, and a quick way to enter exercise and gym workouts would be great. If I don't update my weight in awhile, it ought to pester me.

Provide an intelligent to-do list. This list should also be available online. List sharing between multiple people(phones) is a must. Location aware lists are a plus.

A real Twitter client. Being able to post and read tweets is required, but what I want here is a client that will monitor in the background, and update me to new tweets as if they were sent via SMS. Tweets should show as notifications in the background. Again, the ability of an app to receive updates and post them to the notifications bar (without launching a screen) is beyond the current abilities of the iPhone SDK.

Provide on-the-spot reviews and price comparisons. I'm cheating a little here, as one of the Android Developer Challange winners is CompareEverywhere, which fits this nicely.

Behave appropriately to the situation. Cheating again. When I'm at church, be silent. When at home and plugged into power, connect via Wifi. See Locale.

Make Geocaching dead simple. Allow viewing nearby geocaches, help me find it, and make it easy to log on the spot. A little recreation for a free moment here and there.

Post pictures online. Many online services for this, and I suspect this might be built into the phone upon delivery.

Calendar that talks with Google Calendar. Getting alerts via the phone instead of SMS would be great, as being able to easily add and edit on the phone. I know the mobile version of Google Calendar does this, but having a native app would be nice.

Contacts that sync online. I'm tired of entering all my contact info into each new phone that I buy. I'd like to be able to edit them online, and have them easily (and quickly) move to a new phone. This is a no-brainer with the Google Contacts API, and I suspect this might be baked in as well.

Well, there you have it. As you can see, I'm easily pleased. I've been waiting since the first Android announcement for the experience, and I'm looking forward to my new phone.

Missing Pieces of Cloud Computing

Thu, 21 Aug 2008 09:00:56-0700

Between Google's AppEngine and Amazon's Web Services platform, you can build almost everything you need to run a scalable system. The 'almost' is what is currently preventing these platforms from supporting truly beautiful cloud systems. I believe that the two most important pieces of Cloud Computing are scheduled processes and load balancing.

Scheduled Processes in a cloud computing application are necessary for tasks both simple and complex. On the simple side, a task can be used to send reminder emails to system users, or generate periodic reports based on system data. On the complex side, scheduled processes might communicate with a processing grid via asynchronous queues, or perform orchestration functions to manage a scalable computing cluster. The best provider of this service is likely to be Google's AppEngine.

During an AppEngine session at Google IO, the presenter did mention that offline processing was a planned (and often requested) feature, but they did not give details. Their offline processing may only be the ability to launch threads and return rapidly to the user, but I hope it also contains some mechanism for the timing of script execution.

In the meantime, it is possible to invoke processing scripts from an automated system, either running on hardware you own or a service such as WebCron, but it will be nice to have a more elegant solution.

Load Balancing allows for a group of computers to be joined into a cluster, allowing service of more requests then a single machine can handle. While AppEngine does not require load balancing to scale, clusters within EC2 need this functionality if they service public requests. Load balancers exist in both hardware and software form, and generally provide several methods of load distribution, system failure handling, and sometimes SSL termination for secure connections. Currently, software load balancers must be used to load balance within the EC2 environment, and at least 2 of these machines are required to gracefully handle machine failures.

Providing this service in a scalable way would greatly simplify the design of computing clusters within the EC2 environment. It would also make managing public IP addresses a little easier, as only the load balancing service needs a public address. Amazon is currently testing a form of persistent storage within their EC2 environment, and I imagine that a load balancing solution could be offered in a similar form. Amazon may decide not to offer this service, preferring 3rd party solutions.

In Summary, I look forward to the future of cloud infrastructure. I'm excited to be able to develop to these platforms, and I expect great things in the future from both of these companies and others as well.

Extracting Pictures from Flickr with Python

Sat, 12 Jul 2008 09:03:00-0700

A few years ago, when Flickr was new, we made kind of a silly decision to only store our images on Flickr. While this did make a transition between computers easier and freed up a little bit of drive space, we decided that we'd like to pull those pictures back onto our own system.

In the past, I've tried several Flickr Downloadr (missing 'e' intended as a pun) programs, and everything choked or did strange things. Last night, right after I crawled into bed, I decided that I knew a better way.

I crawled out of bed, did a bit of Googling, and found an excellent Python Flickr library, and someone that wrote a python script to backup Flickr pictures to Amazon's S3. In 15 minutes or so, I had a solution that would page through our public pictures, check to see if they were already downloaded, and store them in year and month folders.


import flickr
import urllib
import os.path
import os
page = 1
total_photos = found_photos = 0
while True:
	photos = flickr.people_getPublicPhotos('68432331@N00', 100, page)
	if not len(photos):
		break
	for photo in photos:
		total_photos = total_photos + 1
		photoYear = photo.datetaken[0:4]
		photoMonth = photo.datetaken[5:7]
		photoURL = photo.getURL('Original', 'source')
		photoPath = "C:\FlickrPics\%s\%s\%s.jpg" % (photoYear, photoMonth, photo.id)
		if not os.path.exists(photoPath):
			if not os.path.exists("c:\FlickrPics\%s" % photoYear):
				os.mkdir("c:\FlickrPics\%s" % photoYear)
			if not os.path.exists("c:\FlickrPics\%s\%s" % (photoYear, photoMonth)):
				s.mkdir("c:\FlickrPics\%s\%s&" % (photoYear, photoMonth))
			urllib.urlretrieve(photoURL, photoPath)
			found_photos += 1
		page = page + 1
		print "  Moving to page %s" % page
print "Found %s photos, saved %s new photos" % (total_photos, found_photos)

While running the script, I noticed that the obvious slow part of the process was downloading the images. I wanted a way to download them in parallel, and found my solution in an excellent Python thread pool solution.

I added the following method to their code, that will tell me the number of jobs pending in the thread pool:


#new method in the ThreadPool class
def getWaitingTaskCount(self):
	self.__taskLock.acquire()
	count = len(self.__tasks)
	self.__taskLock.release()
	return count

I had to rewrite my code a little to put the download code into a method, and then I let 'er rip. Downloading now happened 3 at a time. It took about 2 hours to download our 5GB collection of approx 3,300 pictures, and the job was done!

I don't bother with authenticated requests (all of our pictures are public), and I don't do any error checking. It is a one off script, after all. :)

I love python. I really do.


import flickr
import urllib
import os.path
import os
import threading
from time import sleep

#code from ThreadPool not shown.
#Copied from linked solution, with addition new method listed above

pool = ThreadPool(3)

def getPicture(data):
	photoURL = data[0]
	photoPath = data[1]
	urllib.urlretrieve(photoURL, photoPath)
	print photoPath

page = 1
total_photos = found_photos = 0

while True:
	photos = flickr.people_getPublicPhotos("68432331@N00", 100, page)
	if not len(photos):
		break
	for photo in photos:
		total_photos = total_photos + 1
		photoYear = photo.datetaken[0:4]
		photoMonth = photo.datetaken[5:7]
		photoURL = photo.getURL('Original', 'source')
		photoPath = "C:\FlickrPics\%s\%s\%s.jpg" % (photoYear, photoMonth, photo.id)
		if not os.path.exists(photoPath):
			if not os.path.exists("c:\FlickrPics\%s" % photoYear):
				os.mkdir("c:\FlickrPics\%s" % photoYear)
			if not os.path.exists("c:\FlickrPics\%s\%s" % (photoYear, photoMonth)):
				s.mkdir("c:\FlickrPics\%s\%s&" % (photoYear, photoMonth))
			# Insert tasks into the queue and let them run
			pool.queueTask(getPicture, (photoURL, photoPath))

			found_photos += 1
	page = page + 1
	#don't get too far ahead of the download threads
	while pool.getWaitingTaskCount() > 10:
		sleep(1)
	print "  Moving to page %s" % page
# When all tasks are finished, allow the threads to terminate
pool.joinAll()
print "Found %s photos, saved %s new photos" % (total_photos, found_photos)

jEditable TinyMCE Plugin

Thu, 12 Jun 2008 13:25:00-0700

Over the past view weeks, I've been working with Mika Tuupola, author of the jEditable jQuery extension. jEditable makes edit in place forms easy, and is itself extensible, allowing the creation of custom edit types, such as a date picker.

I've been using TinyMCE as a richtext editor, and decided to create a custom edit type for jEditable that makes TinyMCE easy. Mike added a few additional customization hooks that enabled me to complete the plugin, and I've included example code below. This example is not meant to be a full and complete explanation, so make sure you dig into the jEditable and TinyMCE documentation.

A big thanks to Mike for his help. He has released the updated jEditable code that contains some additional hooks needed by my example code below.

First, I include the necessary javascript files for each library. Your specific versions or names may vary:

Then, I setup TinyMCE, and the TinyMCE jEditable plugin as well:

This next block of code initializes TinyMCE, first as a regular jQuery plugin (not in place editable), and then as the jEditable enabled in place editor:

In this example, both of the selected elements are HTML textareas. Clearly, your settings for jEditable and TinyMCE will change with your own needs, but this should get you started.

Finally, a big thanks to Mike, for writing the jEditable plugin in the first place, and for being so supportive and patient in helping me write this TinyMCE plugin!

Edited 24Jan09: Applied IE6 Fix as recommended by Rick in comments.

SamSpeak - Development

Amazing Things happening at Kynetx - Come to Impact

Changing the World at Kynetx

Sam Rides 1000: Augmenting the Web

Dataset Conversion

Building the App

Annotating My Blog

Sam is riding 1,000 miles. Progress: #{(daymiles > 0 ? daymiles + " Today, " : "")} #{(weekmiles > 0 && weekmiles != daymiles ? weekmiles + " This Week, " : "")} #{(monthmiles > 0 ? monthmiles + " This Month, " : "")} #{totalmiles} Total.

Annotating Google's Homepage

Sam is riding 1,000 miles.

Shameless Plug

Sam Rides 1000: Collecting Ride Data using the Android Powered G1

Additional Stats

Sam Rides 1000: An exercise in collecting data and web augmentation

Tracking My Progress

Installing The Action Card

Progressive Spatial Networks

Trouble with Random Long-Running request in ColdFusion 8

Misc Details

Summary

Sorting SimpleDB queries on Multiple Attributes

Properly pad your fields

Align the sort order of each field

Concatenate attributes in order of sorting preference

Flexibility

Cloud Computing - The 5th Utility

Really Bad reasons not to auto-scale cloud based systems

Summary

Amazon's Dynamo - Highly Available Key Store

Symmetric Nodes

Dynamically Scaling Under Load

Eventual Consistency

Smart Clients

Thoughts

My Android App Wishlist

Missing Pieces of Cloud Computing

Extracting Pictures from Flickr with Python

jEditable TinyMCE Plugin