Over sushi a few nights ago, I had the opportunity to bring Drummond Reid up to speed with what Kynetx has been doing. After several months in the trenches, it was a revealing experience to climb a tower and see just how far we've come. As I unfolded item after item, I was surpised to find another thing to describe.

At our last Impact Conference, we unfolded the Kynetx Engine, and demonstrated some ways to use Kynetx in your quest to do amazing things.

At this next Impact Conference, we will deliver an update on the progress with the engine, the improvement of our tools, and all of the things that have kept us busy over the last few months. If you want to hear about some of the stuff I downloaded to Drummond, you'll want to attend the conference. If you came to the last Impact, this one will be better. If you didn't, then it's time for you to understand what we are doing and what it means for the world.

Oh, and the food will be worth the price of admission alone. Seriously. Sign up for Impact.

Kynetx Impact Spring 2010
April 27-28 2010
Miller Free Enterprise Center (MFEC)
at Salt Lake Community College
9750 South 300 West
Sandy, UT 84070

Use Code FOK2010 for a 33% discount on the conference price.

I've been pretty quiet on my blog lately, and I have a really good excuse. Now that I've graduated and have time to get involved in some serious endeavors, I've joined the folks at Kynetx. I've been contributing in their efforts to change the world. Indeed, this does change everything.

At Kynetx, we believe that experiences can be made better through better use and understanding of context. (Gartner agrees...) Context is data in time and space: who we are, what we are doing, and what our purpose is. Kynetx has what we call a Context Automation Engine, which does the heavy lifting required to produce intelligent applications. Using our engine, you can create complex applications easily, and deploy them fast.

I'll take a break now and again from my work to post more info, including examples and more information about what we are doing. If you want to learn more, attend the Kynetx Impact developer conference Nov 18-19. We will cover our technology and our vision.

If you are interested in using our platform to add contextual intelligence to your applications, go sign up. Use code Windley50 for a 50% discount, and I'll see you there!

In my previous two posts, I introduced my project and described data collection using my G1 and Google Spreadsheets. Today, I'm going to show you how I used Kynetx Network Services to add my ride stats to my personal blog and to the Google homepage.

Dataset Conversion

Google Spreadsheets, where my stats are calculated, can publish data in a variety of formats. It cannot publish JSON data, so I use Yahoo's YQL to convert the data from CSV to json, with the following statement:

(See the raw JSON results)

Building the App

I then build my Kynetx App in AppBuilder, defining the following datasource in the Global block:

Since I ride in the morning and the evening, I cache the dataset for 2 hours. This keeps the data fairly current, but still keeps the service fast.

Annotating My Blog

I have two rules, the first of which adds stats to my personal blog:

I set the rule to fire on my blog's domain, and then use the pick() method to extract different totals from the json dataset declared in the Global block. I construct a message string that varies depending on the different stat values. Finally, I replace the text at the top of my blog page with the message.

To run the Kynetx application on my blog, I plant Kynetx tags on my blog. This enables everyone to see the Kynetx Application with no installs or Action Cards. The html tags are available within AppBuilder, and I simply copied them into the template for my blog.

Annotating Google's Homepage

My second rule is activated by an Action Card installed on the user's computer (instructions for installing this are in my first post). My second rule is very similar to the first rule, with some minor difference in inserted HTML and appending the message to the existing site, instead of replacing anything on the page.

Activating Kynetx Rules with an Action Card also requires an update to the Dispatch block of the rule, adding this line:

I also generate the card inside AppBuilder, providing a custom image that I created using Pixlr

And there you have it. Sam rides 1000 miles, with automated stats provided by Android MyTracks, Google Spreadsheets, YQL, and Kynetx Network Services.

Shameless Plug

Kynetx is a cloud based automation engine, capable of doing the things I've demonstrated and much, much more. If you'd like to use Kynetx Network Services, sign up for an account, and start using AppBuilder.

On my Android powered T-Mobile G1, I'm using the free My Tracks application to record my rides. I start recording just before I start, then throw it in my pocket or bag. I stop recording at the end of my ride, then use the Upload to Google option in the map menu. You can upload the track to My Maps within Google Maps, but my rides are very similar, so I usually only upload to Google Spreadsheets.

Uploading to Google Spreadsheets creates a new spreadsheet in Google Docs, with one page for ride data, and another for stats. The program creates a new spreadsheet for each activity type, so I make sure and select Cycling when I stop recording at the end of my ride.

Additional Stats

It's important not to manually change too much on the Log sheet, as the program will get confused, but everything else in the spreadsheet is open to tinkering. In addition to the total miles and total time stats, I wanted to calculate daily, weekly, and monthly totals.

Before I computed those stats, I had to handle the date field uploaded by the My Tracks application. Google Spreadsheets cannot parse the provided date as a date value, so I had to help it along. I created a Dates sheet to accomplish that task. Rather then try and explain what I did, see this spreadsheet, which contains my modifications and formulas:

Sample Spreadsheet with modifications.

I then added some additional stats, first by calculating the start of the date range I wanted to sum, then using SumIf() to only add the mileage and time from the period I wanted.

Finally, I published the Stats sheet of my spreadsheet, so I could use it as a dataset for my Kynetx Application, which I'll explain in a future post.

May 15, 2009 was National Ride your Bike to Work day, and I pulled out my bike and rode the 6 or so miles between my house and the Kynetx offices at Thanksgiving Point. I enjoyed it, and rode a few more times in the next week. I was musing how many miles I could ride this summer, and mentioned it to my wife. She promptly challenged me to ride 1,000 miles this summer.

I better say here that I'm not a cyclist. I haven't ridden 1,000 miles in the past 5 years, all combined together. While 1,000 miles might not be much for a cyclist, it is quite the challenge for me.

Tracking My Progress

In addition to making a few bike repairs and buying some commuting tires for my mountain bike, I immediately cooked up some geeky ways to track my progress and share my results with my family and friends.

I'm a (BIG) Android fan, and so I'm using my T-Mobile G1 as a cyclometer. I'm using the My Tracks application, which records both the route of my ride and my ride stats. After my ride, the app uploads my stats to a Google Docs Spreadsheet, where stats are calculated. I'll share more about that in a future post.

To share my progress, I'm using Kynetx Network Services (KNS) to augment my personal blog with my stats. If you are reading this post on my blog, look at just under the title for my updated stats. KNS pulls my ride stats from the Google Spreadsheet and annotates my website. All I had to do was plant some javascript tags in my blog's template to activate the Kynetx Application that makes the change.

My friends and family don't visit my blog EVERY day, so I've also produced an Action Card that displays my ride stats on Google's home page. KNS allows me to augment websites for anyone that has my Action Card installed. I'll explain more about how I wrote my Kynetx Application in a future blog post, but for now, you can install my "Sam Rides 1000" card to track my progress on Google's home page.

Installing The Action Card

An Action Card is a type of Information Card that allows your web experience to be augmented with a Kynetx Application. You activate the application by installing an Action Card Selector if you don't already have one, and then installing the card. You can disable or remove the card if and when you don't want to use the application.

1. Verify you have a supported browser: IE/FF on Win, FF on OSX
2. Install Azigo
3. Install the Sam Rides 1000 Action Card

Then, browse to Google's main page to see my stats: http://www.google.com

When I finish a ride and update my stats, you'll see the new numbers. If I'm falling behind, be sure and give me a nudge!

When Twitter was new, the only people that followed me were early adopter, technical people. As twitter has become more mainstream, I have more and more non-technical friends that see my tweets. This is also a result of hooking my twitter feed into Facebook (to keep up appearances), where the tech-to-non ration is very low.

I'll occasionally get a response to a technical tweet that reads "I don't understand any of your updates!" or "Whatever!" and I usually just ignore them. (Seriously people, you know who I am!)

Yesterday, I posted a technical tweet, and the response made me chuckle. My tweet:

The response, posted by my Paramedic Fireman Brother:

Until Twitter (and Facebook) give us better tools to manage different circles of friends, I do enjoy a sense of humor!

Image by stan via Flickr

This post is part of a series of posts relating to distributed system design that I'm completing as part of my Computer Science MS program at BYU.

My 6th paper was written by some good folks at The University of Melbourne, Australia. They discuss the emerging cloud computing paradigm as the 5th Utility, and compare it with both clusters and grids. The paper (PDF) argues that clusters and grids cannot be considered a utility by themselves, but cloud computing fits the necessary requirements.

Clusters are groups of machines that work together to accomplish a single task, such as serve web content. Each machine is the same, and they all perform the same task. Clusters can scale in size to handle varying loads. Grids support multiple jobs of different characteristics, typically within a required framework. Clouds can scale (like clusters), and support a wide variety of jobs simultaneously. Clouds take scaling to such an extreme that they can scale to nothing, which neither clusters or grids support. This minimal commitment, without minimum usage levels, is what makes clouds so useful. Just as water, electricity, gas, and telephony (the first 4 utilities) can scale from nothing to very high usage, clouds can scale to any reasonable load.

As I’ve mentioned in several of my other paper reviews, it is very clear that cloud computing exists in a layer underneath clusters and grids. Clusters and grids can be built on top of cloud computing systems, as cloud systems utilize virtual machines as a hardware abstraction. The unique piece that makes it possible is the dynamic provisioning made available through the API. While some hosting providers can provision servers in only a few hours, cloud providers provision their resources within minutes, and sometimes within seconds.

At this point, there are few cloud providers, and each has their own API, terms of use, and types of services. As more providers enter the market, a consistent interface will be needed in order to tame the API chaos. While some services may end up sharing an API, consistency can also be provided via a meta-interface that can translate the users commands into whatever syntax is required by the particular provider and service being utilized. This layer can either be constructed in the cloud itself, in client based toolkits, or as a combination of the two. The concept of a metalayer is demonstrated in the paper through the creation of a meta-storage service, capable of storing data in several cloud services through a single API.

Just as interfaces are not likely to completely converge, the properties of each service are also not likely to be identical between providers. Speed, price, reliability, and other factors will vary, allowing users to select the proper service to fit their particular need. Some services will provide an SLA, providing service guarantees.

It is still very early in the development of cloud services, and I’m sure that we will be seeing new entries for years to come. Amazon has hinted at some of the services that will be made available in this next calendar year, including load balancing and monitoring and automation management. As we see more entries in the space, it will become easier to understand the strengths and weaknesses of cloud computing, as well as define it’s limits.

I’m excited for the expansion of cloud computing, and I look forward to more studies that can help us understand better.

O'Reilly writer George Reese posted today what I consider to be a poor evaluation of the perils of auto-scaling in the cloud.

He does mention the concept of using a governor to limit the power of the auto-scale agent to spin up servers (and spend money), but his insight ends there. Anyone following cloudy issues will have read Don MacAskill's excellent post this past June, where he explains their auto-scale operation, and the need to set limits.

George also makes a few arguments against auto-scaling, which I'll address briefly:

1. Amazon and other clouds cannot respond fast enough to increased capacity needs.

George claims that a 10 minute instance spin up time cannot respond fast enough to help. This is only true if you start to spin up your service when the existing is already (or nearly) toast. Common strategies involve already having some extra capacity running, so as to not immediately fold under an increase. Solving this problem is just tuning the thresholds.

2. Got any disgruntled employees, unhappy customers, or malicious competitors?

George claims that auto-scaling will waste your money in the event of a denial-of-service attack. What he doesn't mention is that a DoS on a non-auto-scaled system will likely take it down. At the very least, it will artificially inflate your usage anyway, and you will still have to spin up more resources to handle the load. I'd rather spend a few extra bucks and STAY UP.

3. So you think you'll stick some governors in place...

George's main claim here is that your governor is likely to be set at the wrong value. Although he doesn't explicity say, he seems to be implying that a governor can only be used to limit the total number of machines. SmugMug (in the aforelinked post) indicates that their governor limits the rate at which new machines can be started. Using this strategy, only the rate of traffic growth.

4. So what about getting slashdotted?

The main complaint here is that an auto-scale agent cannot tell the difference between true traffic growth and a random spike. Clearly, George has never worked with noise filters, which smooth data to reveal real trends. Evaluating load data from the past few minutes will allow agents to ignore spikes easily. Again, this is reduced to tuning thresholds.

5. Don't you lose a key value of the cloud without auto-scaling?

Despite George's claims that no value is lost, there are clear cases where auto-scaling can save your bacon. He claims that 'capacity planning' is the clear answer. I agree with him on the importance of capacity planning, but disagree that proper capacity planning eliminates the need to auto-scale. A good auto-scaling system can save quite a lot of money in cloud processing expenses, which will do wonders for the bottom line.

Summary

I'm not bashing capacity planning here. I believe that capacity planning concepts work very well with auto-scaling, that that proper user of governors and properly set thresholds are the right way to go.

I rarely respond to lousily written posts and dumb opinions, but this one irked me for some reason. At this point, I have nothing but logic and the experiences of others to rely upon. Over the next few years, I plan on gaining some extensive experience in auto-scaling cloud based systems, and perhaps then I'll be in a better position to dish a proper smack-down.

I was one of the folks to pre-order the first Android phone, the T-Mobile G1. The G1 launched today and is available at T-Mobile locations. I pre-ordered mine (and one for my wife) and received it yesterday afternoon. I've had several requests for a review, so I'll post a few initial thoughts here.

First, I love the phone. Early reviews called it brickish, ugly, and all sorts of horrible names. I find it to be none of those things. In fact, the phone is smaller then I was expecting.

Getting the phone setup was VERY EASY. On first boot, you enter your Google account information, or set up an account if you don't already have one. Within a minute or so, my contacts were downloaded from Gmail, I had a mail alert, and my calendar was available.

I bought a black phone, and my wife the bronze color so we could tell them apart. The bronze color is much nicer then I expected, but the lighter keyboard on the bronze model was slightly harder to read. The keys on the black model were very easy.

I'm not in a T-Mobile 3G area, and I knew that when I ordered the phone. I find the EDGE speeds to be just fine for most uses of the phone. I was able to listen to a podcast within seconds of starting the download, and I find most apps to be responsive when sending and receiving data. The mail app is particularly nice, as it caches data locally and handles sending email in the background.

The capacitive screen is flush mounted, like the iPhone. It's a little smaller then the iPhone screen, but the same resolution. It doesn't feel cramped, and the color is great.

The selection of apps in the Android Marketplace is a little thin at the moment, but user submission will open on Monday. There will also be an Over The Air firmware update in the next day or so, which contains minor updates and fixes.

I'm sure I'll have more to say later, but I wanted to mention these few things that I hadn't found on other websites and reviews.

Every group of users has a few extremists, and I have run into quite a few of the Apple iPhone variety lately. Most of the negative comments about the first Android phone come in the form of an iPhone fan trash talking the G1. I find these types of comments very interesting, because iPhone users should be pretty excited about the G1.

Before I collect even more of the comments I'm talking about, understand that I'm not claiming that all iPhone users should be ready to convert. I'm talking about the positive effects of competition, and the ultimate win for users of either device.

It is clear that the iPhone has influenced the design of the G1. It will also be clear over the next year or so that the Android phones will influence the features of the iPhone. While I am sure that Apple is not about to adopt the physical design of the G1, there are other things there that Apple will be sure to emulate.

The biggest issue at hand is the ability of Android phones to run background processes. Android's architecture allows for applications to perform background tasks, yet allows the operating system to suspend tasks quickly to allow for answering calls and other phone related tasks. The applications that can be built with Android will have a technical leg up on iPhone apps in this important aspect.

It won't be long before Apple follows suit. They will need to allow some sort of background processes in order to match the level of application sophistication and usability of Android Apps.

Also at issue is the restrictions that Apple places on developers with their restrictive NDAs and practice of denying apps that compete with Apple products in any way. Their recent changes to some NDAs are a sure sign they are feeling pressure from a more open platform.

This is where iPhone users gain. By pushing each other's limits, Apple and Google will produce some great technology, and both iPhone fans and Android fans will gain. So seriously, everyone calm down a little, and enjoy the market pressures bring even greater phones to all of us.