high frequency trading

Posted by Don 13 Sep 2010 at 10:37AM

"That was a $200 watch!" "$200 is what you paid for it. What its worth is what you can sell it for."

The "price" or the "worth" or the "value" of something involves time, economic conditions, social norms, even one's perception of reality. High speed trading has caught my attention recently.

Why are "high frequency traders" given special access to order information? I thought these traders competed on speed alone.

From the New York Times, May 2010:

The answer — that it all started with an apparent error — infuriated Mr. Clancy. “The market was never down one thousand points,” he said. “Procter & Gamble should never have traded at $39. But a lot of people lost money as if the prices were meant to drop.” For a short while, traders started to distrust what they were seeing. “There was no pricing mechanism,” Mr. Clancy said. “There was nothing. No one knew what anything was worth. You didn’t know where to buy a stock or sell a stock.”

That sounds like an engineered, intentional process, or a subplot in a work of computer science fiction.

IceCondor Engagement

Posted by Don 17 Aug 2010 at 04:13PM

*unfinished*

What is next for IceCondor? What are the values and aims of the project? How is its success measured?

A story in TechCrunch?

One answer is engagement. The number of people who download the app, then the number of people who use the service.

Possible customers:

  • Delivery companies: Portland Pedal Power, Delivered Dish, Organics to You

federated food carts and beer and venues oh my

Posted by Don 10 Aug 2010 at 03:43PM

One concept, many representations

A pattern has emerged over the last year. The same object is described by multiple services.

Checkins

The first time this issue came up was in the description of a venue for the location check-in service at shizzow.com. The same cafe was place #12 at shizzow, and place #44 at foursquare.com and place #321 at gowalla.com.

Food Carts

The next time this pattern emerged is when foodcartpages.com was being built. Here is a wiki page for each food cart in Portland. There are other fine services that exist to catalog food carts in portland. This service is uniquely accessible and supports details of a cart that noone else does.

Beer

Portland loves its beer. While working on the taplister.com mobile app for Android, the notion of the 'authoritative' record to describe a single beer came up. There are national-size beer databases that do very well at capturing a percentage of all beers made.

Approaches

Two approaches that have come up so far are the commons and layering

The Commons

The commons is a centralized document store, with wiki-style access control, and an easy to use RESTful interface. . Other players would have to see the value in such a commons to be motivated to use it. By document store, i mean in the no-sql sense. A record is a json blob that can contain whatever keys and values the last author felt was important. There would be some keys that become a defacto standard.

Layering

With check-in venues as an example, a record for Bob's cafe on foursquare would have some number of links to the same venue in other services. The client that started with the venue description from a particular service, could retrieve the other records and composite them into a fuller description of the venue.

IRC and Abundance

Posted by Don 03 Aug 2010 at 10:16AM

One of the freedoms that comes from education is the realization that knowledge provides a feeling of abundance. Songs on mainstream radio talk about material wealth and possessions. The realization that knowledge has utility and value much like a dollar or a ring opens worlds of possession beyond count. I think its a better approach to satisfaction. Once it is understood how infinity manifests itself in, well, infinite ways, it can feel like Bilbo Baggins entering Smog's lair inside Lonely Mountain, filled with piles of gold and jewelery.

"The Cosmos is full beyond measure of elegant truths Of exquisite interrelationships Of the awesome machinery of nature" -- http://www.symphonyofscience.com/

I find it interesting that IRC exposes a kind of infinity. If you were to design a system of chatrooms, it might be tempting to say there are system operators and when someone wants a chatroom, they ask a operator for permission and then it gets approved, yada yada. Instead, IRC starts with an infinite number of rooms with no access control. A room is created from your desire for that room to exist. You can be king of your room or any number of other rooms that haven't been visited yet. This is much closer to the workings of the Internet, and I believe, reality in general. The room itself is freely available but starts with no value. Its value comes from who else comes to it. Access control and policy can be layered on a per-room basis.

emotional processing

Posted by Don 03 Aug 2010 at 10:05AM

i woke up this morning and on my way out of the house I called the police on someone who was trespassing. There was some discussion with this person. Some of it was confrontational in the he-said she-said variety. The conclusion with the police was as I expected. I'd like to put aside the specifics and focus on the emotions and processing/thought patterns. I felt hesitant during the approach, defensive but understanding during the discussion, supported but shaky after leaving the scene. Happening at the place where I live added some personal safety risk.

What I noticed was the re-tellings. It was very important to me to retell the story. Not having roommates or a significant other, I rely on a larger number of people i know as friends or acquaintances for emotional reflection and another point of view. There were four re-tellings. The length and places of emphasis adapted to each person and situation. update: at lunch was another more detailed retelling with a close friend. that was particularly helpful.

The re-tellings, in my case, reflect a lack of emotional processing while the event happened. Getting the feelings out at the moment of the trauma is by far the most effective and healthy way to experience these feelings. I tend to become emotionally overwhelmed, and i do an extra-duty amount of filtering and measuring the other person's possible responses. Afterwards, I'm left with residual powerful emotions that take extra effort to process after the fact. Like hot candle wax, emotions happening now can be formed and directed. Once its cool, its a huge pain to do anything with.

LoveJoy Fountain

Posted by Don 28 Jul 2010 at 02:00PM

LoveJoy Fountain is one of the most beautiful places in downtown portland. Its also the entrance to an unknown netherworld, as evidenced by these photos.

The Door

"From what it says on the map I should guess there is a closed door which has been made to look exactly like the side of the Mountain." --The Hobbit The Gateway

The Gateway

The Door

The Dungeon

The Dungeon

Social Networking Near and Far 1

Posted by Don 27 Jul 2010 at 10:14AM

Over the last few weeks, three times I've been amazed by the subtle influence of social networking.

Near

I went to the PDX Bridgefest lighting on Sunday. It turned out to be a non-event - the lighting was overpowered by the streetlights on the bridge. Many people had turned out and were hanging out along the bridge railing. I met @maxticket and@trisimon there who thought they'd do a light show of their own. I took this photo:

I tweeted the photo here:

After the bridge "event", the three of us headed to cartopia for some fried pie. We arrived about 30 minutes later. At the counter was @piesiren who had been working that day. I mentioned the bridge lighting and the photo. She said she already knew about it because she saw my tweet and my photo on her phone. The documentation of my experience is recreating that experience in the minds of people who saw it on my twitter stream. I turned to sit down and wait for the pies to come. Sitting at the next table with three others was @joelgibby, whom i know and usually see at beer-n-blog. After a short chat it turned out that he, too, knew of our trip to the bridge and saw the photo.

Far

I follow a few celebrities on twitter, including @alyankovic, who tweeted a funny photo about being at the Alamo. Peewee Herman's film PeeWee's Big Adventure (1985) has a subplot of reaching the basement of the Alamo. I think its likely that these two stars lead separate lives but twitter created an intersection for them. A 25 year old reference to a plot line made for a hilarious tweet.

Another example is a third person connecting two famous people in an unexpected way. Leonard Nimoy @therealnimoy ends all tweets with an abbreviation of Spock's catch-phrase "Live Long and Prosper" (LLAP). This apparently became popular enough to print T-Shirts with just the acronym, and he tweeted it here.

Eight days later, in another part of the nation, Nichele Nichols @realnichelle, visited NASA.

A fan was at Nichol's talk and wore the LLAP shirt.

I see this correlation interesting. The reach of a tweet can be vast though its impact is impossible to measure as each person takes it differently. This simple choice of t-shirts connected the two former co-stars in a subtle but real-world way.

icecondor background process again

Posted by Don 07 Jul 2010 at 10:07AM

The background process, called Pigeon, in IceCondor is actually kind of a joke. it does not stay running no matter how i coax it. even with android 2's startForeground, it dies like a canary in a coalmine. Pigeon gets killed soon after the app is paused or the phone goes to sleep. What does work is timers. If the background process kicks off a recurring timer, that will stay running quite reliably. Also the callback registration for location change. This continues to fire because the location manager is part of the android system. A system thread, from what i can gather, executes the app's callback.

That also means its a good idea to keep the work done in that callback to a minimum. The callback keeps a pigeon-scope reference to the last receive location (for quick lookup from the app (called Radar)). It has a routine to determine if this fix is a 'valuable' one, then it adds that location in JSON format to a queue. It spins off another thread to push the top of the queue. The callback likes to finish quickly, and yet we want to get a valuable fix to the server ASAP.

 
private Location last_recorded_fix, last_local_fix;

public void onLocationChanged(Location location) {
	last_local_fix = location;
	long time_since_last_update = last_local_fix.getTime() - (last_recorded_fix == null?0:last_recorded_fix.getTime()); 
	long record_frequency = Long.decode(settings.getString(SETTING_TRANSMISSION_FREQUENCY, "180000"));
	rssdb.log("pigeon onLocationChanged: lat:"+location.getLatitude()+
			  " long:"+location.getLongitude() + " acc:"+
		       location.getAccuracy()+" "+ 
		       (time_since_last_update/1000)+" seconds since last update");
		if (on_switch) {
		if((last_local_fix.getAccuracy() < (last_recorded_fix == null?
				                            500000:last_recorded_fix.getAccuracy())) ||
				time_since_last_update > record_frequency ) {
			last_recorded_fix = last_local_fix;
			last_fix_http_status = 200;
			long id = rssdb.addPosition(locationToJson(last_local_fix));
			rssdb.log("Pigeon location queued. location #"+id);
			pushQueue();
		}
	}
}
A push, with some error handling.
 
public void pushQueue() {
	Timer push_queue_timer_single = new Timer("Push Queue Single Timer");
	push_queue_timer_single.schedule(new PushQueueTask(), 0);
}

class PushQueueTask extends TimerTask {
	public void run() {
		Cursor oldest;
		rssdb.log("** Starting queue push of size "+rssdb.countPositionQueueRemaining());
		if ((oldest = rssdb.oldestUnpushedLocationQueue()).getCount() > 0) {
			int id = oldest.getInt(oldest.getColumnIndex("_id"));
			Location fix =  locationFromJson( oldest.getString(
			                    oldest.getColumnIndex(GeoRss.POSITION_QUEUE_JSON)));
			int status = pushLocation(fix);
			if (status == 200) {
				rssdb.log("queue push #"+id+" OK");
				rssdb.mark_as_pushed(id);
			} else {
				rssdb.log("queue push #"+id+" FAIL");
			}
			oldest.close();
		} 
		rssdb.log("** Finished queue push. size = "+rssdb.countPositionQueueRemaining());
	}
}
There is also a timer that fires every 30 seconds to push out any older queue entries.
private void startPushQueueTimer() {
	push_queue_timer = new Timer("PushQueue Timer");
	push_queue_timer.scheduleAtFixedRate(new PushQueueTask(), 0, 30000);
}
This needs to change in that I want a push at the rate of the update frequency, new fix or not. This will give a hearbeat to the server, and can be used to report environmental values which are always changing, starting with the battery level. PushQueueTask needs to be refactored a bit to do this:
  1. build basic json hash
  2. add environmental values
  3. if there is an entry in the queue, add those values too
  4. push

Food Cart Database 3

Posted by Don 15 Jun 2010 at 06:22PM

Portland has a strong food cart ecosystem. The highlights of which are the bigger food cart pods, @dieselboi's Food Carts Portland blog, and @spinnerin's Food Carts Map.

The diversity of the carts is amazing, and that includes the hours they are open. I, along with others in portland, have spent time figuring out how to crowdsource the open/closed status of each food cart. Note their sometimes-posted hours bear little resemblance to actual operating hours especially close to opening or closing time. One of my favorite carts, Yarp?!, which has since closed, had some of the least predictable hours of any cart. That inspired a system to track hours which has yet to be built. An SMS from the cart employees is one idea. Search twitter was another idea - it got partially implemented too. A physical box with an open/closed switch inside each cart is another one I'd like to try.

A mobile app that shows the nearest (and hopefully nearest *open*) food carts has been talked about and a web site concept has been implemented. There are opportunities for client software of all shapes and sizes to make use of a common database of food carts.

Coincidentally, the idea of open city datasets and civicapps.org was well underway. This inspired @maxogden to setup pdxapi.com and I talked to him about the awesomeness of couchdb. Now there is an exposed couchdb server at data.pdxapi.com with a database for food carts (http://data.pdxapi.com/food_carts). The flexibility and web-friendlyness of couchdb makes it easy for anyone to get started right away creating a new client to display, and hopefully update, data in the database.

The data has been pre-populated with the Food Cart Map. The fields that describe a food cart on the map are simple: a name, a longer description, and a latitude/longitude. There is room to store a lot more interesting data, too.

  • Photo(s)
  • Hours of operation
  • kind of food (mexican, thai, italian)
  • twitter account
  • start date of operation
  • final date of operation
  • entire menu with prices

The client I am currently interested in is an all-client-side javascript/html application that speaks to couchdb directly. I believe a generic 'couchapp' has enough bits to do this so it mostly needs a design and some coding.

Iterative

Posted by Don 14 Jun 2010 at 09:44PM

A small working thing can be iterated into a larger, more complex working thing. I try to do this with software. The key concept is that the thing 'works' in all stages and stages are small. If one were building a four-wheeled vehicle with a powerplant of some sort, the analogy might be that a wagon is built first. Its got a chassi, two axles, four wheels, a pivot mechanism at each wheel to support turning. To iterate that into a full-sized vehicle, the chassi would be enlarged and hooks would be there to place a frame on top to create the interior of the car. An engine would be added, probably something too small to use but enough to go around the block and test rolling and steering and the gear and ignition controls.

Somethings cannot be iterated from one to another. The change needed does not lend itself to small steps. Changing a carrot peeler into a squirrel, for instance.