Okay, here is my first report on the results of my hundred minute hack experiment. The good news is the database has video game data from Amazon and I was able to stuff those results into an Excel file. Getting data into a web page is next. The pressure is on to find work in case I can't turn this idea profitable somehow. There's good news even in that regard since I now have good samples I can use to market with to prospective employers. Now for the details...
A third of the time spent on this project was getting that first information request to Amazon. That involved grappling with encryption, url parameter manipulation, unicode issues, and other stuff that Amazon Web Services requires in order for you to get an information request through to them. Eventually, I shifted strategies and turned to APIs for Amazon and MySQL. After some gotchas with versions, platforms, dependencies, and so forth, the first request from Amazon to Python to MySQL made it through.
As data was concerned, a "cheap game" was defined as a physical PS3 game costing anywhere between a penny to 49.99. Getting that data presented difficulties. Just asking for game data causes Amazon to spit back a whole lot of Playstation console bundles, controllers, and so on along with game titles.
Building a Playstation hardware dataset helped even though it meant two more rounds of web service requests. One round was needed for the hardware and accessories while a separate one was needed for game controllers. A MySQL trigger cleared unwanted hardware from the game dataset. The trigger approach made sense because it removed bad data from the game list once it was discovered rather than burden the system needlessly with subqueries and table joins at every single request.
So there you have it. Several hundred web service requests done in 3 parts leading up to a collection of cheap games. All told, that amounted to somewhere between 500 and 750 web service requests which translates to 5000 to 7500 records processed. All this done to amass a collection of 341 Playstation 3 titles.
I have quite a bit to do on this project and I look forward to seeing where all this leads me. I'll share as things progress.
Monday, July 26, 2010
Thursday, July 8, 2010
First Steps
When my contract ended recently, I decided to try my hand at self-employment and build something profitable. I don't know what I'm going to build let alone how I'm going to make money on it but I figured “Why not?”
So without a product or a clue how to run a business, I figured I'd start with an interesting idea and a productivity technique to drive me forward.
The interesting idea is a database of Playstation 3 video games that keeps track of prices. The whole idea is that it would a nice list of all titles, prices, and when those prices change. Beyond that, I don't know.
The productivity technique I came up with is called the “Hundred Minute Hack”. The basic idea is I throw together a todo list of four or five things I'd like to do in 100 minutes. Then I set a timer and I go at it and I don't let myself stop for anything short of coffee or a bathroom break.
The four or five related todo items is a pretty nice trick. If I get sick of doing one thing, I can just do something else on the list. It's a handy way to keep myself from being bored, frustrated, or intimidated by the project.
I'm committing to doing these 100 minute runs two to three times a day. Some flexibility is important and sometimes I have to settle for 50 minute runs. But that's what's pretty sweet about the whole system. On a bad day, I still find I've spent over two hours hacking away on my project. On a good day, I get up to five.
The entries that follow this will probably be a mix of progress reports and little tricks I'm picking up along the way. It's hard to say where this experiment will lead. In the best case scenario, I have a profitable business. In the worst case, it'll be a fun topic to bring up during job interviews.
So without a product or a clue how to run a business, I figured I'd start with an interesting idea and a productivity technique to drive me forward.
The interesting idea is a database of Playstation 3 video games that keeps track of prices. The whole idea is that it would a nice list of all titles, prices, and when those prices change. Beyond that, I don't know.
The productivity technique I came up with is called the “Hundred Minute Hack”. The basic idea is I throw together a todo list of four or five things I'd like to do in 100 minutes. Then I set a timer and I go at it and I don't let myself stop for anything short of coffee or a bathroom break.
The four or five related todo items is a pretty nice trick. If I get sick of doing one thing, I can just do something else on the list. It's a handy way to keep myself from being bored, frustrated, or intimidated by the project.
I'm committing to doing these 100 minute runs two to three times a day. Some flexibility is important and sometimes I have to settle for 50 minute runs. But that's what's pretty sweet about the whole system. On a bad day, I still find I've spent over two hours hacking away on my project. On a good day, I get up to five.
The entries that follow this will probably be a mix of progress reports and little tricks I'm picking up along the way. It's hard to say where this experiment will lead. In the best case scenario, I have a profitable business. In the worst case, it'll be a fun topic to bring up during job interviews.
Subscribe to:
Posts (Atom)