Trusting Others And Technology Choices

Let's talk about trust. It matters in a team environment for sure. That much is a no-brainer. But trusting people also matters in solo projects. Here are three examples where my decision to trust others probably spared some pain.

Using Javascript Semicolons

Semicolons are ugly. One of the nice things about Python is that you don't have to worry about them. The trade off for significant white space may be strange but it works for me. I wish Javascript was more like Python in that regard.

One interesting feature of Javascript is semicolon insertion. The idea being that if you miss a semicolon at the end of a statement, that's okay. Javascript will automatically insert one anyways.

In the past, I skipped semicolons in my own projects. Then, one day I did some research. There is some debate out there on the subject. But ultimately, trust the smart folks that are widely respected in the Javascript world. Here is Doug Crockford's take on the issue.

"Each line should contain at most one statement. Put a ; semicolon at the end of every simple statement. Note that an assignment statement that is assigning a function literal or object literal is still an assignment statement and must end with a semicolon.

JavaScript allows any expression to be used as a statement. This can mask some errors, particularly in the presence of semicolon insertion. The only expressions that should be used as statements are assignments, invocations, and delete."

It's tempting to ignore Crockford. The way my code is formatted leaves me feeling that my source is invulnerable to masked errors that can happen when semicolons are left out. But do I really want to find out I'm wrong the hard way? Not really. It would be foolhardy to pretend Javascript will treat my lack of semicolons the one way Python does. So semicolons it is.

Nginx and uWSGI

Due to desire for control, my best choice was to make architecture choices for my own personal web stack. That meant picking out a web server and/or application server. AngularJS and Flask were the two known factors here. So what to choose? Gunicorn? uWSGI? Apache? Nginx? Something else? Even after much googling around, it was hard to say what the best choice was.

So I asked. From that Reddit conversation, my takeaway was that Nginx and uWSGI were going to be the best bet. 

Now, to be sure, it was tempting to throw out other questions. Why not Gunicorn? Isn't Apache more widely used? Doesn't uWSGI complicate things with too many configuration choices? 

But I didn't ask. It wasn't necessary. For one, googling for those answers is a lot easier if I really wanted to know more. For another, those sorts of questions could potentially start a flame war. Why stir the pot on people who are only trying to help you?

Flask-SQLAlchemy

 This was a real struggle for me. For the longest time, Flashmark used plain SQLAlchemy. I knew of Flask-SQLAlchemy but failed to see the point of it. The non-Flask version allowed a database layer that was indifferent to whether or not it was operating in a web application. It could be supporting a command line app for all it cared. Why ruin that with an extension that makes the model level scream "I depend on Flask!"

Then I came upon this in the SQLAlchemy documentation. 

"Some web frameworks include infrastructure to assist in the task of aligning the lifespan of a Session with that of a web request. This includes products such as Flask-SQLAlchemy, for usage in conjunction with the Flask web framework, and Zope-SQLAlchemy, typically used with the Pyramid framework. SQLAlchemy recommends that these products be used as available."

Even after reading this, it isn't 100% clear to me what flask-sqlalchemy does for me when it comes to reconciling sessions and web requests. Aren't sessions an ORM thing? Do SQLAlchemy Core users like myself need to care about sessions? Is there some sort of connection between database sessions and web sessions? It's all weird to me.

So it came down to this. Who is the smarter and more experienced person when it comes to using Flask with SQLAlchemy? Is it me? Maybe the writer of these docs knows her stuff when it comes to this. In the end, she was the one I trusted more on the issue.

To be sure, it wasn't easy to make the switch over to the Flask-SQLAlchemy extension. There was a lot of legacy code to switch over. It was not fun. But it did work out in the end. Everything is stable. No regrets there.

Humble Pie Is Yucky

It stinks to admit that one doesn't know enough to do one's job "right". Quite frankly, it's easier to not start at all. But such is the nature of software development. It's all about taking those two leaps of faith. It's a leap of faith in others. It's also a leap of faith in yourself. Be bold but also humble.

Fun With Python and Monkey Patching

Monkey patching is about replacing attributes of a Python thing with other attributes. Let's use the word "thing" very loosely and have some fun.

Messing With Self

First, the boring case where classes are classes and instances are instances.

Nothing new there. But what is with self? What's so special about it? Let's rebel. Everywhere self is shall be replaced with the word her.

It turns out self is not that special after all. It's just a convention for distinguishing the instance of a class from the class itself. Strictly speaking, self could be this or alice or bob or the gender pronoun of your choice. Politically correct Python for the win!

But seriously, did you ever stop to think about how strange it is when it comes to self and how calls method calls actually work?

ob1.say_stuff()

The ob1 in this case looks like it's the self that say_stuff refers to. Let's twist that call around a little.

Okay, so there's that. Now, be sure of this. The self term absolutely refers to an instance of the class it's being used in ... right? You have to wonder sometimes.

So much for that idea. At any rate, it's safe to say we've abused self enough for now. 

Time For Some Actual Monkey Patching

Here, we play a game about filling in the missing pieces. What if we had a starting class without an __init__.  To make it more interesting, let's NOT set up another class through which to provide instance variables. Something like this.

And who would have thunk. We can actually make this work in spite of the fact that the instance was born without a msg. Time for some setattr() surgery.

Oh heck, let's go gangbusters and just monkey patch a bare naked class together. Here's what that ends up looking like. Heck, we'll do up the instance too. Here it goes.


What else to play with? Ooo, I know!

Direct Assignments And Dictionaries

Python classes seem mutable enough. Maybe you can even directly assign to one method to take the place of the original. 

Awesome! That opens up quite a few possibilities right there.

Okay, one last thought. Python classes and instances have these things called dictionaries that dwell beneath the surface. Dictionaries represent methods and attributes.

Now, here's an idea. say_stuff isn't a member of the instance dictionary. It's a member of the class that the instance is based on. If there no say_stuff in the instance? No problem. Just look it up in the dictionary of the class instead.

Sooooo, what if we exploited the instance dictionary to subvert that expectation.

And yes, that setattr trick that's commented out there works too. And there are other things to say and do with these underlying dictionaries too. However, exploring much further requires  diving into the mysterious world of metaclasses. Let's not go there.... unless you really want to. ;-)

Well That Was Fun!

This is definitely not something you'd likely want to do with any REAL project unless you absolutely had to. Overuse of monkey patching can make things messy and confusing. At any rate, I hope you enjoyed this read. Have a great week.

Drag and Drop Files with HTML5 and Flask

Doing drag and drop file uploads has interesting uses. Uploading resumes, pictures, and so on are all scenarios where this is handy. So here's a look  at things I think about when it comes to drag and drop file uploads.

The Layout Of The Drop Zone

Since files need to go somewhere, we start by setting up a drop area for them to land in. This just requires a little HTML and CSS. The aim here is a simple box with a border. Code like this should do the trick.

The result looks like this.

And that's all on the look of it. As for DOING something with it, that's another story.

Getting Javascript/JQuery to Handle Drag and Drop

When dragging and dropping into that red square, the browser needs to know how to react. By default, it has its own ideas. That div element will tell you to buzz off if we try dragging stuff over it with the mouse. So we'll throw in a preventDefault invocation to say to the browser "Relax, I got this."

Then there are the drops which are their own kettle of fish. Often times, browsers seem to like popping open a new tab to display dropped content. We don't want that so another preventDefault heads that problem off at the pass.

Now, as for the file, we have to dig into the event to this thing called a dataTransfer. To me, dataTransfer plays the role of a handbag to store things in when taking dragged stuff from point A to point B. I don't know if that's the official definition but, for our purposes, it works.


For the moment, we just fish out the file that got stored there when we dragged it in. Then, we just display the name in the Chrome console. Hey look, a file!

And now I'm bored of Javascript. Let's switch to server side and talk Python. We'll deal with the rest of the client-side of things later.

Using Python And Flask To Give Files Somewhere To Go

Time to escape the browser and hang out in Pycharm and iPython for awhile. The mad plan is to knock out server code that takes in just any old file that wants to come live there.

The code does what it says. We grab the file out of the requests "files" dictionary. Then we put the file name through the Flask "secure_filename" to help prevent users messing with the system via sneaky file names. Save it to the folder and we're done.

Back to Javascript yet? Nah. We can get by fine using requests and ipython to test this code. Banging out something like this should do the trick.

In [11]: from requests import post
In [12]: url = "http://localhost:5000/sendfile"
In [13]: res = post(url, files={"file2upload":open("junkloop.sh", "rt")}
   ....: )
In [14]: res
Out[14]:
In [15]: res.text
Out[15]: 'successful_upload'

Okay, that was easy enough. Check the uploads folder .... it's there. Good. What's next?

Back To Javascript To Upload This Stuff

We can't avoid the client forever so let's get back on that. The thing to do next is to put together a request in a way that does a real upload. To do that, we have to take into account funny quirks as to how browsers like treating file uploads.

One thing to take into account here. Browsers seem to like to turning files into strings that conform to some content type or another. We don't want to mess with content types here. The file should be uploaded just as it is. So what do we do?

Fortunately, jQuery give you a couple of options such as processData and contentType to get around that issue. After making proper modifications, file dropping logic that looks like this.

Now, just double check that uploads folder and say "Hooray, drag and drop uploading works!"

Bonus Round - Listing Uploaded Files

We might as well list the files that are being uploaded. Triggering a fresh list of current files is easy enough. Just attach a request for a fresh list of files as a success response to the upload request. Might as well make the most of those promises, right?  Adding something to the end of dropHandler would be a good start.

        var promise = $.ajax(req);
        promise.then(fileUploadSuccess);

Beyond which you need to implement that function so it actually takes care of getting that fresh list. Easy enough to implement. Standard handlebars templating. Click here and here if curious.

Okay, so then what? That /filenames endpoint in the Flask code. We really should take care of that. But first, a quick tweak to send_file. A line like this after saving the file would be nice.

    # open and close to update the access time.
    with open(save_path, "r") as f:
        pass

The point of this is that I want to see the files in the order that they are uploaded. Doing a quick open/close changes the access time. Now we have files with a chronological upload history.  Coolness!

Oh yes, that /filenames thing. Here's that.


Well, that was kind of weird and hacky. Notice this lambda?

modify_time_sort = lambda f: os.stat("uploads/{}".format(f)).st_atime

So here's an example of sorting based on the access time that you get from the Linux stat command. The "st_atime" stands for "stat access time". That refers to the time the file was last accessed.

One other thing. We know that all those access times are in order of upload time because opening and closing files in /sendfile ensures that.

Let's Dump The Ugly Lambda

But my God this code is awful! That code would have been a lot clearer if I avoided the lambda. Something like this is nicer now that I think of it.


Yeah, that looks better. Avoid cancer in your Python code. Say no to lambda boys and girls.

And Now We Have An End Result

With all being said and done, you now have a setup for uploading files that lists the files you uploaded. It should look a little like this.

FINAL CAVEAT: If you want to go public with something like this, read stuff on how to manage uploaded files in a secure way. Or let this be a starting point for a silly project that you have no intention of putting into prod. Oh, and here is a link to the Github project this article is based on.  Have fun!

When Side Projects Become Brownfields

Brownfield projects are where you maintain old code that's been around awhile.  Greenfields are projects where most of what you work on is fresh and new and innocent.  That is what "brownfield" and "greenfield" means to me.

Spearking of brownfields, suppose there's a pile of work to be done on some legacy code.  The source is ugly.  The technologies it relies on aren't that great.  You need a temporary escape.

Wouldn't it be nice to do a side project where you can pick your own tools and build something cool?  Then the realization kicks in.  This IS the side project.  This is what such projects become given enough time.

It Always Starts Out Innocently Enough

Flashmark was and still is my baby.  And yet, things aren't as ideal as they once seemed when first starting.  It was a greenfield.  Bootstrap was beautiful.  AngularJS was Spring dependency injection done right.  It was to be a single page application with JSON data and a nice API done up in Flask.

Then Reality Sets In

Experience and time served up some reality checks.  Angular turned out to be overkill.  There was nothing there that couldn't have been done easier with JQuery and Handlebars.  But the front end investment is in too deep at this point.

Sometimes, it is okay to accept bad infrastructure decisions that a project uses.  So I do.  Buyer's remorse, by itself, is an insufficient excuse for ripping out and replacing huge chunks of a code base.

For example, Flashmark had a new tagging system added recently.  Unfortunately, it required writing more Angular code.  This was done to make the new code look and feel like the code that's already there.  Consistency is a good thing.

What Makes It All Worth It?

It turns out some choices stood the test of time.  Bootstrap is still good for interface building.  JSON is still a good data format to work with.  Flask and SQL Alchemy are still awesome tools for building a back end API.  Worth it!

As success goes, a worthwhile side project doesn't necessarily have to make money or even have a user base outside of oneself.  Flashmark does neither.  All it does is make learning easier for me.  And that's good.  You can sum up my attitude with the following function...


def is_project_successful(user_count):
    return user_count >= 1


Working on a side project is a little like raising and caring for a pet.  It isn't always joy and bliss.  Things can get downright ugly at times.  But for the right project, the warts are worth it.  And if you stick with it long enough, you can still come out with something cool.  Happiness is a brownfield.