Trusting Others And Technology Choices

Let's talk about trust. It matters in a team environment for sure. That much is a no-brainer. But trusting people also matters in solo projects. Here are three examples where my decision to trust others probably spared some pain.

Using Javascript Semicolons

Semicolons are ugly. One of the nice things about Python is that you don't have to worry about them. The trade off for significant white space may be strange but it works for me. I wish Javascript was more like Python in that regard.

One interesting feature of Javascript is semicolon insertion. The idea being that if you miss a semicolon at the end of a statement, that's okay. Javascript will automatically insert one anyways.

In the past, I skipped semicolons in my own projects. Then, one day I did some research. There is some debate out there on the subject. But ultimately, trust the smart folks that are widely respected in the Javascript world. Here is Doug Crockford's take on the issue.

"Each line should contain at most one statement. Put a ; semicolon at the end of every simple statement. Note that an assignment statement that is assigning a function literal or object literal is still an assignment statement and must end with a semicolon.

JavaScript allows any expression to be used as a statement. This can mask some errors, particularly in the presence of semicolon insertion. The only expressions that should be used as statements are assignments, invocations, and delete."

It's tempting to ignore Crockford. The way my code is formatted leaves me feeling that my source is invulnerable to masked errors that can happen when semicolons are left out. But do I really want to find out I'm wrong the hard way? Not really. It would be foolhardy to pretend Javascript will treat my lack of semicolons the one way Python does. So semicolons it is.

Nginx and uWSGI

Due to desire for control, my best choice was to make architecture choices for my own personal web stack. That meant picking out a web server and/or application server. AngularJS and Flask were the two known factors here. So what to choose? Gunicorn? uWSGI? Apache? Nginx? Something else? Even after much googling around, it was hard to say what the best choice was.

So I asked. From that Reddit conversation, my takeaway was that Nginx and uWSGI were going to be the best bet. 

Now, to be sure, it was tempting to throw out other questions. Why not Gunicorn? Isn't Apache more widely used? Doesn't uWSGI complicate things with too many configuration choices? 

But I didn't ask. It wasn't necessary. For one, googling for those answers is a lot easier if I really wanted to know more. For another, those sorts of questions could potentially start a flame war. Why stir the pot on people who are only trying to help you?

Flask-SQLAlchemy

 This was a real struggle for me. For the longest time, Flashmark used plain SQLAlchemy. I knew of Flask-SQLAlchemy but failed to see the point of it. The non-Flask version allowed a database layer that was indifferent to whether or not it was operating in a web application. It could be supporting a command line app for all it cared. Why ruin that with an extension that makes the model level scream "I depend on Flask!"

Then I came upon this in the SQLAlchemy documentation. 

"Some web frameworks include infrastructure to assist in the task of aligning the lifespan of a Session with that of a web request. This includes products such as Flask-SQLAlchemy, for usage in conjunction with the Flask web framework, and Zope-SQLAlchemy, typically used with the Pyramid framework. SQLAlchemy recommends that these products be used as available."

Even after reading this, it isn't 100% clear to me what flask-sqlalchemy does for me when it comes to reconciling sessions and web requests. Aren't sessions an ORM thing? Do SQLAlchemy Core users like myself need to care about sessions? Is there some sort of connection between database sessions and web sessions? It's all weird to me.

So it came down to this. Who is the smarter and more experienced person when it comes to using Flask with SQLAlchemy? Is it me? Maybe the writer of these docs knows her stuff when it comes to this. In the end, she was the one I trusted more on the issue.

To be sure, it wasn't easy to make the switch over to the Flask-SQLAlchemy extension. There was a lot of legacy code to switch over. It was not fun. But it did work out in the end. Everything is stable. No regrets there.

Humble Pie Is Yucky

It stinks to admit that one doesn't know enough to do one's job "right". Quite frankly, it's easier to not start at all. But such is the nature of software development. It's all about taking those two leaps of faith. It's a leap of faith in others. It's also a leap of faith in yourself. Be bold but also humble.

Fun With Python and Monkey Patching

Monkey patching is about replacing attributes of a Python thing with other attributes. Let's use the word "thing" very loosely and have some fun.

Messing With Self

First, the boring case where classes are classes and instances are instances.

Nothing new there. But what is with self? What's so special about it? Let's rebel. Everywhere self is shall be replaced with the word her.

It turns out self is not that special after all. It's just a convention for distinguishing the instance of a class from the class itself. Strictly speaking, self could be this or alice or bob or the gender pronoun of your choice. Politically correct Python for the win!

But seriously, did you ever stop to think about how strange it is when it comes to self and how calls method calls actually work?

ob1.say_stuff()

The ob1 in this case looks like it's the self that say_stuff refers to. Let's twist that call around a little.

Okay, so there's that. Now, be sure of this. The self term absolutely refers to an instance of the class it's being used in ... right? You have to wonder sometimes.

So much for that idea. At any rate, it's safe to say we've abused self enough for now. 

Time For Some Actual Monkey Patching

Here, we play a game about filling in the missing pieces. What if we had a starting class without an __init__.  To make it more interesting, let's NOT set up another class through which to provide instance variables. Something like this.

And who would have thunk. We can actually make this work in spite of the fact that the instance was born without a msg. Time for some setattr() surgery.

Oh heck, let's go gangbusters and just monkey patch a bare naked class together. Here's what that ends up looking like. Heck, we'll do up the instance too. Here it goes.


What else to play with? Ooo, I know!

Direct Assignments And Dictionaries

Python classes seem mutable enough. Maybe you can even directly assign to one method to take the place of the original. 

Awesome! That opens up quite a few possibilities right there.

Okay, one last thought. Python classes and instances have these things called dictionaries that dwell beneath the surface. Dictionaries represent methods and attributes.

Now, here's an idea. say_stuff isn't a member of the instance dictionary. It's a member of the class that the instance is based on. If there no say_stuff in the instance? No problem. Just look it up in the dictionary of the class instead.

Sooooo, what if we exploited the instance dictionary to subvert that expectation.

And yes, that setattr trick that's commented out there works too. And there are other things to say and do with these underlying dictionaries too. However, exploring much further requires  diving into the mysterious world of metaclasses. Let's not go there.... unless you really want to. ;-)

Well That Was Fun!

This is definitely not something you'd likely want to do with any REAL project unless you absolutely had to. Overuse of monkey patching can make things messy and confusing. At any rate, I hope you enjoyed this read. Have a great week.

Drag and Drop Files with HTML5 and Flask

Doing drag and drop file uploads has interesting uses. Uploading resumes, pictures, and so on are all scenarios where this is handy. So here's a look  at things I think about when it comes to drag and drop file uploads.

The Layout Of The Drop Zone

Since files need to go somewhere, we start by setting up a drop area for them to land in. This just requires a little HTML and CSS. The aim here is a simple box with a border. Code like this should do the trick.

The result looks like this.

And that's all on the look of it. As for DOING something with it, that's another story.

Getting Javascript/JQuery to Handle Drag and Drop

When dragging and dropping into that red square, the browser needs to know how to react. By default, it has its own ideas. That div element will tell you to buzz off if we try dragging stuff over it with the mouse. So we'll throw in a preventDefault invocation to say to the browser "Relax, I got this."

Then there are the drops which are their own kettle of fish. Often times, browsers seem to like popping open a new tab to display dropped content. We don't want that so another preventDefault heads that problem off at the pass.

Now, as for the file, we have to dig into the event to this thing called a dataTransfer. To me, dataTransfer plays the role of a handbag to store things in when taking dragged stuff from point A to point B. I don't know if that's the official definition but, for our purposes, it works.


For the moment, we just fish out the file that got stored there when we dragged it in. Then, we just display the name in the Chrome console. Hey look, a file!

And now I'm bored of Javascript. Let's switch to server side and talk Python. We'll deal with the rest of the client-side of things later.

Using Python And Flask To Give Files Somewhere To Go

Time to escape the browser and hang out in Pycharm and iPython for awhile. The mad plan is to knock out server code that takes in just any old file that wants to come live there.

The code does what it says. We grab the file out of the requests "files" dictionary. Then we put the file name through the Flask "secure_filename" to help prevent users messing with the system via sneaky file names. Save it to the folder and we're done.

Back to Javascript yet? Nah. We can get by fine using requests and ipython to test this code. Banging out something like this should do the trick.

In [11]: from requests import post
In [12]: url = "http://localhost:5000/sendfile"
In [13]: res = post(url, files={"file2upload":open("junkloop.sh", "rt")}
   ....: )
In [14]: res
Out[14]:
In [15]: res.text
Out[15]: 'successful_upload'

Okay, that was easy enough. Check the uploads folder .... it's there. Good. What's next?

Back To Javascript To Upload This Stuff

We can't avoid the client forever so let's get back on that. The thing to do next is to put together a request in a way that does a real upload. To do that, we have to take into account funny quirks as to how browsers like treating file uploads.

One thing to take into account here. Browsers seem to like to turning files into strings that conform to some content type or another. We don't want to mess with content types here. The file should be uploaded just as it is. So what do we do?

Fortunately, jQuery give you a couple of options such as processData and contentType to get around that issue. After making proper modifications, file dropping logic that looks like this.

Now, just double check that uploads folder and say "Hooray, drag and drop uploading works!"

Bonus Round - Listing Uploaded Files

We might as well list the files that are being uploaded. Triggering a fresh list of current files is easy enough. Just attach a request for a fresh list of files as a success response to the upload request. Might as well make the most of those promises, right?  Adding something to the end of dropHandler would be a good start.

        var promise = $.ajax(req);
        promise.then(fileUploadSuccess);

Beyond which you need to implement that function so it actually takes care of getting that fresh list. Easy enough to implement. Standard handlebars templating. Click here and here if curious.

Okay, so then what? That /filenames endpoint in the Flask code. We really should take care of that. But first, a quick tweak to send_file. A line like this after saving the file would be nice.

    # open and close to update the access time.
    with open(save_path, "r") as f:
        pass

The point of this is that I want to see the files in the order that they are uploaded. Doing a quick open/close changes the access time. Now we have files with a chronological upload history.  Coolness!

Oh yes, that /filenames thing. Here's that.


Well, that was kind of weird and hacky. Notice this lambda?

modify_time_sort = lambda f: os.stat("uploads/{}".format(f)).st_atime

So here's an example of sorting based on the access time that you get from the Linux stat command. The "st_atime" stands for "stat access time". That refers to the time the file was last accessed.

One other thing. We know that all those access times are in order of upload time because opening and closing files in /sendfile ensures that.

Let's Dump The Ugly Lambda

But my God this code is awful! That code would have been a lot clearer if I avoided the lambda. Something like this is nicer now that I think of it.


Yeah, that looks better. Avoid cancer in your Python code. Say no to lambda boys and girls.

And Now We Have An End Result

With all being said and done, you now have a setup for uploading files that lists the files you uploaded. It should look a little like this.

FINAL CAVEAT: If you want to go public with something like this, read stuff on how to manage uploaded files in a secure way. Or let this be a starting point for a silly project that you have no intention of putting into prod. Oh, and here is a link to the Github project this article is based on.  Have fun!

When Side Projects Become Brownfields

Brownfield projects are where you maintain old code that's been around awhile.  Greenfields are projects where most of what you work on is fresh and new and innocent.  That is what "brownfield" and "greenfield" means to me.

Spearking of brownfields, suppose there's a pile of work to be done on some legacy code.  The source is ugly.  The technologies it relies on aren't that great.  You need a temporary escape.

Wouldn't it be nice to do a side project where you can pick your own tools and build something cool?  Then the realization kicks in.  This IS the side project.  This is what such projects become given enough time.

It Always Starts Out Innocently Enough

Flashmark was and still is my baby.  And yet, things aren't as ideal as they once seemed when first starting.  It was a greenfield.  Bootstrap was beautiful.  AngularJS was Spring dependency injection done right.  It was to be a single page application with JSON data and a nice API done up in Flask.

Then Reality Sets In

Experience and time served up some reality checks.  Angular turned out to be overkill.  There was nothing there that couldn't have been done easier with JQuery and Handlebars.  But the front end investment is in too deep at this point.

Sometimes, it is okay to accept bad infrastructure decisions that a project uses.  So I do.  Buyer's remorse, by itself, is an insufficient excuse for ripping out and replacing huge chunks of a code base.

For example, Flashmark had a new tagging system added recently.  Unfortunately, it required writing more Angular code.  This was done to make the new code look and feel like the code that's already there.  Consistency is a good thing.

What Makes It All Worth It?

It turns out some choices stood the test of time.  Bootstrap is still good for interface building.  JSON is still a good data format to work with.  Flask and SQL Alchemy are still awesome tools for building a back end API.  Worth it!

As success goes, a worthwhile side project doesn't necessarily have to make money or even have a user base outside of oneself.  Flashmark does neither.  All it does is make learning easier for me.  And that's good.  You can sum up my attitude with the following function...


def is_project_successful(user_count):
    return user_count >= 1


Working on a side project is a little like raising and caring for a pet.  It isn't always joy and bliss.  Things can get downright ugly at times.  But for the right project, the warts are worth it.  And if you stick with it long enough, you can still come out with something cool.  Happiness is a brownfield.

JQuery Ideas That Help Me Memorize Really Big Numbers

The Idea

DomEditor is an app for editing a dominic number list.  Dominic numbers are a mnemonic device for memorizing really big numbers.  It works by tying every number from 00 through 99 with the image of people doing things.  Read more about the Dominic System here if interested.  My focus for this article is Javascript/JQuery techniques that helped with this project.

JQuery Event Delegation

Past projects had event handlers all over the place.  It was messy.  Here is an example from the RESTful notes project.

Behind every one of those "make button" functions is an event handler being created.  That's a lot of event handlers.  Not good.  Fortunately, JQuery has this thing called event delegation which you can read about here.  With this, you can use just one event handler at a parent level in the DOM.  There's no need to register separate handlers for every single button.  Here is how DomEditor does it.

The only catch to that is that DOM navigation is still awkward.  My solution was to do some pre-numbering on elements that needed it.  This setup gives me fuller use of Handlebars templates.  Worth it.

Promises, Closures, and Callbacks

Promises, closures, and callbacks matter because of all the async ajax happening here.  This creates a challenge.  The server-side API uses basic authentication for user-specific operations.  On the client, that means one call needed to get credentials and another call needed to use said credentials.  So how do you deal with that without the heavy nesting of callbacks inside of other callback?

Promises

Promises mean less needless callback nesting.  A call for credentials gets a promise for that info.  From there, you decide what to do with the data.  Here's how Dom Editor applies that idea....

Closure Callbacks

It gets weird trying to save info specific to one dominic number.  After all, the button callback only handles credentials.  The credentials have to be gotten from another call to begin with.  So how do you fit a dominic number argument into the picture?

The solution is to generate a callback.  Have it so the dominic number data to already there begin with.  Suddenly, argument passing is a non-issue.  That is straight up closure action.  Here's how that looks.

If this is all confusing, take heart.  Closures are hard to explain no matter what.  Really getting it requires tweaking existing ones and writing some of your own.  It should click after awhile.

Final Words

Get hands dirty with these ideas.  Write dumb junk pages that use delegate events if you have to.  Make promises.  Jump into that crazy closure world where functions return other functions.  It really is something special once you get these concepts.  You'll be a better programmer for it.  You'll probably also find better uses for these ideas than memorizing a hundred digits of PI.

Making an HTML5 Paint Program For Fun And .... Fun

Here's the project idea.  SVG Paint is an HTML5 art application.  User logs in.  User doodles with different brush types of shape, size, and color.  User can save and load their drawings.  Pretty straight forward.

Now for some thoughts and principles about what went into this project.

Keep The User Interface Simple

The user shouldn't have to think.  Everything in a layout should be straight forward, natural, and familiar to the user.  Here's my take on that.

Logging in.  You start with a login page.  You create a user account with a password.  If everything goes well, you can log in with the new account.  If you messed up the password or tried making an account that already exists, it'll tell you.

Here's what that looks like.

When you log in to the actual paint program itself, it's time to get creative.  After having some fun with it, you might get something like this.

Loading and saving are done through the file menu.  It will load drawings and save them based on the name entered in the "File Name" field.

And that's it.  It's simple, straight forward, and doesn't try to be clever.

Design Good Systems For Handling User Data

Data is everywhere.  It's on the front end.  It's in the database.  It's everywhere in between.  Funny enough, designing a system for getting data from point A to point B is just as much of an art as laying out Bootstrap elements and graphics.  Here's my take on dealing with data for this project.

The front end for making and logging in users is all Ajax and JQuery.  It communicates user id and passwords over plain HTTP using basic authentication.  The back end is done in Flask with an extension called Flask-Login. Passwords are stored and matched using a SHA1 hasher.  Verifying those encrypted passwords ends up looking like this.



Basic authentication on the client side also must be dealt with by the API on the server side.  That helps in managing user-specific data.  This setup was somewhat inspired by Github's API.  It's simplicity is reflected in the fact that the Requests library uses that API as an example in it's documentation.  Flask makes it easy to follow Github's example.  Take saving a drawing for instance.



Loading and saving the drawing data was interesting in and of itself.  The SVG is XML embedded within the DOMhierarchy. That XML gets converted to and from JSON.  SVG.js was used to create the svg data that make up the image on the canvas. On the server side, the data relationships were such that they played very nicely with the SQLAlchemy take on ORM mappings.

Be Critical Of Your Own Work

A critical eye matters, especially for one's own creative works.  Here are some issues about this project that make it less than perfect.

Security Problems
Security features are lacking in this project.  There are no rules for passwords.  Password encryption doesn't use salts.  My ignorance on which hasher algorithm fits the bill probably matters.   Understanding of Flask-Login is hampered by docs that partly depend on already knowing WTForms.  Lastly, usage of HTTP for the api is a less secure communication channel than HTTPS would be.  This project is not something that belongs on a public facing production server.

Data Communication Ineffeciency
It also probably isn't the most efficient data setup in the world.  It only takes a few minutes of doodling to generate tens of thousands of data records.  That means huge data payloads every time a load or save happens.  This strikes me as impractical.

Making Things Is Its Own Reward

All that being said, I had fun with this.  It isn't useful but it's not really trying to be.  Bored programmer has some stupid fun with a project and shares it with the world.  Worth it.  Everybody should do this.

Stepping Backwards From AngularJS to jQuery - An Experiment

Do Javascript frameworks like AngularJS matter that much when it comes to single page applications?  After more than a year of ups and downs with Angular, it was time for me to explore this question.  To do so, I wrote a little note application.  This was going to be done with just jQuery with Handlebars.js being added in later.  Here are some thoughts about that experience.

The Ajax Aspect

Ajax is important to much of what I do in my projects.  Headers and JSON get sent back and forth a lot.  Also, there are plenty of callbacks for dealing with stuff from the server.

So how do Angular and jQuery compare in that department?  To be honest, you don't lose much.  You go without Angular's $http service but you still have jQuery's ajax function to work with.  Here's what a json-in-json-out scenario looks like in AngularJS.



Here's the same thing but done with jQuery.


Overall, the only significant feature here is that $http has to be dependency injected.  I don't know that I gain anything from grabbing a service from out of whatever container it happens to comes from.  It feels vaguely like working with a Spring container in an enterprise Java project.  But that's about it for me.

Know Thy DOM

Giving up Angular did force me to think more about the DOM.  Vanilla jQuery doesn't insert itself as readily into HTML code as an Angular directive can.  That fact made direct DOM dealings more important.  For examples, here's how you generate bullet point list in Angular.

Doing the same thing in jQuery, you get something that looks like this.

While trivial in this example, string concatenation can gets messy as the HTML gets more involved.  However, you do get that upshot in jQuery of having debugger features more conveniently available.  Setting a breakpoint in straight Javascript is easy.  I have no idea how to do that with an ng-repeat loop.

Filling The Template Void with Handlebars.js

The good news is that living without Angular doesn't mean giving up on client-side templates entirely.  It turns out that Handlebars.js fills that void nicely.  Drop a template between script tags.  Load it up.  Insert the data.  All is well.


Yes, there is DOM dirty work to do but that's okay.  At least there isn't that error-prone process of piecing strings together so much.  That's actually pretty nice.

My Dynamic Button Trip Up

One area that tripped me up with Handlebars was button handling.  Buttons for deleting and changing notes are specific to each note.  The event handlers for those buttons need to reflect that.

Handlebars is really good at taking data text and plugging it into template text.  Where it gets weird is when events come into the picture.

The thing is this.  You generally don't attach event handlers directly to the text that Handlebars generates.  You attach them to some kind of DOM object like this...

$("#myButton").click(function(e){})

Okay, so what to do?

Here was my solution.  Let Handlebars do the whole template thing.  Have jQuery wrap the text up as a DOM element.  Then, just let jQuery attach event handlers to that.  Here is what that code ended up looking like.

So that gets the job done.  Once the DOM is created, we can attach a DOM button to that structure.  And that's it.  In case you're curious, you can find the full js file where this is set up over here.  And if I'm "doing it wrong", feel free to yell at me in the comments section.

Final Thoughts

Going without Angular was nowhere near as hard as I thought it would be.  The Ajax parts are reasonable thanks to the core jQuery libraries.  The DOM aspects I found myself needing to handle really weren't that burdensome.  And thanks to Handlebars, I didn't even have to give up templates.

What Angular is good for is catering to my enterprise and Java background.  I've done my share of mixing logic and HTML together from my experiences in JSP and ColdFusion.  The dependency injection that happens in Angular feels like a simplified version of what happens in a lot of Spring projects.

All being said, I don't think I'm going to disavow AngularJS entirely because I'm still not convinced that it's a "bad" framework.  For bigger web applications, it might well be ideal for all I know.  It's just that I now know that the set of problems Angular fits for isn't as big as I thought.  And that's an okay thing to come to terms with.  Here's to the programmer's journey!