JQuery Ideas That Help Me Memorize Really Big Numbers

The Idea

DomEditor is an app for editing a dominic number list.  Dominic numbers are a mnemonic device for memorizing really big numbers.  It works by tying every number from 00 through 99 with the image of people doing things.  Read more about the Dominic System here if interested.  My focus for this article is Javascript/JQuery techniques that helped with this project.

JQuery Event Delegation

Past projects had event handlers all over the place.  It was messy.  Here is an example from the RESTful notes project.

Behind every one of those "make button" functions is an event handler being created.  That's a lot of event handlers.  Not good.  Fortunately, JQuery has this thing called event delegation which you can read about here.  With this, you can use just one event handler at a parent level in the DOM.  There's no need to register separate handlers for every single button.  Here is how DomEditor does it.

The only catch to that is that DOM navigation is still awkward.  My solution was to do some pre-numbering on elements that needed it.  This setup gives me fuller use of Handlebars templates.  Worth it.

Promises, Closures, and Callbacks

Promises, closures, and callbacks matter because of all the async ajax happening here.  This creates a challenge.  The server-side API uses basic authentication for user-specific operations.  On the client, that means one call needed to get credentials and another call needed to use said credentials.  So how do you deal with that without the heavy nesting of callbacks inside of other callback?

Promises

Promises mean less needless callback nesting.  A call for credentials gets a promise for that info.  From there, you decide what to do with the data.  Here's how Dom Editor applies that idea....

Closure Callbacks

It gets weird trying to save info specific to one dominic number.  After all, the button callback only handles credentials.  The credentials have to be gotten from another call to begin with.  So how do you fit a dominic number argument into the picture?

The solution is to generate a callback.  Have it so the dominic number data to already there begin with.  Suddenly, argument passing is a non-issue.  That is straight up closure action.  Here's how that looks.

If this is all confusing, take heart.  Closures are hard to explain no matter what.  Really getting it requires tweaking existing ones and writing some of your own.  It should click after awhile.

Final Words

Get hands dirty with these ideas.  Write dumb junk pages that use delegate events if you have to.  Make promises.  Jump into that crazy closure world where functions return other functions.  It really is something special once you get these concepts.  You'll be a better programmer for it.  You'll probably also find better uses for these ideas than memorizing a hundred digits of PI.

Making an HTML5 Paint Program For Fun And .... Fun

Here's the project idea.  SVG Paint is an HTML5 art application.  User logs in.  User doodles with different brush types of shape, size, and color.  User can save and load their drawings.  Pretty straight forward.

Now for some thoughts and principles about what went into this project.

Keep The User Interface Simple

The user shouldn't have to think.  Everything in a layout should be straight forward, natural, and familiar to the user.  Here's my take on that.

Logging in.  You start with a login page.  You create a user account with a password.  If everything goes well, you can log in with the new account.  If you messed up the password or tried making an account that already exists, it'll tell you.

Here's what that looks like.

When you log in to the actual paint program itself, it's time to get creative.  After having some fun with it, you might get something like this.

Loading and saving are done through the file menu.  It will load drawings and save them based on the name entered in the "File Name" field.

And that's it.  It's simple, straight forward, and doesn't try to be clever.

Design Good Systems For Handling User Data

Data is everywhere.  It's on the front end.  It's in the database.  It's everywhere in between.  Funny enough, designing a system for getting data from point A to point B is just as much of an art as laying out Bootstrap elements and graphics.  Here's my take on dealing with data for this project.

The front end for making and logging in users is all Ajax and JQuery.  It communicates user id and passwords over plain HTTP using basic authentication.  The back end is done in Flask with an extension called Flask-Login. Passwords are stored and matched using a SHA1 hasher.  Verifying those encrypted passwords ends up looking like this.



Basic authentication on the client side also must be dealt with by the API on the server side.  That helps in managing user-specific data.  This setup was somewhat inspired by Github's API.  It's simplicity is reflected in the fact that the Requests library uses that API as an example in it's documentation.  Flask makes it easy to follow Github's example.  Take saving a drawing for instance.



Loading and saving the drawing data was interesting in and of itself.  The SVG is XML embedded within the DOMhierarchy. That XML gets converted to and from JSON.  SVG.js was used to create the svg data that make up the image on the canvas. On the server side, the data relationships were such that they played very nicely with the SQLAlchemy take on ORM mappings.

Be Critical Of Your Own Work

A critical eye matters, especially for one's own creative works.  Here are some issues about this project that make it less than perfect.

Security Problems
Security features are lacking in this project.  There are no rules for passwords.  Password encryption doesn't use salts.  My ignorance on which hasher algorithm fits the bill probably matters.   Understanding of Flask-Login is hampered by docs that partly depend on already knowing WTForms.  Lastly, usage of HTTP for the api is a less secure communication channel than HTTPS would be.  This project is not something that belongs on a public facing production server.

Data Communication Ineffeciency
It also probably isn't the most efficient data setup in the world.  It only takes a few minutes of doodling to generate tens of thousands of data records.  That means huge data payloads every time a load or save happens.  This strikes me as impractical.

Making Things Is Its Own Reward

All that being said, I had fun with this.  It isn't useful but it's not really trying to be.  Bored programmer has some stupid fun with a project and shares it with the world.  Worth it.  Everybody should do this.

Stepping Backwards From AngularJS to jQuery - An Experiment

Do Javascript frameworks like AngularJS matter that much when it comes to single page applications?  After more than a year of ups and downs with Angular, it was time for me to explore this question.  To do so, I wrote a little note application.  This was going to be done with just jQuery with Handlebars.js being added in later.  Here are some thoughts about that experience.

The Ajax Aspect

Ajax is important to much of what I do in my projects.  Headers and JSON get sent back and forth a lot.  Also, there are plenty of callbacks for dealing with stuff from the server.

So how do Angular and jQuery compare in that department?  To be honest, you don't lose much.  You go without Angular's $http service but you still have jQuery's ajax function to work with.  Here's what a json-in-json-out scenario looks like in AngularJS.



Here's the same thing but done with jQuery.


Overall, the only significant feature here is that $http has to be dependency injected.  I don't know that I gain anything from grabbing a service from out of whatever container it happens to comes from.  It feels vaguely like working with a Spring container in an enterprise Java project.  But that's about it for me.

Know Thy DOM

Giving up Angular did force me to think more about the DOM.  Vanilla jQuery doesn't insert itself as readily into HTML code as an Angular directive can.  That fact made direct DOM dealings more important.  For examples, here's how you generate bullet point list in Angular.

Doing the same thing in jQuery, you get something that looks like this.

While trivial in this example, string concatenation can gets messy as the HTML gets more involved.  However, you do get that upshot in jQuery of having debugger features more conveniently available.  Setting a breakpoint in straight Javascript is easy.  I have no idea how to do that with an ng-repeat loop.

Filling The Template Void with Handlebars.js

The good news is that living without Angular doesn't mean giving up on client-side templates entirely.  It turns out that Handlebars.js fills that void nicely.  Drop a template between script tags.  Load it up.  Insert the data.  All is well.


Yes, there is DOM dirty work to do but that's okay.  At least there isn't that error-prone process of piecing strings together so much.  That's actually pretty nice.

My Dynamic Button Trip Up

One area that tripped me up with Handlebars was button handling.  Buttons for deleting and changing notes are specific to each note.  The event handlers for those buttons need to reflect that.

Handlebars is really good at taking data text and plugging it into template text.  Where it gets weird is when events come into the picture.

The thing is this.  You generally don't attach event handlers directly to the text that Handlebars generates.  You attach them to some kind of DOM object like this...

$("#myButton").click(function(e){})

Okay, so what to do?

Here was my solution.  Let Handlebars do the whole template thing.  Have jQuery wrap the text up as a DOM element.  Then, just let jQuery attach event handlers to that.  Here is what that code ended up looking like.

So that gets the job done.  Once the DOM is created, we can attach a DOM button to that structure.  And that's it.  In case you're curious, you can find the full js file where this is set up over here.  And if I'm "doing it wrong", feel free to yell at me in the comments section.

Final Thoughts

Going without Angular was nowhere near as hard as I thought it would be.  The Ajax parts are reasonable thanks to the core jQuery libraries.  The DOM aspects I found myself needing to handle really weren't that burdensome.  And thanks to Handlebars, I didn't even have to give up templates.

What Angular is good for is catering to my enterprise and Java background.  I've done my share of mixing logic and HTML together from my experiences in JSP and ColdFusion.  The dependency injection that happens in Angular feels like a simplified version of what happens in a lot of Spring projects.

All being said, I don't think I'm going to disavow AngularJS entirely because I'm still not convinced that it's a "bad" framework.  For bigger web applications, it might well be ideal for all I know.  It's just that I now know that the set of problems Angular fits for isn't as big as I thought.  And that's an okay thing to come to terms with.  Here's to the programmer's journey!

Python and Javascript Jam Session - A Web Interface for Apt-Cache

Just fun fun.... How easy is it be to make a web interface over the apt suite of tools?  That's what AptPackageShow seeks to explore.  This post will cover aspects of this project that I personally found to be interesting.

Technically, this is not a FUMBLINA project.  The Debian packaging system is already a database of sorts.  No need to drag MySQL into it.  So scratch the M out of the acronym for this project.  No biggie.  There's still plenty of neat stuff to check out.

Building The API

So the first thing to fire up a Python editor to put together an api.  Starting with a model.py file means a short path from coding to seeing actual results.  Said results might be in a terminal shell but that still counts for something.  

For this model interface, we'll work with parts of apt-cache.  The "search", "show", and "stats" sub-commands are of interest here so we build functions to interface with them.  

A lot of the implementation details of the api is just Flask boilerplate and manipulating of string, dictionaries, and lists.  It's fun to do but, to be honest, it's pretty boring to talk about.  The most I can say on that matter is don't do things like this....

That's pretty bad.  If you must write code like this, make sure you refactor it ASAP.  Future You will thank you for it.

Working The Subprocess Module

The check_call function from the subprocess module is a key ingredient for this project.  Here's an example of this....

There are a few takeaways from this experience with invoking outside processes in this way.

1. If you have to make calls in a context where you'd normally need shell access, set the shell=True option.
2. Using absolute paths for commands simplifies things.
3. If you deploy to a test server, make sure the running Python process uses an account with shell permissions.

UPDATE:

Someone on Reddit pointed out the shell=True thing in point 1 isn't true.  Setting shell=False actually does turn out to work just fine.  You only need make it True if you run commands that are shell specific.

One last thought on this subtopic.  Running something like this in production feels wrong.  We are allowing shell access to a process that gets information directly from the user.  That strikes me as kind of a bad thing.  So in no way should this be considered an example of secure code.

Building Up An AngularJS Interface

Time to switch from Python to Javascript.  There's this idea that a component style to Angular code means fewer headaches with respect to where your data is available.  This isn't wrong but it isn't completely true either.  For example, component hierarchy for this project looks like this.....

And then there is the thing about having to wire together variables like this....

From there, you're supposed to pass info on like this....

So there's a lot of wiring to think about when you trade your scope hierarchies for component hierarchies.  Those can still be painful in the same way that ANY code hierarchy is painful.  That's why "search results" and "package details" are part of the same component.  It helps keep the hierarchy flatter.

Last Thoughts

What's the point of all this?  This was a fun diversion mainly done out of curiosity in the "What does this button do?" sense.  Having gotten practical experience with Angular components from this, I regret nothing.  I hope you enjoyed reading this little spiel about my project.  Again, go here if you want to see the project for itself and/or play with it.  Have fun!

Making Things Within The Quadrants of Want

"The way to get startup ideas is not to try to think of startup ideas. It's to look for problems, preferably problems you have yourself.  The very best startup ideas tend to have three things in common: they're something the founders themselves want, that they themselves can build, and that few others realize are worth doing. Microsoft, Apple, Yahoo, Google, and Facebook all began this way." - Paul Graham - How to Get Startup Ideas

My motivations for Flashmark were based on this essay.  It is a tool aimed at my learning issues.  I wanted this for myself.  However, recent attempts to market it drew little interest.  Nobody tried it.  Nobody commented.  I'm mostly okay with this!  But it did get me thinking of the whole idea of building software people want.

The Four Quadrants Of Want

There are four kinds of "want" when it comes to a software product.

Quadrants Of Want

Quadrant 1.) The developer wants it and the users want it.

This is ideal.  The developer is a user and other people use it too.  Mark Zuckerberg has a Facebook account as do over a billion other people.  There is also the case of the smaller developer who uses her own SaaS product but also has a couple hundred paid subscribers.  Whatever the scope of your user base is, this is the best quadrant to be in.

Quadrant 2.) The developer doesn't want it but the users do want it.

This isn't a bad deal either.  You have no dog in the fight when it comes to the product.  Still, users like it enough to pay which means you get paid too.  I've worked on software for archiving airplane part drawings and tracking phone tolls in office telephone systems.  I have no personal interest in planes or phones.  Still, the process of working on these things was enjoyable and co-workers were awesome.  These were good gigs.

Quadrant 3.) The developer wants it but the users do not want it.

This has happened for me twice with Flashmark being my most recent instance of this.  In the past, the Uncle Squirrely was a project I did that notified me of videogames that just dropped to "greatest hit" price levels.  In neither case did these projects get anything in the way of a user base.  But I didn't care.  I was scratching my own itch.  It didn't matter whether or not others had an itch in that same spot.

Quadrant 4.) The developer does not want it and the users do not want it either.

This is what Paul Graham wants people to avoid.  It happens when you don't want the product but you THINK other people will buy into it.  When they don't, the end result feels like a waste of time.  Pure demoralization.  Graham felt it first hand and would rather other startups not be crushed by that same defeat.

Moving Between Quadrants

When you start in quadrant 3, you are guaranteed a user.  It is you.  You are a worthy customer base to develop a quality product for.  Know it.  Believe it.  Even if the product never leaves this zone, you are at least spared the quadrant 4 nightmare scenario.

Quadrant 3 projects can serve as potential gateways to quadrants 1 and 2 if you allow it.  The key thing here is to be open.  Share what you are do as early and often as possible.  This allows for feedback for improvements.  It could lead to a new job.  It could make your project profitable.  But most importantly, feedback makes projects more fun.  You'd be out of your mind to not want to up the fun factor!

Practicing What I Preach

Don't do what I did with Flashmark.  There was very little communication with other people during that project's development.  Only two release announcements were ever made about it.  Both of them were made when the project was close to "done".  The end result is still a success.  However, I will admit it would have been nice if it were a different kind of success.

Instead, do something like this..... Last week, I started a just-for-fun project to put up a web interface to the apt-cache tool commonly found on Debian and Ubuntu systems.  I'm doing it because I'm curious.  How easy would it be to build a small AngularJS project using a component-based approach?  How would it work out if I put a Flask API on top of a command-line tool?  That's what I'm interested in finding out with the AptPackageShow project.  It's incomplete but you can go here if you want to check it out.  Comments welcome!

See the difference there?  With AptPackageShow, the conversation opens up almost right away instead of the "not talking about it much" approach taken with Flashmark.  I'm going to go out on a limb in saying that this is a better way.

Flashmark - What It Is And Why I Made It

Consider this an experiment or perhaps a tech demo if you will. The idea here is to see how other people react to a project built with the FUMBLINA tech stack.  It's not perfect.  Eventually though, one has to expose a work to the world and see what kind of reaction it gets.  So here goes nothing.

What Flashmark Is

Flashmark makes learning easier by mixing together the ideas of a flashcard and a bookmark.  First, you create a flashcard.  Second, you attach bookmarks to that flashcard.  That's a flashmark.

Here is what a list of flashmarks looks like.

Below is a question and answer for one of them.

This is a what an attached learning resource looks like.

Flashmark Q and A should be short as should the bookmarks also be.  The 140 character limit imposed on everything enforces that.

It's been tested on Chrome on a Linux desktop and Chrome for Android on a Samsung Galaxy phone.  It should work well on anything that plays well with AngularJS and Bootstrap.  If it fails on your platform, leave an issue here.

Personal Testimonial

I built this because I like teaching myself stuff.  So here is my use case for this project.

Every morning, I train myself using Anki flashcards.  If I do bad on a card, a note goes on the whiteboard and I move on to the next one.  This continues until my daily review allotment is done.

Afterwards is my "attack the weakness" phase.  This is where Flashmark comes in.  New flashmark questions and answers get made for whatever I put on the board.  I google up relevant links and attach them to the new cards.  Ideally, each card will get more than one bookmark if I can help it.

Mobile-friendliness is a sweet bonus.  Today, for example, a grocery line wait gave me a moment to review worker connection setup for Nginx configuration.  It's a great time optimizer as studying goes.

My Anki deck statistics for software development got a boost from this project.  Right now, my current "mastery" is at 98%.  This is for a collection of about 700 cards.  Not bad!

Some caveats to my results are due however.  For one, a 98% self assessment is no substitute for outside parties reviewing one's work.  Also, if I had to be completely honest with myself, some skill gain was likely the result of building Flashmark itself.  So I naturally have reason to be skeptical of my own success with this software.

Feel Free To Try It

Go right on ahead and try Flashmark for yourself.  Comments and criticism are welcome.  The reactions, good or bad, should hopefully be instructional.

Using Python To Play With Binary Files

Suppose you were to open up an image file in Python 3. Even without knowing much about image formats or binary files, what's possible here?  Let's see what we can find out about these files.  Let's also see what it says about learning strategies.

Some General Ideas

Reading binary files isn't much different than reading text.  One difference is that you open it with "rb" for "read binary" instead of "rt" for "read text".  If the file in question is small enough, you can grab the data with a one-liner like this....

raw_data = open(file_path, "rb").read()

One issue with managing bytes is that there is sometimes text in the mix of it worth considering.  Normal text operations often won't play well with bytes.  For that reason, we decode the bytes so that we have actual text strings to work with.

text_data = byte_data.decode("ascii", errors="ignore")

And yes, a lot of data in the source binary won't translate nicely to text.  Right now, we don't care about that stuff so we ignore it.  

What about going the other direction?  Sometimes, a text string needs to be a pile of bytes.  So we want to encode this.

byte_data = text_data.encode("ascii")

Do you need to do encoding and decoding everywhere you go?  Not necessarily.  It's just a generally good thing to understand the idea of working in both directions. Now, with those ideas established, let's poke around with some image files.

Identifying Image File Formats (sort of)

Here's a thought.  You can dump bytes from an image file straight to the console.  A good Linux shell environment is handy like that. Heck, you could even page through the mess if you felt so inclined.  For example, you can do this.....

cat pinkie1.png | less

From that, you get something that looks like this....

While most of the data isn't readable, there are bits of text there.  Also to note is that PNG, GIF, and JFIF (aka JPEG) appear to have something in common.  The first instance of text that shows up seems to consistently identify the image format.  So that's useful.

Finding text looks like a job for regex.  But wait a minute!  This is bytes.  Regex is meant to be used with text.  No problem.  We'll just decode the bytes we can and toss the rest.  The result is a function that looks like this.

So if you put in image_format("pinkie1.png"), you get PNG as the result.  Pretty nifty.

But also pretty useless.  This "pick-the-first-ascii-text-block-you-find" approach has only been seem to work with samples of PNG, GIF, and JPEG files.  It bombs on TIFF, Windows BMP, and probably several other image formats too.  Oh well.

Stirring Up Curiosity Is Its Own Reward

Making a turd like this one did get me thinking.  How would the pros do it?  There is this function in the official Python 3 libraries called what from the imghdr module.  How, exactly, did they do it?  Oh wait!  There is source code out there.  Here's an unofficial Github resource right here.  This implementation is good reading and beautiful in its simplicity.

This is a good learning strategy for a lot of things.  You poke around with no particular goal.  You stumble upon something.  You dive in.  You screw it up.  You step back. You see how the experts handle it.  Lather.  Rinse.  Repeat.  It's both educational and a lot of fun.  So go do that!  

Speedy Deployments And The Meaning Of Devops

Okay, so here's the project on my mind.  Create and provision a local Fumblina server.  Deploy a simple web app to it.  Time the setup process.  Why?  Because Devops!

The key tools in use are as such...

1. 64-bit Linux Mint
2. Virtual Box 4.3 
3. Ansible 1.5
4. Vagrant 1.7
5. Git 1.9

Technically, this whole arrangement works just as well with a standard Ubuntu desktop distro with Virtualbox 5.  But whatever!  We hit the stopwatch start button and off we go.

The Server Setup

We need a local server.  There's a project called "basemachine" on my Github that would help.  So that gets git cloned over and started up like this...

cote@mymachine ~/PycharmProjects $ git clone https://github.com/pcote/basemachine.git

Cloning into 'basemachine'...

remote: Counting objects: 14, done.

remote: Compressing objects: 100% (3/3), done.

remote: Total 14 (delta 0), reused 0 (delta 0), pack-reused 11

Unpacking objects: 100% (14/14), done.

Checking connectivity... done.

cote@mymachine ~/PycharmProjects $ cd basemachine/

cote@mymachine ~/PycharmProjects/basemachine $ vagrant up

Conveniently, there is a Vagrant box image installed locally so that saves us some time.  But still, it is going to take a few minutes to make and provision this server.  On to other things while this job bakes.

A Skeleton Web Application

While we're busy provisioning, there's a generic web application skeleton in Github to get.  We're also going to need the contents copied into an actual project folder called "DomEditor".  Here it goes!

cote@mymachine ~/PycharmProjects $ git clone https://github.com/pcote/appskeleton.git
Cloning into 'appskeleton'...
remote: Counting objects: 38, done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 38 (delta 6), reused 36 (delta 4), pack-reused 0
Unpacking objects: 100% (38/38), done.
Checking connectivity... done.
cote@mymachine ~/PycharmProjects $ cd appskeleton/
cote@mymachine ~/PycharmProjects/appskeleton $ cp -R * ../DomEditor/
cote@mymachine ~/PycharmProjects/appskeleton $ cd ../DomEditor/
cote@mymachine ~/PycharmProjects/DomEditor $

We switch over to that and get app-specific things setup.  There's a folder in the Ansible role that needs renaming by the way.  It needs to match the name we gave this app.

cotejrp@mymachine ~/PycharmProjects/DomEditor/roles $ mv myapp domeditor

Very good.  So now, we deal with configuration with the deployvarsdev.json file.  

{

    "hostname": "all",

    "domain":"domeditor.com",

    "appname":"domeditor",

    "mysql_root_password":"rootPassword",

    "proxyport":3032

}

Careful for that mysql_root_password by the way.  That variable has to be the same here as it is in basemachine's Vagrantfile.  Yes, the setup is stupid and the root password should be established in one place rather than two.  For the time being, we just accept that the value must match lest the Ansible playbook barf on you later.

Oh yeah.... The /etc/hosts file needs a domain name alias to localhost.  That's going to have to match the domain variable set in our json file.  So this line goes into hosts....

127.0.0.1       domeditor.com

Time to check on the base machine provisioning.  Ding!  It's done.  Time to move forward with application deployment.

Application Deployment

Deployment needs one more thing.  Time to open up provisiondev.sh for a little editing...  

#!/bin/sh

basemachine=/path/to/base/machine

inventory=$basemachine/.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory

privatekey=$basemachine/.vagrant/machines/default/virtualbox/private_key

ansible-playbook -u vagrant -i $inventory --private-key=$privatekey --extra-vars="@deployvarsdev.json" playbook.yml

See that basemachine variable near the top?  We replace that with the path to wherever we git cloned the basemachine folder to in the first place.  Something like this does the trick nicely.

basemachine=$HOME/PycharmProjects/basemachine/

And there's that.  Time to run this deployment script....

cote@mymachine ~/PycharmProjects/DomEditor $ ./provisiondev.sh 

A couple seconds pass.  Script completes.  Open up this url in a browser...

http://domeditor.com:8080/

And once we see the message "This is an angularjs stub page." in the browser, we know we're done.

Total Time Elapsed: 4 minutes 39 seconds.

Could it be faster?  Maybe.  One of these days, I'll have to figure that one out.

Digression: What Does Devops Mean Anyways?

A conversation with a friend raised the question of exactly how one defines the term "devops".  It depends on who you ask.  I can think of it in terms of management reality, my reality, and wishful thinking.

For a manager, it's about improvement in team dynamics.  Developers are paid to change I.T. systems.  Operations people are paid to keep everything stable.  These job definitions naturally put these two camps at odds.  The idea of devops in this context is to get developers and ops people on the same page to reduce the conflict.

For a solo act like myself, devops is about self-reliance.  There are no operations people in my world.  Naturally, it is my responsibility to set up, modify, and maintain those servers myself.  Devops tools like Ansible and Vagrant just happen to make that business easier for me.

Lastly, there is the wishful thinking vision of devops.  It's that dream of sitting down in a quiet environment to work.  You check out the project from the local Git server.  You do a "vagrant up".  Bang!  There's a virtual version of the same server as what ops is managing in production.  That's what you get to develop against.  It sure would make change change control discussions easier.  One can dream.

Making Your Own Google Tag In AngularJS

Playtime blog post!  So I made a custom tag on a lark the other day.  Here it is.



It is what it looks like.  It makes a link to Google with search results for Chef Ramsay.  Let's make this tag.

Taking Advantage Of Directives In Angular

Directives can do a few interesting things.  For this project, the goal is to make a new HTML tag.  A skeleton for this directive will get us started.




We really just want this directive used as a tag element and nothing else.  Setting the restriction to "E" will impose that limit.

jQuery level manipulation sounds nice given what we're going for.  The link method that Angular allows for is good for that.  Let's stub that one out a little.




What to do next?  That element argument is useful.  It's a jQuery object of sorts that seems to be referring to our custom tag.  What can we do with a jQuery object?  Messing with the information between the two "google" tags could be nice.




Pretty straight forward.  Grab the text.  Cut together a string representing a hyperlink.  Plug the resulting link back in.  All done.

In Summing Up

This is a just me fooling around here.  If custom tags are the sort of thing that you want to take a more serious look into, there are better tutorials out there.  Jakob Jenkov is one of my favorite resources for Angular JS and I recommend checking him out.  Just make sure to read his critique section first.  There's wisdom in what he's saying there.

And that's all for now.  Cheers.

Keep Your Database Permissions Small

Update: March 15, 2016
There were several interesting bits of legitimate criticism on this post on Reddit.  The idea of the Principle of Least Privilege that this post communicates is right.  Some of the implementation details in the examples fell a little short though.  A thousand thanks to all who commented.  I learned a lot from this exchange.  The original post below........
 

A typical web application needs an account to access a database in order to do it's job.  These applications often face the outside world.  As such, direct access to your database is only a few degrees of separation from your users.  How does one secure this?  Start with minimalism.

Make A Minimal Database Account

Start by setting up a database account.  When doing this, err to too few permissions rather than too many.  It is inconvenient in the short term.  However, over the life of your project, you'll be glad you did.  Things will be more secure and easier to maintain.

An example.  There is a feature in one of my projects where the user can delete learning resources.  Here is a piece of Ansible configuration that sets up the MySQL database account.



Notice that the account does not offer any delete permissions at all.  So how is deletion possible?

Coding Within Constraints

The trick is this.  When the user "deletes" a learning resource, they don't directly delete anything at all.  Instead, they put in a request to delete it.  This much only requires insert privileges.  Here is an example using Python and SQLAlchemy.



As far as the user is concerned, the resource is "deleted" right away.  This is thanks to a subquery used to help with the lookup.

Doing Real Deletions Safely

To be sure, that can take up a good bit of space over time.  A real deletion of data is ultimately called for.  Here's a snippet from a script that does that job.



Not that this script does use a database account with delete permissions.  It is also a task run through a cron scheduler on an Ubuntu Linux server.  This is important because it is not invoked by the main web application at all.  The system is more secure because actual deletion is treated as an internal detail that exists outside of public access.  Below is one way to set up a cron scheduled job to run every other hour.



The takeaway is this.  A good security mindset can lead to solutions that provide features to user while also keeping data safer.  So the next time there comes a situation where you think you need more permissions, stop and reconsider.  You might be able to get by with less than you think.

The Fumblina Full Stack Philosophy

The Fumblina full stack philosophy is a mix of tools and ideals that I swear by in my web application projects.  The core ideals are that you should pick your own tools, make stuff for yourself, be small, and have a sense of humor.  Let's expand on that.

Pick Good Tools For Yourself

Given the choice, pick tools that work for you.  Ask yourself, "Will this make it easier for me to make good software"?  What is your situation?  What are you building and why?  For point of reference, here is my thought process for picking tools.

The Front End

HTML and Javascript are givens for web applications.  Since I'm not a web designer, Bootstrap makes it easier to put together interfaces that work on a 22 inch monitor and Android phone.  

Usually, my project is some kind of single page application.  At the time the decision was made, AngularJS looked right for this.  That might change some time in the future but, for now, it's the Javascript resource I know best.

Middle Pieces

Choices of middleware were informed by my Python 3 biases.  Flask and SQLAlchemy are the two major pieces as libraries go.  Flask makes creating JSON-based web APIs easy.  SQLAlchemy let's me use a database toolkit without being forced to use an ORM.

As application and web servers go, googling around and asking questions on Reddit helped me decide.  That's how uWSGI and Nginx ended up in the picture. uWSGI handles my Flask stuff.  Nginx plays proxy to Flask and static files.

Back End Parts

MySQL on an Ubuntu server is something I know so I stuck with it.  The rest of the pieces of my stack fit naturally in that context.  It's easy to find help online if stuck.  These were easy choices.

Working a custom tech stack generally requires a custom server setup.  Virtualization and cloud environments make sense here.  On the development side, Vagrant and Virtualbox do the job easily.  For production, Digital Ocean is a cheap no-frills service for setting up a Linux server.

For deployment, my pick is Ansible.  Knowledge of YAML plus native provisioning support in Vagrant swayed me on this one.  Ansible uses an idea called playbooks.  Playbooks used to provision a virtual machine can also work on a production server with little to no changes.

And that's how my tools were chosen.  I went with what made the most sense and asked for input when in doubt.

The Acronym

By the way, if you do pick the same tools that I do, you can make a nice acronym out of it!

FUMBLINA = Flask uWSGI MySQL Bootstrap Linux Nginx AngularJS

I made that up myself.  Kind of proud of it!

The Small and Silly Fumblina Philosophy

Here's the thing.  A full stack developer is not an unstoppable force ready to tackle any large project thrown her way.  She is not the developer equivalent of this guy.

Mr. "Full Stack" I presume?

Below is a better way of thinking about what "full stack" means.

Being A Part Of Your User Base

The best kind of product to be working on is one where you are one of the users.  For example, I am the sole user of Learning Machine.  A big user base isn't the goal.  I want to make things easier for myself when it comes to learning new things.  There are tools to build that help me in that regard.   If a user base ever did happen, great!  Having a head start as "User 0" puts me in a better position of understanding their needs.

Keeping the Feature Set Small

My favorite explanation of the "Minimal Viable Product" concept comes from Extra Credits, a Youtube show about game design.  The video emphasizes that a new game should start out with nothing but the absolute core mechanics and nothing more.  This is what you want to get right before adding all the supporting attributes.

In a recent round of commits on my Learning Machine project, everything was stripped out except for three things.  

1. Making practice exercises.
2. Quizzing and rating yourself on existing exercises.
3. Reviewing a report to see where you're doing well and where you aren't.

Focusing on the core essence of a project is crucial because, as a full stack developer, there are limits to what one can do.  A full stack project for something small is challenging.  One with a huge feature set is going to spread you incredibly thin.  That's not a good thing.

Not Taking Yourself Too Seriously

Fumblina is a funny-sounding name.  It's the kind of name you give to a fairy with a lazy eye who can't quite fly straight.  When calling myself a Fumblina developer, I'm aware that I'm making fun of myself.  Laughing at oneself is a good thing and more people should do that.

Any "serious" full stack developer will have her share of stressful moments.  A sense of humor relieves stress.  Less stress leads to better judgment calls.  Better judgment calls are more likely to lead to the ultimate success of a project.  

And that's the whole Fumblina philosophy.  Or at least, that's how I like to think of it.

Processing Lots of Text Files with Python

Tasks involving work with text and files containing text is very common.  Python makes dealing with single files a breeze.  But what if you need to deal with LOTS of text files?  That can be a bit trickier.  Fortunately, Python is very much up to that challenge as well.

My Setup For This

The example discussed here runs on Linux Mint 17.2 64-bit.  Since we're dealing with Blender here, you can download the 64 bit version for Linux here.  The code being shown here is confirmed to run on 64 bit Python 3.4 for Linux.

The Goal In This Example

There's a piece of art software called Blender that interests me.  In Blender programming, there's an idea called "region types".  Region types help position interfaces.  My interest in is in lines that look similar to this.

bl_region_type = "WINDOW"

What region types are commonly used within the Python scripts that come packaged with Blender?  That is the question to answer here.  Answering it means scanning for info within the contents of multiple files.  There are four general steps to take in this sort of situation.

1. Decide on what files are of interest.
2. Decide what lines of those files are of interest.
3. Get whatever it is you need from those lines.
4. Act on that extracted info.

Let's break this down.

Picking Your Files

For this example, we are interested in files ending in .py within a specific set of directories.  

The walk function from the os module is very handy here.  It makes iterating through nested directories very simple.  It is also a generator.

There are two choices for dealing with generator based info.  You could use the generator to return a list of python file paths on one hand.  On the other, you could yield each of the created paths one at a time.  We're going with this second choice.  There's no reason that we need a full list of python file paths in memory all at once.  Having pyfiles as a generator is fine.

Picking The Lines

Let's assume that the line of code in question won't be broken up across multiple lines.  We'll also assume that the expression in question won't have comments on the same line.  That allows us to write naive code like what's below.

Some things to note here.  First is the usage of the with statement.  Using with in this way allows the file object's context manager to help us along.  We are dealing with dozens of files here.  Letting that context manager help close those files for us make sense.  

The second thing to note is that pick_lines is a generator too.  We don't need a full collection of file lines in memory so why have it?  

The third thing is the set up of the for loop. It is double-nested. It handles both file name collecting and file reading duties.  And yet, it is still easy to read and doesn't burden resources.  It's lazy.  It's beautiful.  It's also practical for what's being done here.

Getting What's Needed From The Lines

The goal is to get specific information from this kind of line.

bl_region_type = "WINDOW"

We want the WINDOW part and nothing else.  This only requires a few lines of straight line imperative code.  Adding more functions or generators isn't called for here.  Let's expand on our loop.

This is self-explanatory.  The important stuff comes after the equal sign.  The stuff before it doesn't matter.  Quotes and extra spaces don't count either.  All that gets tossed before throwing the desired info into a list.  And since this list is our final target, having it concretely in memory is the right thing to do.

Act On The Information

This can be whatever the situation calls for.  That could be writing info to a database or another file or whatever else you need.  Here's what I did.

This is pretty much what it looks like.  My main interest was seeing what region types were actually being used in Blender addon scripts.  Having that info and count tallies output to the console is enough.

And Finally The Completed Script

And here are the results.

WINDOW: 90
TOOLS: 56
UI: 78

A testament to the power of Python is what can be accomplished in barely a couple dozen lines of code.  Those few lines can parse through thousands of lines of text spread across dozens of files from various directories.  What you can do beyond that is up to your imagination.

Code Scribbling with Fizz Buzz

There's this thing that I like to do called code scribbling.  Code scribbling is about letting loose.  You noodle around with a piece of code for no particular reason.  Fun is the only goal here.  But it's an important goal.  Fun helps loosen you up for the important programming work you do.  The code scribbling session below is one such example of me just enjoying myself with a little Python 3.4.

A Code Scribbling Session

My mad plan here is to play a little Katamari Damacy with the Fizz Buzz problem.  I want to start simple and just add thing after thing until it turns into a big ball of chaos.  We'll start with a basic loop and modulus operators.  Here is that.

Let's shorten this up with some ternary operators.

Okay, that was concise.  Time to make it more interesting.  What if the two ternary operators were concatenated together and thrown into a list comprehension?  The fizz and buzz ternary expressions have to be wrapped in their own sets of respective parenthesis.  This ensures that it evaluates correctly.  The "==" aspect of the modulus expression can be reduced to something more "truthy" so that it fits nicer in a blog post.  Here's what that looks like.

Interesting if a bit of a space eater.  A generator expression would make it lazier.  At the very least, it won't make the code look any worse than before.  Here's that code.

What else could be done with this?  Here's an idea.  Those ternary expressions could be wrapped into a lambda function.  Here's the result of that.

What else?  We could take the string formatting out of the print statement and put that in the expression.  Then, the for loop just has to loop through the resulting list of strings. 

Hmm.  Come to think, why have the for loop at all?  The string join method could just join them all together with newline characters to ensure it prints like before.  Then, the print statement just has to print one big string all at once.

Hooray, no more normal for loop!  But this is interesting.  That entire expression we have could be made into it's own lambda expression in and of itself.  You could even pass an argument and print the result of that.  Here's what that looks like.

Okay, what's next?  There's no reason why we can't just move the entire expression and evaluation into the print statement itself.

It's all one big blob now.  The print statement uses the expression here.  But wait!  There's no reason the print couldn't be a PART of the expression.

Where do we take it from here?  Let's see.  Meta-programming perhaps?  There's nothing saying we can't create a nameless class with a __call__ method.  Then it's just a matter of making an instance.  From there, it's just a matter of taking advantage of __call__ and invoking the instance as if it was just another function. 

This cake is baked.  Let's decorate!  I'll just wrap the __call__ function in a staticmethod decorator and make that troublesome self argument go away.

Awesome!  All wrapped and decorated.  And we're done with this scribbling session.

What a Mess!

This is obviously not meant for production code.  This is essentially unfettered silliness enjoying some time in the sun.  Such is what creativity is.  It might accidentally lead to actual innovation but that's besides the point.  This is about taking some time to set your brain free.

So make it a priority this year to relax.  Have fun.  Scribble.