Speedy Deployments And The Meaning Of Devops

Okay, so here's the project on my mind.  Create and provision a local Fumblina server.  Deploy a simple web app to it.  Time the setup process.  Why?  Because Devops!

The key tools in use are as such...

1. 64-bit Linux Mint
2. Virtual Box 4.3 
3. Ansible 1.5
4. Vagrant 1.7
5. Git 1.9

Technically, this whole arrangement works just as well with a standard Ubuntu desktop distro with Virtualbox 5.  But whatever!  We hit the stopwatch start button and off we go.

The Server Setup

We need a local server.  There's a project called "basemachine" on my Github that would help.  So that gets git cloned over and started up like this...

cote@mymachine ~/PycharmProjects $ git clone https://github.com/pcote/basemachine.git

Cloning into 'basemachine'...

remote: Counting objects: 14, done.

remote: Compressing objects: 100% (3/3), done.

remote: Total 14 (delta 0), reused 0 (delta 0), pack-reused 11

Unpacking objects: 100% (14/14), done.

Checking connectivity... done.

cote@mymachine ~/PycharmProjects $ cd basemachine/

cote@mymachine ~/PycharmProjects/basemachine $ vagrant up

Conveniently, there is a Vagrant box image installed locally so that saves us some time.  But still, it is going to take a few minutes to make and provision this server.  On to other things while this job bakes.

A Skeleton Web Application

While we're busy provisioning, there's a generic web application skeleton in Github to get.  We're also going to need the contents copied into an actual project folder called "DomEditor".  Here it goes!

cote@mymachine ~/PycharmProjects $ git clone https://github.com/pcote/appskeleton.git
Cloning into 'appskeleton'...
remote: Counting objects: 38, done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 38 (delta 6), reused 36 (delta 4), pack-reused 0
Unpacking objects: 100% (38/38), done.
Checking connectivity... done.
cote@mymachine ~/PycharmProjects $ cd appskeleton/
cote@mymachine ~/PycharmProjects/appskeleton $ cp -R * ../DomEditor/
cote@mymachine ~/PycharmProjects/appskeleton $ cd ../DomEditor/
cote@mymachine ~/PycharmProjects/DomEditor $

We switch over to that and get app-specific things setup.  There's a folder in the Ansible role that needs renaming by the way.  It needs to match the name we gave this app.

cotejrp@mymachine ~/PycharmProjects/DomEditor/roles $ mv myapp domeditor

Very good.  So now, we deal with configuration with the deployvarsdev.json file.  

{

    "hostname": "all",

    "domain":"domeditor.com",

    "appname":"domeditor",

    "mysql_root_password":"rootPassword",

    "proxyport":3032

}

Careful for that mysql_root_password by the way.  That variable has to be the same here as it is in basemachine's Vagrantfile.  Yes, the setup is stupid and the root password should be established in one place rather than two.  For the time being, we just accept that the value must match lest the Ansible playbook barf on you later.

Oh yeah.... The /etc/hosts file needs a domain name alias to localhost.  That's going to have to match the domain variable set in our json file.  So this line goes into hosts....

127.0.0.1       domeditor.com

Time to check on the base machine provisioning.  Ding!  It's done.  Time to move forward with application deployment.

Application Deployment

Deployment needs one more thing.  Time to open up provisiondev.sh for a little editing...  

#!/bin/sh

basemachine=/path/to/base/machine

inventory=$basemachine/.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory

privatekey=$basemachine/.vagrant/machines/default/virtualbox/private_key

ansible-playbook -u vagrant -i $inventory --private-key=$privatekey --extra-vars="@deployvarsdev.json" playbook.yml

See that basemachine variable near the top?  We replace that with the path to wherever we git cloned the basemachine folder to in the first place.  Something like this does the trick nicely.

basemachine=$HOME/PycharmProjects/basemachine/

And there's that.  Time to run this deployment script....

cote@mymachine ~/PycharmProjects/DomEditor $ ./provisiondev.sh 

A couple seconds pass.  Script completes.  Open up this url in a browser...

http://domeditor.com:8080/

And once we see the message "This is an angularjs stub page." in the browser, we know we're done.

Total Time Elapsed: 4 minutes 39 seconds.

Could it be faster?  Maybe.  One of these days, I'll have to figure that one out.

Digression: What Does Devops Mean Anyways?

A conversation with a friend raised the question of exactly how one defines the term "devops".  It depends on who you ask.  I can think of it in terms of management reality, my reality, and wishful thinking.

For a manager, it's about improvement in team dynamics.  Developers are paid to change I.T. systems.  Operations people are paid to keep everything stable.  These job definitions naturally put these two camps at odds.  The idea of devops in this context is to get developers and ops people on the same page to reduce the conflict.

For a solo act like myself, devops is about self-reliance.  There are no operations people in my world.  Naturally, it is my responsibility to set up, modify, and maintain those servers myself.  Devops tools like Ansible and Vagrant just happen to make that business easier for me.

Lastly, there is the wishful thinking vision of devops.  It's that dream of sitting down in a quiet environment to work.  You check out the project from the local Git server.  You do a "vagrant up".  Bang!  There's a virtual version of the same server as what ops is managing in production.  That's what you get to develop against.  It sure would make change change control discussions easier.  One can dream.

Making Your Own Google Tag In AngularJS

Playtime blog post!  So I made a custom tag on a lark the other day.  Here it is.



It is what it looks like.  It makes a link to Google with search results for Chef Ramsay.  Let's make this tag.

Taking Advantage Of Directives In Angular

Directives can do a few interesting things.  For this project, the goal is to make a new HTML tag.  A skeleton for this directive will get us started.




We really just want this directive used as a tag element and nothing else.  Setting the restriction to "E" will impose that limit.

jQuery level manipulation sounds nice given what we're going for.  The link method that Angular allows for is good for that.  Let's stub that one out a little.




What to do next?  That element argument is useful.  It's a jQuery object of sorts that seems to be referring to our custom tag.  What can we do with a jQuery object?  Messing with the information between the two "google" tags could be nice.




Pretty straight forward.  Grab the text.  Cut together a string representing a hyperlink.  Plug the resulting link back in.  All done.

In Summing Up

This is a just me fooling around here.  If custom tags are the sort of thing that you want to take a more serious look into, there are better tutorials out there.  Jakob Jenkov is one of my favorite resources for Angular JS and I recommend checking him out.  Just make sure to read his critique section first.  There's wisdom in what he's saying there.

And that's all for now.  Cheers.

Keep Your Database Permissions Small

Update: March 15, 2016
There were several interesting bits of legitimate criticism on this post on Reddit.  The idea of the Principle of Least Privilege that this post communicates is right.  Some of the implementation details in the examples fell a little short though.  A thousand thanks to all who commented.  I learned a lot from this exchange.  The original post below........
 

A typical web application needs an account to access a database in order to do it's job.  These applications often face the outside world.  As such, direct access to your database is only a few degrees of separation from your users.  How does one secure this?  Start with minimalism.

Make A Minimal Database Account

Start by setting up a database account.  When doing this, err to too few permissions rather than too many.  It is inconvenient in the short term.  However, over the life of your project, you'll be glad you did.  Things will be more secure and easier to maintain.

An example.  There is a feature in one of my projects where the user can delete learning resources.  Here is a piece of Ansible configuration that sets up the MySQL database account.



Notice that the account does not offer any delete permissions at all.  So how is deletion possible?

Coding Within Constraints

The trick is this.  When the user "deletes" a learning resource, they don't directly delete anything at all.  Instead, they put in a request to delete it.  This much only requires insert privileges.  Here is an example using Python and SQLAlchemy.



As far as the user is concerned, the resource is "deleted" right away.  This is thanks to a subquery used to help with the lookup.

Doing Real Deletions Safely

To be sure, that can take up a good bit of space over time.  A real deletion of data is ultimately called for.  Here's a snippet from a script that does that job.



Not that this script does use a database account with delete permissions.  It is also a task run through a cron scheduler on an Ubuntu Linux server.  This is important because it is not invoked by the main web application at all.  The system is more secure because actual deletion is treated as an internal detail that exists outside of public access.  Below is one way to set up a cron scheduled job to run every other hour.



The takeaway is this.  A good security mindset can lead to solutions that provide features to user while also keeping data safer.  So the next time there comes a situation where you think you need more permissions, stop and reconsider.  You might be able to get by with less than you think.

The Fumblina Full Stack Philosophy

The Fumblina full stack philosophy is a mix of tools and ideals that I swear by in my web application projects.  The core ideals are that you should pick your own tools, make stuff for yourself, be small, and have a sense of humor.  Let's expand on that.

Pick Good Tools For Yourself

Given the choice, pick tools that work for you.  Ask yourself, "Will this make it easier for me to make good software"?  What is your situation?  What are you building and why?  For point of reference, here is my thought process for picking tools.

The Front End

HTML and Javascript are givens for web applications.  Since I'm not a web designer, Bootstrap makes it easier to put together interfaces that work on a 22 inch monitor and Android phone.  

Usually, my project is some kind of single page application.  At the time the decision was made, AngularJS looked right for this.  That might change some time in the future but, for now, it's the Javascript resource I know best.

Middle Pieces

Choices of middleware were informed by my Python 3 biases.  Flask and SQLAlchemy are the two major pieces as libraries go.  Flask makes creating JSON-based web APIs easy.  SQLAlchemy let's me use a database toolkit without being forced to use an ORM.

As application and web servers go, googling around and asking questions on Reddit helped me decide.  That's how uWSGI and Nginx ended up in the picture. uWSGI handles my Flask stuff.  Nginx plays proxy to Flask and static files.

Back End Parts

MySQL on an Ubuntu server is something I know so I stuck with it.  The rest of the pieces of my stack fit naturally in that context.  It's easy to find help online if stuck.  These were easy choices.

Working a custom tech stack generally requires a custom server setup.  Virtualization and cloud environments make sense here.  On the development side, Vagrant and Virtualbox do the job easily.  For production, Digital Ocean is a cheap no-frills service for setting up a Linux server.

For deployment, my pick is Ansible.  Knowledge of YAML plus native provisioning support in Vagrant swayed me on this one.  Ansible uses an idea called playbooks.  Playbooks used to provision a virtual machine can also work on a production server with little to no changes.

And that's how my tools were chosen.  I went with what made the most sense and asked for input when in doubt.

The Acronym

By the way, if you do pick the same tools that I do, you can make a nice acronym out of it!

FUMBLINA = Flask uWSGI MySQL Bootstrap Linux Nginx AngularJS

I made that up myself.  Kind of proud of it!

The Small and Silly Fumblina Philosophy

Here's the thing.  A full stack developer is not an unstoppable force ready to tackle any large project thrown her way.  She is not the developer equivalent of this guy.

Mr. "Full Stack" I presume?

Below is a better way of thinking about what "full stack" means.

Being A Part Of Your User Base

The best kind of product to be working on is one where you are one of the users.  For example, I am the sole user of Learning Machine.  A big user base isn't the goal.  I want to make things easier for myself when it comes to learning new things.  There are tools to build that help me in that regard.   If a user base ever did happen, great!  Having a head start as "User 0" puts me in a better position of understanding their needs.

Keeping the Feature Set Small

My favorite explanation of the "Minimal Viable Product" concept comes from Extra Credits, a Youtube show about game design.  The video emphasizes that a new game should start out with nothing but the absolute core mechanics and nothing more.  This is what you want to get right before adding all the supporting attributes.

In a recent round of commits on my Learning Machine project, everything was stripped out except for three things.  

1. Making practice exercises.
2. Quizzing and rating yourself on existing exercises.
3. Reviewing a report to see where you're doing well and where you aren't.

Focusing on the core essence of a project is crucial because, as a full stack developer, there are limits to what one can do.  A full stack project for something small is challenging.  One with a huge feature set is going to spread you incredibly thin.  That's not a good thing.

Not Taking Yourself Too Seriously

Fumblina is a funny-sounding name.  It's the kind of name you give to a fairy with a lazy eye who can't quite fly straight.  When calling myself a Fumblina developer, I'm aware that I'm making fun of myself.  Laughing at oneself is a good thing and more people should do that.

Any "serious" full stack developer will have her share of stressful moments.  A sense of humor relieves stress.  Less stress leads to better judgment calls.  Better judgment calls are more likely to lead to the ultimate success of a project.  

And that's the whole Fumblina philosophy.  Or at least, that's how I like to think of it.

Processing Lots of Text Files with Python

Tasks involving work with text and files containing text is very common.  Python makes dealing with single files a breeze.  But what if you need to deal with LOTS of text files?  That can be a bit trickier.  Fortunately, Python is very much up to that challenge as well.

My Setup For This

The example discussed here runs on Linux Mint 17.2 64-bit.  Since we're dealing with Blender here, you can download the 64 bit version for Linux here.  The code being shown here is confirmed to run on 64 bit Python 3.4 for Linux.

The Goal In This Example

There's a piece of art software called Blender that interests me.  In Blender programming, there's an idea called "region types".  Region types help position interfaces.  My interest in is in lines that look similar to this.

bl_region_type = "WINDOW"

What region types are commonly used within the Python scripts that come packaged with Blender?  That is the question to answer here.  Answering it means scanning for info within the contents of multiple files.  There are four general steps to take in this sort of situation.

1. Decide on what files are of interest.
2. Decide what lines of those files are of interest.
3. Get whatever it is you need from those lines.
4. Act on that extracted info.

Let's break this down.

Picking Your Files

For this example, we are interested in files ending in .py within a specific set of directories.  

The walk function from the os module is very handy here.  It makes iterating through nested directories very simple.  It is also a generator.

There are two choices for dealing with generator based info.  You could use the generator to return a list of python file paths on one hand.  On the other, you could yield each of the created paths one at a time.  We're going with this second choice.  There's no reason that we need a full list of python file paths in memory all at once.  Having pyfiles as a generator is fine.

Picking The Lines

Let's assume that the line of code in question won't be broken up across multiple lines.  We'll also assume that the expression in question won't have comments on the same line.  That allows us to write naive code like what's below.

Some things to note here.  First is the usage of the with statement.  Using with in this way allows the file object's context manager to help us along.  We are dealing with dozens of files here.  Letting that context manager help close those files for us make sense.  

The second thing to note is that pick_lines is a generator too.  We don't need a full collection of file lines in memory so why have it?  

The third thing is the set up of the for loop. It is double-nested. It handles both file name collecting and file reading duties.  And yet, it is still easy to read and doesn't burden resources.  It's lazy.  It's beautiful.  It's also practical for what's being done here.

Getting What's Needed From The Lines

The goal is to get specific information from this kind of line.

bl_region_type = "WINDOW"

We want the WINDOW part and nothing else.  This only requires a few lines of straight line imperative code.  Adding more functions or generators isn't called for here.  Let's expand on our loop.

This is self-explanatory.  The important stuff comes after the equal sign.  The stuff before it doesn't matter.  Quotes and extra spaces don't count either.  All that gets tossed before throwing the desired info into a list.  And since this list is our final target, having it concretely in memory is the right thing to do.

Act On The Information

This can be whatever the situation calls for.  That could be writing info to a database or another file or whatever else you need.  Here's what I did.

This is pretty much what it looks like.  My main interest was seeing what region types were actually being used in Blender addon scripts.  Having that info and count tallies output to the console is enough.

And Finally The Completed Script

And here are the results.

WINDOW: 90
TOOLS: 56
UI: 78

A testament to the power of Python is what can be accomplished in barely a couple dozen lines of code.  Those few lines can parse through thousands of lines of text spread across dozens of files from various directories.  What you can do beyond that is up to your imagination.

Code Scribbling with Fizz Buzz

There's this thing that I like to do called code scribbling.  Code scribbling is about letting loose.  You noodle around with a piece of code for no particular reason.  Fun is the only goal here.  But it's an important goal.  Fun helps loosen you up for the important programming work you do.  The code scribbling session below is one such example of me just enjoying myself with a little Python 3.4.

A Code Scribbling Session

My mad plan here is to play a little Katamari Damacy with the Fizz Buzz problem.  I want to start simple and just add thing after thing until it turns into a big ball of chaos.  We'll start with a basic loop and modulus operators.  Here is that.

Let's shorten this up with some ternary operators.

Okay, that was concise.  Time to make it more interesting.  What if the two ternary operators were concatenated together and thrown into a list comprehension?  The fizz and buzz ternary expressions have to be wrapped in their own sets of respective parenthesis.  This ensures that it evaluates correctly.  The "==" aspect of the modulus expression can be reduced to something more "truthy" so that it fits nicer in a blog post.  Here's what that looks like.

Interesting if a bit of a space eater.  A generator expression would make it lazier.  At the very least, it won't make the code look any worse than before.  Here's that code.

What else could be done with this?  Here's an idea.  Those ternary expressions could be wrapped into a lambda function.  Here's the result of that.

What else?  We could take the string formatting out of the print statement and put that in the expression.  Then, the for loop just has to loop through the resulting list of strings. 

Hmm.  Come to think, why have the for loop at all?  The string join method could just join them all together with newline characters to ensure it prints like before.  Then, the print statement just has to print one big string all at once.

Hooray, no more normal for loop!  But this is interesting.  That entire expression we have could be made into it's own lambda expression in and of itself.  You could even pass an argument and print the result of that.  Here's what that looks like.

Okay, what's next?  There's no reason why we can't just move the entire expression and evaluation into the print statement itself.

It's all one big blob now.  The print statement uses the expression here.  But wait!  There's no reason the print couldn't be a PART of the expression.

Where do we take it from here?  Let's see.  Meta-programming perhaps?  There's nothing saying we can't create a nameless class with a __call__ method.  Then it's just a matter of making an instance.  From there, it's just a matter of taking advantage of __call__ and invoking the instance as if it was just another function. 

This cake is baked.  Let's decorate!  I'll just wrap the __call__ function in a staticmethod decorator and make that troublesome self argument go away.

Awesome!  All wrapped and decorated.  And we're done with this scribbling session.

What a Mess!

This is obviously not meant for production code.  This is essentially unfettered silliness enjoying some time in the sun.  Such is what creativity is.  It might accidentally lead to actual innovation but that's besides the point.  This is about taking some time to set your brain free.

So make it a priority this year to relax.  Have fun.  Scribble.

Using Decorators to Validate Flask Views

Flask is an awesome micro-framework and a great way of handling http requests when creating web apis.  What I'm proposing here is that custom Python decorators can make development of such projects even better.

Consider, for example, the http requests that post of json data.



What do we have here?  The model is an interface to the database.  It sends an exercise id and score to be persisted to it.  However, the get method calls on those json dictionaries can and will return None if there is no matching key.  Consequently, there is the potential for empty values to be inserted into the database.  We don't want that.  What to do then?

Validating Without A Decorator

One option is to look to make sure the keys are there and abort the api call with a 400 error.  Since status code 400 signals a bad request, the client will find out that the request bombed because they submitted the request incorrectly.  That would look something like this.

Validating With A So-So Decorator

The validation in the example above can be a little distracting.  Code is read more than written.  Extra safeguards are wise but also kind of distracts from the main point of what the function is ultimately trying to do.  So let's try a simple decorator and offload validation to there.




Not bad.  The only thing to bear in mind is that decorators are being chained together in this context.  The top decorator decorates the function that results from the decorators below it.  Flask is a framework.  We want it to register the final result of our customization with minimum confusion for all parties concerned.  For that reason, you generally want put the Flask routing decorator on top.

Generating Decorators

We're not done yet.  What about the other Flask functions dealing with json data?  Do ALL the json requests passed in to other functions use the exact same keys?  If not, the decorator we made so far won't work on those functions. We could create separate decorators specific to the keys of each function's json arguments.  Unfortunately,we'd end up with the very code repetition we've been trying to avoid.  What to do then?

One solution would be to write code to automate the creation of decorators that fit each target function's validation needs.  So what we need is something that creates other decorators.  When we do that, the end result is a decorator that takes arguments.  Here's what that looks like.


Not bad!  Not bad at all.  Accepting a variable length argument list allows us to put in as many string arguments as needed for the keys that need checking.  Those arguments strings are used to create a custom decorator on the fly.  That newly created decorator wraps the function we're targeting.  As a result, we now have a special kind of decorator that can be used to validate any json data at all.  Just specify the names of the keys and you're golden.

More Resources

There's a lot about decorators that hasn't been covered in this blog post.  Here are some other angles on what I've written about here.

• The Code Ship Guide to Python Function Decorators
• Stack Overflow Question on Chaining Decorators