Ignored By Dinosaurs 🦕

Hello, and welcome back to “Platform.sh from scratch”, see also the first post in the series. The goal here will be to augment the official documentation with a short tutorial that shows how to set up a project for proper functioning on Platform.sh. We'll dive into the “why” as little as possible here. For now let's dive straight into the “how”.

We're going to start with a very basic application, the example Silex app on the front page of the Silex website. This will be a standard Composer based project, so we'll need a composer.json file to start with.

The project structure will look like this —

jgrubb in ~/play/php/silex-test on master λ tree -I vendor
.
├── app
│   └── index.php
├── composer.json
└── composer.lock

The composer.json file can be created by running composer require silex/silex, or you can just copy this into composer.json at the root of your project directory —

{
 "require": {
 "silex/silex": "^2.0"
 }
}

Run a quick composer install and the rest of the dependencies will be pulled down and placed into the standard vendor directory. We're going to be using Git here, and in general you don't want to version 3rd party dependencies like those that Composer downloads. Let's create a .gitignore file and add the vendor directory to it.

echo "vendor" >> .gitignore

The entirety of the application codebase looks like this —

// in app/index.php
require_once __DIR__.'/../vendor/autoload.php';

$app = new Silex\Application();

$app-get('/hello/{name}', function($name) use($app) {
  return 'Hello '.$app->escape($name);
});

$app->run();

Very simply, all this app does add a route that responds to requests along the path of hello/{whatever}. As long as you've used the same directory structure, you can cd into the app directory and run php -S 127.0.0.1:8080 to fire up the local php webserver and then head to localhost:8080/hello/user.

If all is working as expected, let's head to Platform and get this thing ready for the development process.


Navigate to the platform website and register a new account. You have 1 month to (freely) evaluate whether or not Platform fits your needs, so let's get going. I'm assuming that you can find your way through the registration and login workflow and find your self back on the “your account” page, so let's add your SSH public key into your account and that will be all for configuring your account for now. Under “account settings” –> SSH keys you can add a public key.

[!info] Sidebar – SSH public keys

This is required for one main reason – it allows us (Platform) to securely authenticate you when you push code to a project. This is standard for most public code repos (GitHub, Bitbucket), and we use this method as well. A massive side benefit of this workflow is that it allows any Platform account holder to invite any other Platform account holder to their project. This means that agencies can invite developers to projects, users can invite other developers to help with their projects, and the overall friction of matters of authentication and authorization on projects is reduced to virtually zero. You won't likely notice this benefit until a little later on, but when you want to share a project (or even a specific branch of a project) with another user no new account/password/validation workflow is required, and work can begin immediately.

Once that's done, let's go back to the main account page and “create a new platform”.

To me this workflow is pretty self explanatory and the defaults are the correct settings for now, so select which region you'd like your project to be hosted in and get through the checkout workflow. Like I said, nothing will be charged for a month, so have no fear. I have to go through this flow as well, and I work here...

After you get through that flow, you'll be dropped into the “projects” admin of Platform.sh. This is where you'll likely be spending the vast majority of your time.

Name your app (I'm creatively naming mine “Example Project”), and then for this project you'll want to choose to “import your existing code”. This option will present you with a URL for a git repo to which you'll be pushing your code. Now is the time to initialize a git repo in your codebase.

git init


[!info] Sidebar – infrastructure as code

We'll get into the mechanics later, but this would be a good opportunity to explain the overall ethos of Platform.sh which is that you are going to be specifying your infrastructure – that is the underlying software systems on which your project will be running (MySQL, Redis, etc) – in code. You'll be able to manipulate the infrastructure required to run your project in the same way that you manipulate the behavior of your app through writing code, and you'll push these hardware requirements to us in the form of code.


Run git add ., which will add all 4 files in your project to git, and then git commit -m "init commit" to commit your code. After that you'll want to cruise back over to the Platform project admin screen and copy the lines under “Push an existing repository on the command line” and drop them into your terminal. This will add the Platform git server as a remote for your project, so now you have somewhere to push your code. We're almost there!

But we're not totally there yet, there's one more step. You need to tell Platform what your project needs to run or you won't be able to push your code up to us.

All Platform hosted projects require 3 files – .platform/routes.yaml, .platform/services/yaml and .platform.app.yaml. The routes file is just that – it's sort of like a front controller for your entire project. What this means in practice is that you can have multiple applications running in your project (under different paths), but for now you really just need to route any request to your little example application.

This is a nice standard starting point for any given PHP project, so place this in .platform/routes.yaml

"http://www.{default}/":
 type: upstream
 upstream: "app:http"
"http://{default}/":
 type: redirect
 to: "http://www.{default}/"

No, this is not the most beautiful file, but all you really need to know about this is that all URLs that enter this project will either have a base URL of

  • www.whatever.foo and will be routed to your codebase, or they'll be
  • whatever.foo and will be redirected to www. See step 1.

Another file that you need to have in place is the .platform.app.yaml file, which is a file that describes the high level requirements of this application. The most bare bones file is all that we need and it'll look like this —

# The name param is linked to the "upstream" parameter in
# routes.yaml. If you called the app "foo", then the
# upstream parameter would look like `upstream: "foo:http"`
name: app
# The "type" parameter takes the form "language:version".
# This could be `python:3.5` for example
type: php:5.6
# Look for a composer.lock (or composer.json) and download
# the listed dependencies
build:
 flavor: composer
# How much disk space will this app need? This is primarily used for
# user uploaded assets, so for this application you don't really need
# anything here, 256 would be fine. You can always grow
# this later, so this is a safe starting point. (in MB)
disk: 2048
# Now that a request has gotten this far, how do you want
# it handled? We'll go into more detail about these params
# in a later post. This section can be thought of as
# somewhat analogous to an Apache or Nginx config file.
web:
 locations:
 "/":
 root: "app"
 passthru: "/index.php"
 index:
 - index.php
 allow: true

There is more information on the documentation website about this file, and it's all worth your time.

The services file will define what other services (this is where MySQL comes in) your app depends on but since we don't need any yet, this can remain empty. It *does* need to be there however, or you won't be allowed to push your project, so for now just create an empty file – touch .platform/services.yaml.

Your project's file layout should now look like this (excluding git stuff) —

├── .platform
│   └── routes.yaml
│   └── services.yaml
├── .platform.app.yaml
├── app
│   └── index.php
├── composer.json
└── composer.lock

So with those three files in place, add them and commit them to your git repo —

git add . && git commit -m "adding platform config"

git push platform master and you are off! If you are still looking at the dialog screen in the Platform project admin, you can click “continue” now and you will (or will shortly) see a log screen of all the relevant activity for this project – git commits and the project creation before that.


After this and every successful git push to Platform those files will be analyzed to make sure that the infrastructure that your project requires and the infrastructure that is available to that project are in line. If something has changed or if this is the first time you've pushed code to this project, the environment will need to be set up with the services that are expected. This takes a moment, and then your code will be mounted into the environment. At this point you'll have a running project that you can visit by going to the project admin dashboard and following the “access site” link near the top of the page.

This concludes this step! It may seem like a lot to get a 5 line PHP project going, but think of what you *didn't* just have to do – spin up a server, set up a shell environment that feels home-y on that server, set up a LAMP stack, set up a build process for getting your code onto the server in a defined, runnable state, fuss with DNS or local host entries. We haven't even touched on the aspects of Platform that will completely blow your mind, so stay tuned.

#platformsh

So this is it, my week in between my old job and my new job. And I'm bored out of my mind. So I'm going to do something I've never done before – write a blog post on my phone. We'll see how this goes.


I was texting w one of my former coworkers a little earlier today. He was picking my brain about how the AWS command line tools work and so I was explaining some things to him in too much detail. I've noticed this thing that I have where I want to explain the why of things and not just give a one liner that can be copied and pasted to accomplish a job. If this blog has any regular readers maybe you've noticed this as well. Whatever.

So he was kicking some cli one liners to me to look at and in one of them was providing incorrect arguments to one of the options. I referred him to the excellent documentation page on this particular command and I think he must've gotten it sorted out after that because i didn't hear back from him after. I realized consciously something that i guess I've been doing a lot of the last few years – reading a lot of documentation, for fun.

I think one of the common modes to be in when you're reading documentation is that of trying to figure out a solution to a specific problem. I'm trying to figure out how to upload some files from my laptop to S3, so I'm researching the page for the correct arguments to pass. I'm trying to figure out how to avoid the N+1 problem on the blog listing page, so I'm reading up on which methods are available in the Django ORM. This is fine obviously, but I think what really separates senior devs from non-senior devs are aimless wanderings through the documentation for a project.

It's in these wanderings that you discover what a tool can do before you actually need it to. When the need finally does arrive, it's much less of an interruption to your workflow to look up the correct syntax for a feature that you already know exists. Otherwise you're stopping productive work for X amount of time to see if the your concept for the solution has some corresponding feature in said tool and by that time your flow is shot.

So when admonished to RTFM, take it as a passive aggressive offering of really good advice.

#generaldevelopment

I'm starting a new gig in a couple weeks. I found out about the existence of the gig in the first place because I follow the guy who will be my new boss on Twitter, since he's been around the Drupal scene and very long time and following the leaders in your industry is basically one of the core things that Twitter is *for*.

You don't need to necessarily know every single leader in every single lang, but if you work on the web and want to move your career forward, you'd do well to know who most of these people are, in absolutely no order. This is absolutely (and hopefully obviously) not an exhaustive list, more of a starter pack.

  • Dries Buytaert
  • Taylor Otwell
  • Fabien Potencier
  • Guido van Rossum
  • Jacob Kaplan-Moss
  • Kenneth Reitz
  • Matz
  • David Heinemeier Hansson
  • Rich Hickey
  • Brendan Eich

If you're a Drupal dev, you should add these to the list

  • Larry Garfield
  • Robert Douglass
  • Peter Wolanin
  • Angie Byron
  • Jeff Eaton

Getting to know not just the tools out there but the folks who made them and their reasons for making them can really help you decide if a given tool was created for a use case like yours.

#generaldevelopment

Problemspace

We recently migrated from using a local Drupal filesystem (Gluster) to using S3 to house our uploaded site assets. This was relatively simple, and killed at least two birds for us, metaphorically speaking. Some of my findings are chronicled in the previous post linked above.

We are loving that we don't have to worry about syncing files between environments anymore, which means that when we are developing a site locally, the image sources are all pointing to their S3 URLs and so everything Just Works. The only tiny problem is that if anyone needs to upload an image in development it goes up to the same production S3 bucket. Obviously this costs us next to nothing, but it bothers my sense of cleanliness.

Solutionspace

S3 has a “lifecycle management” feature that will let you Do Stuff with your bucket assets. Do Stuff is things like delete assets after a certain period, or move them to another “storage class”, which is not in the scope of this post...

The limitations of their lifecycle mgmt are a major bummer. They can only be applied to directories within a bucket or to entire buckets themselves. They cannot be applied (simply) to individual objects. If they could, then the fix would be simple – have a hook on file uploads that adds a “delete-after” header to objects that are uploaded from anything but the production environment.

I'm starting a new job in a few weeks, and they use Ceph for managing network files. I haven't even gotten on board with this gig yet, so I don't know what's going on behind the scenes, but Ceph does have this individual object expiration feature, at least according to their bug tracker. I'm wondering if this can be brought to bear on this issue, because once you don't have to move or copy files between environments anymore, going back feels kind of anachronistic.

#devops

I work with a guy. He's incredibly smart. He's the seniormost developer here, and if you need to learn something new and get something large done, he's the guy to do it. We basically dropped him off in the AWS jungle and told him to learn Hadoop and the entire Hadoop ecosystem for a data warehouse project and he did it.

I work with another guy. He's also incredibly smart. But he asks me for the answer before attempting to find it on his own more often than not. He's got a point when he says “it's a lot faster for me to just ask you rather than spend time trying to find it on my own”, because he's here to do a job after all. I get that. But the best analogy I can come up with is a spin on the old adage -

You can give a man a fish, and he eats for a day. You can teach a man to fish and he eats for a lifetime.

There's a third kind of person, though – the person who goes out and finds out about fishing on their own and then teaches themselves how to fish. This person will be your boss, and will always be employed.

#generaldevelopment #life

Because sometimes you need to roll out a bunch of taxonomy terms across 26 sites, and you just don't feel like clicking those buttons.


$terms = [
  'iReport',
  'Infographic',
  'Video',
  'Case Study',
  'Application Note',
  'Data Sheet',
];

$vocab = taxonomy_vocabulary_machine_name_load('vocab_machine_name');

foreach($terms as $term) {
  $t = new stdClass;
  $t-name = $term;
  $t->vid = $vocab->vid;
  taxonomy_term_save($t);
}

Save this to something like create_terms.php and then run it with drush!

$ drush @site scr path/to/create_terms.php

#drupal

Problemspace

  • You're working with a managed hosting provider and have begun to run out of room on the local/networked filesystem. Vendor wants to upsize your storage for a charge.
  • You've got several different environments that you're working with (local|dev|prod|etc) and syncing the filesystem between them is an annoying chore in 2016. The various methods out there of proxying these requests don't really excite you.

Solutionspace

How about moving the Drupal filesystem up to The Cloud? One of Amazon's earliest products in AWS was the Simple Storage Service (S3), and one of it's core usecases is serving public files like images for websites, removing the need for storage of the assets and the (admittedly minimal) compute resources to serve them.

We had both of the issues outlined above and have just completed a migration of all our files up to S3, so I thought I'd write down some discoveries.


s3fs module

This is a rather unfortunately named module, since there is another open source project out there with the exact same name. I knew of it first and so assumed that this module had something to do with that, but that's not the case. Btw, I learned this in Steven Merrill's excellent session at DrupalCon in May, check out the video if you're still with me.

In a nutshell, s3fs hijacks the Drupal filesystem. You can put both your public and private filesystems up there, simple to do since S3 has a very rich permissions feature set. Just don't deliberately make your private filesystem “public” and you're set. It then rewrites any URLs that would've been to assets on your local filesystem to point to their new location in S3.

The setup is pretty straightforward, so just a few observations.

  • For multisite, you need to override a few things, namely the default setting for “S3 root folder”. For our install we needed to separate each sites assets into site specific folders with the same S3 bucket, so we filled that setting in with a string unique to the site, something like “nameofsite.com”.
  • There are UI buttons for moving your local files up to S3, but the AWS CLI works *WAY* faster. There are a wealth of well-documented options to pass, but the gist is this —
aws s3 sync . s3://NAME_OF_BUCKET/nameofsite.com/s3fs-public/ --acl public-read
  • After moving the files up to S3, the module needs to be made aware of what files exist up there, so you'll need to refresh the file cache. If you forget to do this part and flip the switch to have s3fs take over the public filesystem, you'll see bad things.
  • With a multisite setup, I found it much easier to flip the switch that says “Do *not* rewrite JS/CSS URLs”. The downside of this is that I have to make sure that random assets in the Drupal filesystem (ie not within public://) also exist in S3, since so many CSS and JS files refer to assets by root relative paths. This is a hack, but that's life sometimes.
// from Drupal docroot
$ rsync -av --prune-empty-dirs --include='\*/' / 
--include='\*.jpg' --include='\*.png' --include='\*.svg' / 
--include="\*.js" --include="\*.css" --include="\*.gif" /
--include="\*.woff" --include="\*.ttf" --include="\*.map" /
 --exclude='\*' . ~/some/destination/dir

This says “gimme all those file types in the whole Drupal file tree and move them over to some other dir” that I can then use AWS CLI to sync up to S3. You should take the opportunity before running this to delete all your local public:// files, because they'll get sucked up in this command as well. You won't need them anymore after you do this migration anyway.

// from ~/some/destination/dir
$ aws s3 sync . s3://NAME_OF_BUCKET --acl public-read

All in all fairly simple, and in theory makes our setup much more portable between environments as well as vendors. Another excellent writeup of this module can be found here.

#drupal

How I very deliberately ended up in tech

This journey is fairly well documented on this blog, but I'll distill it down to a post, since I've answered this question a few times lately.

Early years

I started playing guitar and bass as a teenager, and upon arriving at college had to declare a major. I chose music since that's the only thing I could see devoting the majority of my time toward studying, and picked the Recording and Production track in the Music Industries Studies dept at Appalachian State University. Only later did I realize that my affinity for recording and production was due as much to my love for computers as my sterling ears.

After college I joined a band and spent 7 years on the road, traveling mostly all of North America, but occasionally the world too. I did not have a desire to spend my life on the road playing music, however. I always pictured moving from a music performance career into a music business career as the industry hadn't completely imploded yet. It was actually at the same party described in this post that I had the vision of where I wanted to be later in life, and it was in “the business”.

The road

So years went by and I had more or less learned everything I cared to know about the music business when I had another epiphany. This post chronicles that one, and it was about technology. It was on a long car ride that I discovered, truly discovered, the economy that was about to come. “Oh! This is going to be a thing!” I thought to myself that first day of the iPhone App Store. “A thing that if I learned it, I could continue to be my own boss, continue having a creative rewarding career, and probably even make enough money to feed my kids!”

So I went for it. I bought myself a Mac laptop in July of 2008 with the goal of teaching myself iPhone app programming. I had no idea what I was doing or even how to learn.

After the road

I got truly sick of my previous career in early 2009 and spent the rest of that year planning my exit. After quitting at the end of 2009 I had nothing but time on my hands to spend about 100 hours a week banging this stuff into my brain.

I had a contact that threw me some work and out of that came the first Drupal site I ever built. In 2010. 2010 was a rough year by every metric, but we got through it and by 2011 I had a new music gig and a full funnel of contract work. Life was looking up.

Child #3 and beyond

Life got turned upside down again with the arrival of son #3 in 2012. At this point I had no time to fill the contract funnel anymore and decided to take the dreaded “straight job”. I'd only intended to stay for about 6 months until the waves calmed down at home, but as it turned out I really like working with people, and the people at the job were really cool for the most part.

I've been here since, almost 4 years now. I've moved from mostly front end developer to mostly Chief Architect of this entire joint, since almost everything interests me on some level and my boss and I have a really symbiotic working relationship. I also (in my opinion) excel at seeing the big picture of the system and figuring out how to get it done.

#life

Because sometimes you need to roll out an image style across 26 websites, and dammit you just don't feel like dealing with Features.

php

/**
 \* Adds mobile_content_image style
 \*
 \* @param $sandbox
 \* @return bool
 \*/

function hook_update_N(&$sandbox) {
  $style = image_style_load('mobile_content_image');
  if (!$style) {
    $style = image_style_save([
      'name' = 'mobile_content_image',
      'label' => 'Mobile Content Image (500 x 250)'
    ]);
    $effect = [
      'name' => 'image_scale_and_crop',
      'data' => [
        'width' => '500',
        'height' => '250'
      ],
      'isid' => $style['isid'] // presumably returned by the call above?
    ];
    image_effect_save($effect);
  }
  return TRUE;
}

#drupal

Problemspace

I want to be able to link a set of posts together in an order. If there is a next post relative to the one I'm on, I want a button to show up that says “next post” and links to it. If there is a previous post relative to the one that I'm on, I want a button that says “previous post” and links back to it. Pretty simple, conceptually. Basically I want to reproduce parts of the Drupal book.module as minimally as possible.

So my first naive attempt was to add 2 ForeignKey fields to the Post model – “previous” and “next”.

class Post(models.Model):

	title = models.CharField(max_length=255)
	body = models.TextField()
	summary = models.TextField(null=True, blank=True)
	slug = models.SlugField(max_length=255)
	pub_date = models.DateTimeField('Published at')
	published = models.BooleanField()
	tags = models.ManyToManyField(Tag)
	created = models.DateTimeField(auto_now_add=True)
	updated = models.DateTimeField(auto_now=True)
	previous_post = models.ForeignKey(
		'self',
		related_name='previous_post',
		blank=True,
		null=True,
		on_delete=None
	)
	next_post = models.ForeignKey(
		'self',
		related_name='next_post',
		blank=True,
		null=True,
		on_delete=None
	)

This worked on the front end but immediately raised a stink alarm, for a couple of reasons.

  • You'd have to go and save this info twice for it to really work. Once on the current post and again on the referred post to link it back. == Workflow suck

  • The truth about this ordering would be stored in two places, so it'd be really easy to mess something up and get out of sync.

This is essentially a doubly-linked list if you're keeping score, with the encumbant maintenance problems.

So I thought to perhaps override the save() method in order to hook into the operation and automatically populate the correct field on the referred item, but then of course, I'd have to do all kinds of gymnastics to watch for if that field were to be removed at some point and remove the corresponding field on the referred item, etc. I mean, it's a blog who gives a shit, but I've been doing this for long enough now that I can't help myself.

Another option in this same vein is to use the Django “signals” subsystem to hook into the same functionality, but the smell remains.

After coming home from DrupalCon it occurred to me that really all I need is the one pointer, since I should be able to derive the pointer back. I just had to figure out how to do it...

This is a pretty obvious use case – automatically deriving any pointers back to the current item. It just requires one extra DB query to ask “give me any items where the previouspostid is this item's id”.

The key is the related_name argument to the model.

I think this is automatically set for a normal ForeignKey field, but on models where the foreign key points back to the same model it's required. Judging from the docs, I was trying all manner of post.post_set, etc but it's actually just post.previous_post, which is counter-intuitive since what you're actually getting back from that is the “next” post. I chose to keep the “previous” field since you could just add the previous post as you're authoring the current one.

Current post model looks like this —

class Post(models.Model):

	title = models.CharField(max_length=255)
	body = models.TextField()
	summary = models.TextField(null=True, blank=True)
	slug = models.SlugField(max_length=255)
	pub_date = models.DateTimeField('Published at')
	published = models.BooleanField()
	tags = models.ManyToManyField(Tag)
	created = models.DateTimeField(auto_now_add=True)
	updated = models.DateTimeField(auto_now=True)
	previous = models.OneToOneField(
		'self',
		related_name='previous_post',
		blank=True,
		null=True,
		on_delete=None
	)

And the prev/next fields look like this —

{% if post.previous %}
 
[← previous: {{ post.previous.title }}]({% url )

{% endif %}
{% with next_post=post.previous_post %}
 {% if next_post %}
 
[next: {{ next_post.title }} →]({% url )


 {% endif %}
{% endwith %}

note

This might not technically be a linked list in the strictest sense, since a singly-linked list has pointers to the next node in the chain. I've implemented it here as a “previous” pointer, since it makes more sense in the edit workflow. Since it makes more sense, hopefully we'll make more cents!

Stay tuned for the next episode where I decide that I'd like to have a Table of Contents and rip this whole thing out and do it over again.

#generaldevelopment #django