Hey there, just wanted to leave a signpost for you. My usecases lately have been something like -

Computing a BIG table, with a lot of math in it, over a LOT of rows of data, and then joining in other data to enrich the primary set. Specifically, this is container usage data, which I'm attempting to blend with our AWS bill to arrive at something like “cost per container” per time period.

I don't want to have to rebuild this table every day because most of the data is static once it shows up in the warehouse. An incremental strategy would be perfect BUT, some of this data arrives late, which means that if I do the standard DBT routine of

WHERE timestamp > (SELECT MAX(timestamp) from {{this}})

then I will have gaps. Indeed, I have gaps. I haven't rolled out any reporting on this table, or made any announcements because I felt a disturbance in the force, confirmed by some light analysis this morning.

I've recently discovered a new DBT hammer in the incremental_strategy parameter for incrementally built tables, and specifically the insert_overwrite option. From the DBT docs:

The insert_overwrite strategy generates a merge statement that replaces entire partitions in the destination table.

In short I can just always recompute yesterday and today, or the last 7 days, or whatever full partitions-worth of data I want. Yes, I'm recomputing more than I strictly need to, but it assures me that there will be no gaps in the results.

This operation seems pretty foolproof so far, check it out.

#analytics #DBT

The problem I have with the world is this –>

It's too friggin hard to build a good website.

I spent the better part of 2009 studying this problem. I was bummed out about RRE's website and the fact that it was just a little too static, a little too disconnected from the rest of the internet. It was a custom built solution, and God bless you Berry for that, but I didn't even know what that meant at that time. It took me several months of poking, prodding, and pulling my hair out before I started to get the idea.

People have been here before.

There is a thing out there called a CMS. That stands for Content Management System, and what it does is take care of the nuts and bolts behind the scenes so that you can get on with posting cool stuff. That's what you do after all, right? Tour dates, news, pictures, it's all just stuff. People have been putting up stuff since the dawn of the internet, so it's not surprising that some enterprising types have tried to simplify the process. There's just one problem.

It's still too hard to build a friggin website.

I mean, let's be honest –> raise your hand if you know what DNS is, and why you should care. Yeah. Now, if you're not a musician, put your hand down. That's what I thought. And that's only one infinitely tiny little facet of a web site. Good luck with the rest. I'm a pretty driven person at this point in my life, and after banging my head on the internet wall for a year, the best I could come up with is this. I had to learn an inordinate amount of routine, sub-geek level internet crap that has nothing to do with putting up cool stuff before I could even get started. Can't someone please just take all of this responsibility away from us musicians?

next

#the-idea

The ongoing sorrow in my life is this –>

You can build an awesome website, and then where are you?

The answer is that you're ahead of at least some of the craptastic pack. The rest of the answer is that you've just taken a major step into proving that you take yourself seriously on the internet. You have opened Pandora's box, my friend, and if you think you can shut it, the world will know just what a punk you are. After all, the days when your website was the totality of your internet presence have faded into our collective long-term memories by this point. I'm not gonna say “social networking is just as, if not more important” because that would be insulting your intelligence. Besides, you already know how much more time you spend on Facebook than the best band website out there.

So now I have to babysit this thing, too?

And this is where the drudgery begins, my friends. Take it from someone who valiantly attempted to up the ante on RRE's web presence –> the internet is a big place. There are plenty of musicians out there that are intensely adept at using one or more facets of their web presence to the fullest effect, but it takes practice. If you're on Twitter, would you do me a favor and follow the @stringdusters? I'm not sure who does the posting for them, but I think it's the one Anders calls “Panda”. He gets it. And he gets it more every day. It's been something to behold over the past few months, and it makes me envious somewhere in my heart that I'm not as good a Twitterer as they. But I also know that someone with a hammer that bangs Twitter nails probably isn't as sharp or as diligent about banging the Facebook nails, or the Tumblr nails, or hell, what's going on on their website?

Are you with me so far?

next

#the-idea

Shouldn't it be a little more like this –>

You're a new user to the iBD CMS. The initial login screen is simple, clean. It has 3 fields that you need to fill in to register – Your name, your password, and your band's name. You press the submit button and that's it. Your new website is active. The second screen you come to asks for your Facebook login information. You enter it, and since your an admin of your bands Facebook page, you just directly imported all the data that is present on your Facebook page – that's a bio, your tour dates, band members, any contact info, links to videos and photos, pretty much everything a website needs is automatically populated with the press of a button.

That's cool! But wait, this bio is a little outdated. It still has the story about carving a path deep and wide. Let's change this a little bit. And once we're done, we press save and it's automatically sent to Facebook, since your admin account is already linked.

Are you starting to get the picture?

beginning middle

#the-idea

Quick bio – The difference between As in, the difference between your brain and your computers CPU. The difference between how you think and speak and interact, and how your computer does. The difference between languages, be they interpreted or compiled, object oriented or procedural. This first chapter will hopefully assist in the process of the first few days after the moment where it first occurs to you that you want to learn how to develop software, be they games or business or iPhone apps. The real difference for me was the the one between what I thought the process was going to be and what it's actually turned out to be.

The moment occurred for me in a car. I had bought an iPhone a few months after it first came out. It was the first such early adopter move that I'd made, technologically speaking. I knew as soon as I started screwing around with it that this was a tool that could really help you get your life organized, something that I'd managed to get around doing for the previous 30 years. All of the built in applications that Apple provided for their first tier of users were most helpful in that task – email everywhere, calendar, internet, maps, oh yeah, and a phone. The real fun didn't begin for me until they (Apple) released the 2.0 software update. This was the update that heralded the opening of the AppStore, where there were now hundreds of apps available from 3rd party developers that could extend the usefulness of your phone. Anyway, to make a long story no less long, the first long car ride I took after updating my phone was the one where the real potential of mobile computing dawned on me.

I came home that weekend determined to figure out what it would take to learn the art and craft of building my own applications, as this was clearly a vast and uncharted territory for creative people to make things that could affect the lives of huge numbers of people (and maybe even make a living doing it).

I got home that week and started Googling. A search on Computer Programming turned up a mountain of information that was so staggeringly huge as to be almost useless. I did learn a few things – Wikipedia can be your best friend; there are many different ways to program computers; there are endless amounts of information on programming computers, most of it written by computer programmers with no conversational or literary ability whatsoever.

I was able to discern a few important pieces, however. One was something called “compilation”. I wasn't sure what this was or why it was important, but hey. Another was that if you're going to develop software, and you're used to working in a more or less standard Windows environment, you're in for a ride. Another was that there was something called Object Oriented programming, that supposedly is some kind of elevated way to program. iPhone apps are written in an OO manner. Oh, and if you really want to develop apps for the iPhone, you have to buy a Mac. That's some clever marketing, and the push over the cliff that I'd been waiting for for years. The bonus was that once you bought the Mac, all of the tools to develop apps, as well as an enormous user community was there to help, for free. Step 1.

#the-idea

Now, this part won't be any news to anyone, but will mainly serve to help organize the thoughts in my brain. First, the old way:

---

The old way involved the “record industry”. The record industry used to exist because recording was very expensive. It was expensive to record a song, it was expensive to reproduce the recording of the song, and it was really expensive to warehouse, distribute, and sell the recording of that song. Thus a whole industry cropped up to take advantage of the fact that the barrier to entry for your average recording artist, say Ma Carter out of the hills around Bristol VA, was so astronomically high that nobody really thought about releasing their own music. Show up, play my tunes, get paid for them? Okay! This worked great for long enough for the basic oligarchic framework of the major label system to rise to power.

A moment now to reflect. It's extremely popular to bash record labels, and with good reason. Let us now take a moment, however, to reflect on the cultural purpose that they've served...

I take it for granted that America is, on balance, the coolest nation on the face of the earth, in the history of humanity. Citizens from other countries would doubtless dispute this claim, but I would submit to you that even those societies that profess to hate everything America stands for only hate so strongly because they don't enjoy the liberties and luxuries that Americans have long grown accustomed to. Now, imagine if the record business never existed, that there was no such thing as an LP or a CD, and the only way to enjoy music was to go listen to it live. There were no Beatles, except for those of you lucky enough to be in the Cavern Club, no Elvis, unless you went down to the BBQ shack to see him in Memphis, no Chuck Berry, no Led Zeppelin, no Pink Floyd, no Eagles, no Neil Young, no Bob Dylan, no Coldplay, no U2. I know, a lot of these bands are from the UK, but I consider the give and take of the UK and the USA to be part of the competitive exchange that has pushed the limits of musical creativity for the better. And we can safely lump all these bands into the category of Western Music. Now, imagine if Western Music never existed. How would we export our culture abroad? At gunpoint, probably. How would generations of kids be incited to stand up and make change happen? At gunpoint, probably. My point is that exactly at the moment in human events that the USA and the UK became 2 of the dominant political powers in the world (1946 or so), rock and roll made an appearance. It moved legions of Japanese kids just like it did American kids. The cultural service that the major label system provided these kids and us to come later is therefore immeasurable. And the world danced together...

Now, back to the point, to be covered in another blog, because really long blogs are fucking boring...

#music #business #the-idea

To speak into the void.

So I've had this idea simmering on the front burner of my brain for several months now. But first, I should probably introduce myself, or do a bio, or something. Maybe this will take several parts, because I've never tried to write a bio, and I feel like I've done a good job living unconventionally with my first 30 years here...

I was born in ATL, Georgia, that is. No brothers or sisters. Great parents. Always totally behind me for some reason. I got pretty mediocre grades at the private school they sent me to, and spent most of my time getting in trouble. Nothing big ever, no real cause for concern, I just didn't really feel like fitting into their mold over there. I realize now in hindsight that I left high school with pretty low self esteem after being told for half of my life what an underachiever I was, how much potential I had that if I just “applied myself”. I had some really good friends, though, and at some point found music.

I'd been in the school band since 6th grade, trombone. I realize now that I never knew how to read music. I think I learned the parts by listening to whomever I was sitting next to. That amazes me in hindsight, but nobody picked up on it, least of all me. I was that into the trombone anyway, I was much more into the drums. I used to get on my band directors nerve because as soon as we got into the room I'd be back in the percussion section, not learning the trombone parts.

“Why didn't you just switch to percussion?” you might ask..

“Because we need trombones” was the answer I got for years and years whenever I asked to switch to percussion.

There was one time that I was really into building model planes. I didn't use plastic cement, for some reason I used a hot glue gun. It was messy and my models turned out looking like crap, but that's not the point. One time I'd set my glue gun down, with it pointing into my glass of water. It had dripped some hot glue into the glass, apparently, right before I took a sip. I burned the crap out of my top lip and had a huge ugly scab for weeks. The band concert was coming up that week. I couldn't play the trombone. I got to be a percussionist for a week. It was like heaven. I couldn't read music for the bass drum either, but I damn sure could make the part up. Nobody ever knew, and I never told them. I totally forgot about that until now.

Anyway, at some point I got more or less kicked out of the band. I skipped a mandatory concert to go on a youth group trip with my church. I asked them if I could skip the concert to go on a youth group trip and they said “no”. This would've been a reasonable answer, except that five minutes before my best friends James had asked them if he could go on the exact same trip that I was asking to go on, and they told him “sure”. So I went anyway. Now I remember why. Amazing this blogging...

Anyway, now I'm in a band, and right now I'm on a tour bus heading to the GA theater in Athens. I've got some time to kill, so I'll keep catching you up, whoever you are. At this rate, I'll get to my big idea in another 20 posts or so.

#memories #the-idea