Ignored By Dinosaurs 🦕

I was thinking about this a few hours ago, and so the thoughts aren't quite as sharp as they were as I was doing dishes after my morning coffee. This is often the case. Coffee provides me the wakeup that it does most people, but it also brings a ... idk, how else to put it. Coffee makes me kind of aggro. Fortunately I have realized this, since some years ago I would use the morning energy to write thoughts of importance to people of importance, and then later think to myself “maybe I shouldn't have done that.”

Anyway, in D&D and elsewhere there is this concept of character “alignment”. I never played D&D, sad to say, but am as familiar with some of the alignments as I used to be astrology. Categorizing, fitting things into labels that I myself resent nevertheless brings me some kind of comfort I guess. I like pattern matching.

I was thinking about my own character alignment, which I would either admit or claim proudly to be “chaotic good”. There are 9 alignments in this system, and you can picture a 3x3 grid. Across the top of the grid or the X axis you have “lawful”, “neutral”, and “chaotic”. Down the Y axis you have “good”, “neutral”, and “evil”. The most classic examples of some of these would be Captain America as “lawful good” or Donald Trump as “chaotic evil”. “Lawful evil” is an interesting and dangerous alignment. Russ Vogt, our current head of mgmt and budget in DC is easily a lawful evil character. He's spent his life learning the system he wants to destroy, and now he's in the seat to do so. What a prick. Some people will do anything but talk to a therapist.

Anyway, I find myself at the opposite corner of the grid from him. The reason this is on my mind this morning is because chaotic good is a tough row to hoe in corporate America. Corporate America and end stage capitalism are pretty simple – whoever is seen as generating the most profits for the investors right now is given more. A “short term gain, long term loss” is someone else's problem. Job hopping is rewarded, and long term efforts are generally ignored. Whatever you can produce now, regardless of value, is better than whatever you might produce next year, regardless of value. If you are a human who doesn't like job hopping and who prefers to think systemically, holistically, and longer term, then you have to steal your fun where you can. You have to keep an eye on the short term flashy things in order to buy a job long enough to see through the longer term things.

See, the thing is, it's time for me to have a point about my own alignment and how it fits into corporate culture. I don't have one.

This is strictly for me to write some thoughts down that I can refer back to later, and hopefully begin to rebuild some of the facility and fondness for writing that I used to have. Writing and thinking used to cost so much time and effort, and now that it's perceived as being free it feels more valuable than ever. So that's my point – to be human, and incomplete.

AI is good at writing code. The code it writes is a distillation of essentially every line of code ever written. Maybe a “generalization” is a better word. It is an average. The average of the world's code is generally better than what I'd be able to write on my own, and certainly it would take me a lot more time.

What I'm think about this morning though is that the code it writes is not inspired, and that's ok for me. I get inspired by solving problems at work, and this tool is kind of a game changer for me but I completely understand that the craft of writing code was what drew a lot of developers into the field over the last several decades.

Can your inspiration move upstream? To the system you're trying to build to solve the problem you're trying to solve? That tends to be where mine lives, and that's why I don't mourn the passing of a craft that I never felt that good at.

I'm a simple man. I like to tinker with things.

Recently we received a long awaited upgrade – I'm talking decades – to our home internet situation. We had been on DSL provided by a well known telco, one so hated on that they rebranded themselves a few years ago without bothering to also improve the service. Their DSL was the fastest and most reliable service we could obtain where we live, but it was by no means fast, nor reliable.

So, after a decade of calling a local fiber provider to request a service extension all the way out here, and then another couple of years of watching them slowly extend service all the way out here, one day in Early November we got a 1gb symmetrical fiber connection to our house. It's great. I never notice it, and that is all I want from the internet line.

I had never really bothered upgrading the local network situation, because the service to the house was so poor that it would be wasted effort but now, that is no longer true. So, I've been upgrading...

After a mis-start with an Open WRT router, I eventually just did the prosumer thing and bought a couple of Eero access points, because our fiber provider gave us an Eero gateway with the service and I figured I'd just go with it. I don't like that I am disallowed from changing say, the DHCP server that would let me set the pihole as local DHCP and block ads in the entire house, but at least it lets me set the pihole as the DNS server for the house. It's close enough, and does the job.

I bought an outdoor Eero unit to cover the yard and garage area where I'm working most of the spring and summer. The coverage was usually fine on the porch, but only a little way out into the yard it would fall off pretty hard. Now I have solid coverage all the way out to the garage, so what to do with that?

Smart home stuff

I was introduced to Home Assistant at my last job with Platform.sh. A couple of my colleagues ran HA and eventually I got curious. I'd pretty been anti-smart-home because my sense was that it was largely a ploy by big tech corporations that I mostly hate but still work with daily to further invade our homes and our privacy. Why else would Google buy Nest, and Amazon buy the Roomba company except to use them as data vacuums for the physical infrastructure that is your house? Yes, Amazon owns Eero, I know.

Anyway, HA is an open source smart home thing, so I figured “what the heck” and spent a little bit learning my way around it, albeit laboriously. This was maybe 18 months ago. The house's DSL finally died and we moved to a T Mobile cellular hotspot for the whole house in Sept-Nov while we waited for the fiber to be lit up, and the point is that I baiscally had to tear down the local network stuff that I'd set up before that because the T Mobile thing was seriously locked down. No way to use it except as the only wifi broadcaster, no way to run ethernet to a switch, for example.

So – come November and fiber to the house I decided to rebuild everything from scratch. This second time I had a much stronger mental model for how HA actually worked, which helped a lot. The other thing that helped a lot was installed Claude Code on the Raspberry Pi 4 that I was running HA on. I wanted a slightly more involved than vanilla HA setup, and so I was able to chat through the setup in a couple of hours a day with Claude. The setup is currently:

• HA running in Docker on pifour.local
• Metabase (for dashboarding) also running on pifour.local
• ESPHome also running on pifour.local
• Postgres for the HA backend, rather than the default SQLite database. I wanted this for making it easier to use with Metabase, which also use the Postgres instance as its backend as well.
  • After a month or so of running all of this on the R Pi 4, I moved Postgres to a R Pi 5 that I have attached to the same switch in my office. Claude helped me set up replication to copying the data to the new instance, and then cutting it over to pifive.local as the primary with no data loss.
    

This is all stuff I know how to do conceptually, but the time spent on googling and trial and erroring on it would've taken me much longer than my patience for getting to the end of it. Claude helped me do that Postgres migration to pifive.local in about ten minutes.

Sidenote – we had a power outage last week and not everything came back up as planned, because Postgres was on another machine and the startup dependency ordering in a little Docker compose file couldn't work across to another machine. Chatting with Claude about whether or not this was the problem that Kubernetes solves helped me understand K8s a little better – a tech I've mostly kept my hands off of in the last decade – and also to realize that it was total overkill for my little 2 host setup.

The real fun

Ok, so, last little bit on this for today – I like data and maybe in some other post someday I'll go into detail about why I'm so obsessed with electricity usage and heat pumps and other eco-tech. If I were going to drop technology right now, I would go into HVAC and insulation, just sayin.

Anyway, my first time around I also bought one of those whole house electricity usage monitoring setups, the kind that you attach to each circuit in your breaker panel. I made a huge mess the first time because so many wires, but this time I went slower and much tidier. You can see that chart in this whole house dashboard that I've been working on for the last few weeks.

So I’m a bit of a geek, as it turns out, and one of the things that I’ve really been enjoying this year (in fact, having more fun than in my entire technical career) is using all this AI coding assistant business. What we professionals call LLMs.

It’s kind of a funny story how I came around to it. I’ve been idly tinkering with it for the last couple of years. I was a GitHub Copilot user, and I’ve long had this idea for an open source project that I’ve tinkered with off and on over the years. But I just don’t quite have the time and frankly never really had the coding chops to build the entire thing from scratch.

So when Copilot came along sometime in 2023, I was like cool, this actually might be just what I need. I know what I want to get done. I just don’t know the ins and outs of the Boto3 library for interacting with an S3 bucket with Python. If this thing can help me with that part of it, and I know what I’m trying to do with that S3 bucket, then great. And it actually was pretty good for that. But it was still a lot of writing code, a lot of my time. Also, the autocomplete functionality was just kind of annoying, frankly. It would try to complete entire sections of code which was really kind of distracting. I tried it out for a few weeks and remained a ten dollar a month subscriber for over a year before finally shutting it off sometime in 2024.

That was kind of the extent of my interaction with any of this AI stuff since ChatGPT made a big splash in late 2022.

---

Then I started a new job earlier this year.

I had been at my old job for nine years, so basically I had all the context and felt like I knew everything I needed to know to do my job at that place. But landing in a new org was quite a head fuck, honestly. I was in a management role with a lot of direction to set and context to inhale. There were things to read, things to write. It was an incredible amount of work, and I constnatly felt behind the 8 ball trying to get caught up.

Sometime in May, before going on vacation, I needed to write a multi-page strategy document to help communicate the picture in my head for one of the teams I was managing. I kicked this thing around for a week. In a moment of weakness I used Google Gemini to help me clean up a couple of paragraphs, but I just really couldn’t get it in the shape that I wanted it. Finally, after about a week and with the Friday deadline looming, I tried Claude out for the first time. I fed in my several pages of strategy that I just wasn’t happy with and asked it for a critical review. It said well yeah, there’s plenty of gold in here, but the narrative structure’s just a little bit off. It made some suggestions for how I could tighten up the prose. I said okay, well can you rewrite this thing for me and incorporate those suggestions? It did, and I shipped the data strategy about 10 minutes later.

Needless to say, a light bulb went on in my head.

That was about 8 months ago. About four to six weeks after that episode, I was using Claude Desktop pretty much exclusively. Around that same time, they released Claude Code to their flat-rate subscriber tiers. I was paying, I think, $10 or $20/month at that point. I tried that out and converted to a higher tier later that day because it truly felt like the thing that I was looking for all along. I wasn’t looking for better autocomplete; I was looking for a better rubber duck that understood how to write code and understood the ins and outs, could read documentation and spit out working code infinitely faster than I could, but still needed a very firm guiding hand.

I’ve been having a lot of fun over the last 8 months working with that and building things in a matter of hours that would have taken me weeks before. Granted, I’m writing a ton more code than I would have written by hand, but honestly, who’s going to be reading code in the future? We don’t read machine code now; nobody’s going to be reading Python 5 years from now. That’s kind of the mindset I’ve been adopting.

It’s a lot of fun.

I was recently inspired by a buddy of mine to re-launch this blog for probably the 18th or 25th time on a new framework. This time on a little one called Write Freely, which is just a little Go binary that's hosted here on a Raspberry Pi in my office. That's fun. That's all. I'm mostly just testing and making sure that posting stuff actually works this time, because I've broken it a few times over the years.

The total setup is a raspberry pi in my office, tunneled via Tailscale to a Digital Ocean droplet that's what you're interacting with. It handles the TLS certificate and then just hands all requests back to the Pi, which is something that's imminently feasible now that we've upgraded to a fiber internet connection. Been upgrading everything, everywhere, all at once for the last month or so.

I've been backing on a side project lately to try and make open sources some of the bones of a FinOps visibility tool. You can find the FinOpsPod episode I recorded on the topic recently here. Well now that that's out, I've been properly motivated to ship and while AWS is done enough for now, I have been wrangling the Azure side of things over the weekend. This is what I learned in the last 72 hours.

Azure Blob Storage download progress with TQDM

I searched the internet high and low for how to handle this. AWS makes it fairly easy with the Callback argument you can pass when downloading an object from S3. I guess Azure's version is more recent and it goes like this:

def download_object(self, blob_name, destination_path):
    blob_client = self.container_client.get_blob_client(blob_name)
    filesize = blob_client.get_blob_properties().size
    # just in case the destination path doesn't exist
    os.makedirs(os.path.dirname(destination_path), exist_ok=True)
    with open(destination_path, "wb") as file, tqdm(
        total=filesize,
        unit="B",
        unit_scale=True,
        desc='whatever helpful file description you wish',
        colour="green",
    ) as t:
        bytes_read = 0
        # the progress_hook calls with 2 args - the bytes downloaded so far and the total
        # bytes of the object.  t.update wants the bytes read in that iteration, so we have to
        # make that happen by keeping track of what the total was as of the previous
        # iteration.
        def update_progress(bytes_amount, *args):
            nonlocal bytes_read
            t.update(bytes_amount - bytes_read)
            bytes_read = bytes_amount

        blob_data = blob_client.download_blob(progress_hook=update_progress)
        file.write(blob_data.readall())
        t.close()

In the API docs for the download_blob function you can find the progress_hook kwarg. It isn't called as often as its AWS counterpart so the progress bar isn't nearly as fine-grained, but it's better than nothing in my opinion. The whole thing in general requires more wrangling than the AWS version, but I learned quite a lot in the process.

DuckDB, the ultimate dataeng swiss army knife?

One helpful thing that AWS does in their billing data export is to include a metadata file with each night's export that tells us facts about the export in general. Things like

• the timestamp that the export was generated
• where you can find the associated billing data files
• a unique ID on that particular version of the export and most helpfully
• a list of columns and their datatypes in that export.

For this side project I'm using Clickhouse as the backend warehouse. It's really fun to run huge queries on a huge dataset and have them come back in what feels like 100ms, so I'm a rather big fan of Clickhouse at this point though I'm only just getting to know it. There are fussy things, too. Things like its CSV importing, which is ... not super friendly. Here's an example:

Azure's billing exports generate with a Date field that tells you the date of the charge/line item. For some reason, even though my employer is a French company and our bill is in euro, all of the date fields in this bill come across with the US date format – MM/DD/YYYY. After exhaustive searching, I did find a clue in the Clickhouse documentation that it could parse US style dateTime strings, but I cannot find that piece of documentation again AND it was only available after you'd gotten the data into the warehouse (presumably as a String). I want the thing stored as a date to begin with so I started to wonder if I could grab DuckDB and use it to parse this stupid Date column for me correctly.

The answer is yes. DuckDB is also a pretty cool piece of gear and so I'm playing with both of these open-source columnar things at the moment. One thing that Duck has gone out of their way with is making it super easy to ingest data, and to specify all the little weird things that can go wrong in their extremely generous default CSV importer, things like “hey, the dateformat should look like this – {strptime string}”. Super cool and works like a charm, so now I have this CSV in memory as a DuckDB table. What else can I do with it?

Well, why spit it back out as CSV, how about spit it back out as Parquet? Clickhouse will have a much easier time reading a Parquet file as it comes along with all the column names and datatypes, so that's what I'm doing. So, I have this function that downloads all the data_files for a given billing export and for the sake of brevity I'll put it here in its current, non-optimized form:

def download_datafiles(self, data_files: List[str]):
    local_files = []
    # this downloads each of the CSV files and puts them in a local
    # tmp directory
    for data_file in data_files:
        destination_path = f"tmp/{data_file}"
        print(f"Downloading to {destination_path}")
        self.storage_client.download_object(data_file, destination_path)
        local_files.append(destination_path)
    dirname = os.path.dirname(local_files[0])
    con = duckdb.connect()
    # Here we convert the csv to Parquet, because DuckDB is excellent with
    # parsing CSV and Clickhouse is a bit fussy in this regard.  The Azure
    # files come over with dates in the format MM/DD/YYYY, which DuckDB
    # can be made to deal with, but Clickhouse cannot.

    # moreover, Duck can grab any number of CSV files in the target directory and
    # merge them all together for me.  This allows me to generate a single Parquet
    # file from all the CSV files in the directory.  Given that Azure doesn't even
    # gzip these files, this turns 500MB of CSV into 20MB of Parquet.  Not bad.
    con.sql(
        f"""CREATE TABLE azure_tmp AS SELECT * FROM read_csv('{dirname}/*.csv', 
            header = true, 
            dateformat = '%m/%d/%Y'
        )"""
    )
    con.sql(
        f"COPY (SELECT * FROM azure_tmp) TO 'tmp/azure-tmp.parquet' (FORMAT 'parquet')"
    )
    con.close()
    # yes, we do two things in the function right now, it's ok.  We'll refactor
    # and use the "parse the columns and datatypes out of this parquet table" probably
    # all over the place.
    return "tmp/azure-tmp.parquet"

#data #data-engineering #azure #finops

Hey there, just wanted to leave a signpost for you. My usecases lately have been something like -

Computing a BIG table, with a lot of math in it, over a LOT of rows of data, and then joining in other data to enrich the primary set. Specifically, this is container usage data, which I'm attempting to blend with our AWS bill to arrive at something like “cost per container” per time period.

I don't want to have to rebuild this table every day because most of the data is static once it shows up in the warehouse. An incremental strategy would be perfect BUT, some of this data arrives late, which means that if I do the standard DBT routine of

WHERE timestamp > (SELECT MAX(timestamp) from {{this}})

then I will have gaps. Indeed, I have gaps. I haven't rolled out any reporting on this table, or made any announcements because I felt a disturbance in the force, confirmed by some light analysis this morning.

I've recently discovered a new DBT hammer in the incremental_strategy parameter for incrementally built tables, and specifically the insert_overwrite option. From the DBT docs:

The insert_overwrite strategy generates a merge statement that replaces entire partitions in the destination table.

In short I can just always recompute yesterday and today, or the last 7 days, or whatever full partitions-worth of data I want. Yes, I'm recomputing more than I strictly need to, but it assures me that there will be no gaps in the results.

This operation seems pretty foolproof so far, check it out.

#analytics #DBT

Supposing a hypothetical organization that sold a product whose feature set and COGS closely followed a typical CSP like Amazon Web Services. That organization allows its customers to change products at will but must manually invoice a significant percentage of those sellables, therefore it needs a robust system to track changes to those sellables and ensure that they are properly charged at each turn of the billing cycle.

I'm picturing reporting format that reports on 2 different types of metrics -

• Accruing (non-temporal)
• Static (temporal)

Accruing metrics are easy, they're things like outgoing bandwidth. These are capped monthly and overages should be trued up, therefore 2 measurements could be helpful on these -

• Month to date sum (this will end up being your billable, since the bounds of the billing cycle are likely set at the calendar month)
• Rolling 30 day average (typical month's usage, helps you notice a customer who is tracking above what you sold them)

Static metrics require a bit more understanding. These are things like CPU cores in a given VM that you're selling. The tin says “8 CPUs” and gives you a monthly rate for those 8 CPUs but you're allowed to upsize that 8 core machine any time you want. Those 8 cores might become 16 for a week, then back to 8. That means you're charging for neither the 8 core machine nor the 16 core machine, but a blend of both.

This is what I mean by “temporal”, you have to generate a time component, divide your 8 or 16 cores into that time component, prorate the usage by that time component, and ultimately arrive at a piece of usage that accrues just like the other.

Given the example of 8 cores to 16 cores and using a 30 day month (720 hours) we get something like this:

You're actually charging for CPU/Hours, firstly. If an 8 core machine is $720/month ($1/hour) and a 16 core machine is $1440 ($2/hour) then your hourly CPU rate is $.0125/hour. This makes it very simple to track (and bill!!) the changes to your sellables that your customers are using.

The metrics you might want to watch on these types of sellables/COGS are almost the opposite of the accruing type:

• Month to date average
• Rolling 30 days average

The monthly vs. the 30 day average would tell you if they are tracking above or below recent historical averages. It would be trivial to compare the two and throw an alert if the month to date or shorter term rolling average is trending significant above the 30 day average.

Note: I'm on vacation and just want to remember some stuff for when I get back so don't dock me on this, I'm just spitballing

#business #analytics

First day of my job at my current employer (almost 7 years ago now) I crack open the company handbook to start onboarding. I remember it saying something like “Reservations are the lifeblood of this company” and thinking “wow, what does that mean?”

I'd had a job prior to this one that had some Stuff on AWS so I was familiar with the concept – something something pay upfront get reduced prices – but it as far from the lifeblood of the company. It was something the IT head took care of, and he didn't seem that pleased to do it either. So, six years after that I find myself in charge of FinOps here and it has a lot to do with reservations. Indeed, reservations are the main lever that you have to pull on the job. Let's talk bout it…

What are reservations?

I recently heard reservations described as a “forward contract”, which Investopedia (one of the most useful resources in this leg of my career) describes thusly

A forward contract is a customized contract between two parties to buy or sell an asset at a specified price on a future date.

The promise of the cloud is that you, developer, can push a button and spin up a VM or any number of other network-connected computing resources that you didn't have to ask IT for. It's why the default pricing model for these resources is called “On Demand”. AWS invented this idea several years ago that if you committed to running that resource for a year or more, you could receive a reduced price. You could pay for the year either entirely upfront to receive the deepest discount, entirely over time as a monthly payment for a shallower discount, or somewhere in between – the “partial upfront”, which mixes an upfront payment with a monthly payment for those resources.

At this point, you might be starting to learn concepts like “amortization” and “the time cost of money” to distinguish why you would choose one of these, but if not I suggest looking them up.

Uh, ok

There is a balancing act in place with reservations. Obviously you want to receive those lower rates wherever possible, but your resource usage might be, indeed probably is, rather variable. “Elastic”. Suppose you overpurchase a reservation, all upfront so that the entire year is paid for, but then the resource is turned off after 6 months. So my latest analog for what FinOps is largely about is a double ended inventory job. You have an inventory of resources to cover with reservations and you have an inventory of the reservations themselves. One can be created or turned off in an instant, the other lives for 1 or 3 years.

#business #finops

I've been kicking around this thought for a year or so now – to the outsider a career in data looks like a technical path. The data practitioner learns SQL, how to query data stored in a database somewhere using SQL, and if you know enough SQL you can answer any question whose artifacts are stored in that database somewhere.

The reality is that SQL is the very last mile. SQL is code, and so it looks to the non-practitioner like the act of creation, like code written in any imperative language creates motion and process and a webapp or piece of automation that didn't exist before. SQL does not create. SQL encapsulates that which already exists as a business process.

SQL is a contract. SQL puts business conditions and processes into code. If the business processes are ill-defined, then the SQL that has to be written to handle all the various cases will sprawl. (Most business processes are ill-defined as it turns out, made up in a time of need by a human, and probably one who doesn't spend their day thinking about data modeling.) If the business process is well-defined, but the SQL author's understanding of it is wrong or incomplete, then you'll end up with a poorly written contract that spits out wrong or incomplete answers.

That's what makes Data the hard part, because to write that contract down always requires the author to have spent time reverse-engineering the business process. I view this as an inherent good for the business as a whole – it forces the business to reckon with itself and to better define how it operates. The road to get there is tough though and in my experience it's often the data analyst who is actually pulling the cart.

#analytics #business #databases