Andrew Shay's Blog and Digital Garden

Note Taking

2026-08-09

My Note Taking System

There are many note taking systems and information organization tools and tips out there. I'm frequently changing my methods for what is most productive for me. I've also come across great ideas from other folks that I've tweaked and integrated into my work flow. Some I keep, and some I've dropped.

The Starting Points

Johnny Decimal & PARA & Never Ending Txt

There were a few specific ideas that I have come across over time that I really like and have heavily influenced my work flow.

1. Johnny Decimal - Organizing into a max of 10 groups
2. PARA - Project Area Resource Archive
3. Never Ending Txt File - One long text file that contains everything you do, each day.

Obsidian

The best note taking software that I've come across is Obsidian. It seems to be very popular in the software industry.
It is extensible with a large number of plug-ins available.

My Setup

Top Level Organization

In Obsidian I have 10 folders which are basically the following:

00 Journal/
01 Claude/
02 Project A/
03 Project B/
04 Team A/
05 Team B/
... 
10 Helpers/
CLAUDE.md

The Notes

Each folder contains notes. Sometimes named with a starting number (e.g. 01 02 03 etc) to force order.
Each note has headings like Links, Meetings, Notes.
Each meeting gets a sub entry. Notes are for generic information.
Following a similar structure for each note makes it easy to find and reference information.

00 Journal

Journal is a special folder with a unique setup.
I have the !Log note which starts with daily reminders on things I should check on.
Then a TODO section with bullet points that are on my TODO list to get done "soon". Really just things I do not want to forget about.
A list for "Sync" which is what I want to mention in the next team sync.

Then finally a long list of notes for each day. Notes, chat links, important information.
This idea has evolved for me over time. At first, I was diligently documenting every day. Every chat convo and every Jira ticket I made.
This was kind of helpful, but over time it just felt like busy work and did not add much usefulness. I was not coming back that often to this list to find something I needed.
Now I might have some entries for a day, but it's rare. I think my usage with Claude Code has made this even less useful.

In 00 Journal, I was also storing notes that I considered "Artifacts". These were snippets of whatever I needed, but did not really want to categorize. I just wanted to capture it so that I could search it later. Like a stacktrace and some info on it. That way if I ever saw it again in the future, I could reference the existing note.
I use alphabetical ordering to keep the latest on top. So my oldest note is a9999 and my newest note would be a9910.

This is useful, but I've also stopped collecting these artifacts. I guess it's not useful enough! Now that I think about it, a Claude integration to automatically create these notes with data from my clipboard might be useful. 🤔

Claude Code

My latest integration is with Claude Code and I do find this to be very helpful.

I have a CLAUDE.md file in my root Obsidian directory that tells Claude its only job is to search my notes and not use plugins or external sources.
I can open a Claude Code session in this directory and have Claude find notes for me, and even open them in Obsidian! This has been great.

I did not want another terminal tab open, so I had Claude vibe code an Obsidian extension that opens a new tab that has a terminal window inside the Obsidian page. I execute a bash alias that starts Claude code in that directory.
Now I have a really convenient way to keep Claude open in Obsidian.

In the CLAUDE.md I also explained to Claude that when I type "mem: something", that it should store that text in the !Mem note with the current date and time.
I am really liking this because it's so easy for me to just quickly drop notes and links into a file quickly. And Claude can easily search it back up when I need it. It's like a scratch pad.

Having Claude integration is really useful because I can easily store data in Obsidian without thinking about how to organize it. Claude can just find it.
I'm definitely going to keep expanding my Claude usage in Obsidian.

Helpers

The next folder that I'll mention is my Helpers folder. Each note in this folder is usually a code snippet or a "problem and solution". Like an error message, and a command that will resolve it.
Another great way to store this semi-random stuff that is easy to search up later.

Other Note Taking Tools

There are lots of other note taking tools out there that are also nice.

1. Tiddly Wiki - A very cool single file Javascript based "app".
2. Zim - Desktop wiki
3. Apple Notes - The default Apple Notes software is also really good. It's pretty close to Obsidian if you just want basic note taking. When I tried it last, it did not have the ability to link to other notes though.

Biggest Tips

1. Just get the data into a note. It can be searched up later
2. Limit the number of folders to force organization and limit sprawl
3. A lot of information can go into one note. For instance for small projects, just put it all in one note.
4. Use software that you find useful

BoxDiff: New project for detecting changes on Windows

2025-04-02

I created, BoxDiff, a simple Python script for detecting changes on Windows. It shows you what was added, removed, or changed.
There are no dependencies, and it's also extendable with a single Python module.

A small subset of the data that is collected

Devices
Services
Installed Programs
Startup Programs
Users
Tasks
Certificates

Download from GitHub

Vendor Base64 Project for Rust

2024-07-01

What is Vendor Base64

rust_vendor_base64 is my implementation of base64, in rust, that is meant to be vendored in rust projects.
It is not available on crates.io to be installed.

I created it because it's something so simple that I think it should be included in the rust stdlib.
And when the base64 crate broke their API, I decided I should implement it and have a module that others could vendor without risking the API breaking.

Auto Magor: Python Program for Generating Print Magazines

2024-02-29

What is Auto Magor

Auto Magor is a simple Python program that I created that can generate 3 column print magazines from text files and images.
The user just has to create simple text files with the article body, metadata about it, and an article image.
That is then turned into a 3 column magazine, as a pdf, that can be printed at home.

I created it because I have been trying to use my phone less at night before going to sleep. I don't like reading on my phone, and my eBook is alright, but the blue light still isn't great.
Reading physical books and magazines is really nice though. However, there is a still a lot of online content that I'd like to read offline.
I thought, if only I could take that online content and make a magazine out of it 🤔
Making it look like a print magazine (instead of just printing webpages outright) is just more fun.

If I want it, and it can be built, I must build it.

How Auto Magor Works

Auto Magor is surprisingly simple, in general.

Articles are stored in simple text files with the following format.

title:The Title of This Article
subtitle:A subtitle for this Article
author:Foo Bar
date:January 1, 2024
image:article_01.png
source: Text that will appear after the date
body:
The first sentence of the first paragraph. The second sentence of the first paragraph.

The second paragraph is here. Another sentence in the second paragraph.

The Pillow library is used to generate each page.
Each column of text is drawn onto the image once the column is full of text.
A column of text is generated by appending one word at a time and calculating if it fits on the current line.
Once the current line is full, the next line is started. This continues until the max column height is reached, and then the next column is started.
I use the method draw.multiline_textbbox() to get the dimensions before having to actually draw the text.
This way of generating columns makes Auto Magor very slow though.

It's simple, but along the way I ran into a few things that were a little tricky at first...

The first thing I hit, was that some of the last lines of a paragraph were unexpectedly reaching the max column height, mid-sentence.
I realized that it was caused by "low hanging" characters like pqgy,).
Before writing a new line, I check if it will be able to fit.
At first, I was (re)checking the height as each word was added. This was just for simplicity. So, once a word with a low hanging character was attempted, it would push to the next line.
I had to change it to test the new line height one time, and with the low hanging characters instead.

I also had to start tracking when I was at the start of a new column.
Blank lines between paragraphs in the article files were pushing the column start down.
New columns were also being started with a space since I was appending with "a space" followed by the next word.

How to Make Auto Magor Faster

Since I won't be running Auto Magor very often, I prefer code simplicity to execution speed.
But there are some things that I assume could make Auto Magor faster.

I get the width of the column by appending the word, then calculating the new column width.
But I do this after every word. So, it's constantly recalculating the line. i.e. every word that has already been applied.
I think I could maintain a width variable manually, get the width of each word by itself, and increment it until the max width is reached.

If the above works, a cache of the widths for the most frequent words could be generated, and then maintained across runs for a given font and size.

Download Auto Magor

Making Transmitic Faster...By Doing Less!

2023-11-12

What is Transmitic & The Prototype Phase

Transmitic is an encrypted, p2p, file transfer and sharing program. I created it because, while there are many file transfer programs out there, there was nothing quite like Transmitic.
The main difference is in the UI. You add users you want to share with and which files and folders each user can download from you. Then users can download those files whenever they want. Completely p2p.

A few years ago, I wanted to learn a new programming language and decided on Rust. I then decided my first project with Rust should be Transmitic.
Learning Rust was quite difficult at first, and when combined with creating a new p2p application, it was a lot to try and handle after work.
Therefore, I could not try to optimize, cache, or really architect anything.
In order to not get overwhelmed by the project, I decided that I would just learn, and code, little bits at a time, without being concerned about code quality (except for the crypto) or performance.

This worked out well because I have been able to get Transmitic to a very functional (yet still Beta) state. It now includes all the primary features I wanted it to have.
There is still much more I want to do though.

Until very recently, it was really more in an alpha/prototype state. There was wasted bandwidth and things that could be cached, but weren't.
I will discuss some of the enhancements that helped make Transmitic faster and use less bandwidth, which can be summarized as "do less!"

None of this is novel. It's all obvious, and if you're able to truly focus on a project and plan it, you won't hit these.
I just couldn't think about it at the time. Once the feature worked, ship it!

Overview of Features

A quick overview of the features that will be discussed.

The protocol is simple. It sends u16 messages to instruct the action to be performed, with a u32 payload size, followed by the payload itself.
File Listings are JSON strings. They contain all file paths, and their sizes, that are shared with an individual user.
To determine when a file download is complete, the client sends the file path and current file size on disk to the server. The server sends more chunks of data or a download complete message.
"Reverse Connection" is a feature, that when enabled, Transmitic will periodically connect to each user (outgoing) to establish a connection for sharing. This feature is for users with networking limitations that cannot accept incoming connections.

Cache File Listings

When a user refreshes files that are shared with them, they download JSON data containing the file paths shared with them and the file sizes.
This is the "File Listing".
Initially, when a user started downloading files, the file listing was requested again. This ensured they had the latest, valid, list of files shared with them.
The client could then determine locally if the file was invalid or not (of course, the server would still reject any invalid files that were requested).
However, this is wasted bandwidth. Especially if the file listing was large, like 50MB. A user downloads 50MB worth of JSON to see what's available, and once they start downloading, they immediately download 50MB again.
Wasteful!

The same scenario occurred with the Reverse Connection feature.
A user with Reverse Connection enabled, periodically connects to everyone they share with, and "hands off" that connection to allow file listings and downloads to occur.
Because the File Listing was not cached, every Reverse Connection would trigger the file listing to be downloaded.
Therefore, if you were sharing with anyone who used Reverse Connection, they would continually trigger your client to download the File Listing even if you already had the latest File Listing!
Wasteful!

Let's cache it!

Reducing this waste is simple. We just need to cache the File Listing, ensure the appropriate threads have access to it, and determine when we need a new copy.

When a client starts downloading files, instead of requesting a new File Listing immediately, first it checks if it already has one. If it does, it attempts to download the file.
Now, it will only get a new File Listing if the server responds with the file choice being "Invalid" i.e. Not shared with you.

The Reverse Connection logic now maintains a list of users who have the latest File Listing. This list is reset when the user changes which files they're sharing.
When the Reverse Connection is established, there is now a new message that can be sent, MSG_REVERSE_NO_LIST.
This tells the client a Reverse Connection is being established, but the server thinks you have the latest File Listing.
The client does not need to request a new one.

If the Client Can Do It Alone, Do It Alone

When a user downloads a directory, the client loops through every sub file in the File Listing and requests it from the server.
If the download is interrupted, it can continue where it left off.
It would NOT redownload every file, but it would start from the first file in the directory and loop through them all again.
Originally, the logic was simple. The next file is pulled from the file listing, check its size on disk, and send it to the server which would respond.
This would happen for files that were already finished. The client would tell the server the current size and the server would respond, download finished.
Wasteful!

We have the File Listing locally, which already contains the expected file sizes!
Now, when the client loops through the File Listing again, if the file size on disk matches the File Listing, it just moves onto the next file.
The client does not contact the server for finished files anymore!

After that change, a bug was introduced. I noticed it when transferring some test directories.
0 byte files.
When getting ready to process a file for download, the client sets the current size on disk, but if the file does not exist, it sets it to 0.
That, combined with "if the size on disk matches the File Listing, move on to the next file", meant 0 byte files were not being created anymore. Oops!
That was an easy fix though. If the File Listing says a file's size is 0 bytes, the client just creates the file locally. There is no need to ask the server anything!

Redundant Server Messages

Originally, the server would send a MSG_FILE_CHUNK message which contains a chunk of file as the payload.
When the file completed, the server would then send a MSG_FILE_FINISHED message with no payload.
This means every file download ended with an extra message, MSG_FILE_FINISHED. And it has no data...
Wasteful!

So, I extended MSG_FILE_FINISHED to also contain a payload.
Now, the last chunk of file would be MSG_FILE_FINISHED instead of MSG_FILE_CHUNK.
I was able to remove the pointless extra message of MSG_FILE_FINISHED with no payload.

I also uncovered another wasteful message.
The code looked like this

bytes_read = -1
while bytes_read != 0:
    bytes_read = read_file_into_buffer(buf)
    send_to_client(MSG_FILE_CHUNK, buf)

The intent is to keep sending chunks until nothing left was read from the file, i.e. Everything has been sent.
However, while bytes_read != 0 means loop UNTIL bytes_read == 0, which means we need a pass through the loop that has read nothing.
This causes the server to send a MSG_FILE_CHUNK with a payload of nothing!
The server tells the client, "Here's a 0 byte chunk of your download."
Wasteful!
So, I refactored the loop and the inner logic to not send this useless message.

In Summary

Cache data where you can so you do not have to rerequest it
Look for useless communication. i.e. Data with no payloads
If you can operate with only local data, do not contact the server

It's always fun to make these optimizations! Big or Small.
Not worrying about them from the beginning allowed me to keep working and deliver value to users.
Thinking about everything from the beginning would have been too much, burned me out, and I would have given up. I was already close to giving up as is, but I'm so glad I didn't!

https://transmitic.net/
https://github.com/transmitic/transmitic/

Transmitic: Encrypted p2p file transfer and sharing. Built with Rust.

2022-08-09

I created, Transmitic, desktop software for encrypted, peer to peer, file transfer and sharing.
It's built with Rust and the GUI is created with sciter.
I'm working to make it as easy as possible to use.

It automatically generates an x25519 public signing key. Users must then exchange their public keys and IPs out of band.
They can choose files and folders on their computer to share, and select which users they want to share those files with.
Connections are encrypted with AES-GCM.

Users can download the files, and the downloads will resume if it gets disconnected.
This means it's easy to transfer huge files (no file size limits) without needing to compress them, break them into smaller files etc., or worry about disconnects while uploading.

There's no third party servers involved and no cloud storage, so there is no cost associated with transferring, other than each user's bandwidth.

Nothing else quite like Transmitic

I've always wanted software like Transmitic, especially for transferring 100GB files, but could never find anything quite like it.
I would think to myself "I just want to send these files directly from this computer to another."
Why is that so difficult and not secure?

There is BitTorrent, but that's difficult to use (in comparison), no out of the box encryption, no easy way to select files to share with specific users.

There is file syncing software, but that syncs whole folders, is harder to quickly add users, and harder to choose what to share with them (more like all or nothing).

Transmitic
Download from GitHub

Demo

Python How to Replace Multiple Regexes Without Overlaps

2022-08-06

When I was building a new blog generator for this site, I did not want to use any third party packages, therefore I needed some simple markdown like parsing.
I need to convert markdown in a line, to html.

Problem: Overlapping Regex Replacements

I started with multiple regexes using re.search(), and doing a simple replacement on the each line.
However, I was running each regex on the line, one after the other, which performed multiple replacements on the substrings which was bad.
Let's start with these two:
[some url](https://andrewshay.me/__somewebsite__/). This should get converted to an html a href with text.
__text__. This should get converted to italics.

Notice that the url text contains the regex for italics. What was happening, first the markdown url would be converted to html a, and then part of that url would get html tags for italics injected. Bad.
What I needed to do was apply all the regexes, but not overlap them. Once one is used, do not apply another.

Mega Regex with re.findall()

I have no idea if this is the correct way to handle this, or if there a term for this, but I couldn't find it online.
This is easiest to use by using the debugger and looking at the output of re.findall()

The process is:

Create variables for each of your patterns (for readability).
If a regex has multiple groups (e.g. my img pattern), wrap the entire regex as a group too.
Create a new variable that ORs them together with |.
Execute re.findall(combined_pattern, line)
Each index in the match objects is the match group(s) for the patterns, in order, from the combined regex.
My img pattern will always be index 0: the entire img markdown. index 1: the alt text. index 2: the img src.
I can trim off the markdown characters from 1 and 2, form my html string, then on the line, do a replace of 0 with my new html.
And the rest of the regexes will not run on the text that already matched.
Order of the patterns in your combined regex is important.

Here is the code from this blog.

img_pattern = r'((!\[.+\])(\(.+\)))'
url_pattern = r"((\[.+\])(\(.+\)))"
i_pattern = r'(__.*?__)'
strong_pattern = r'(\*\*.*\*\*)'
code_pattern = r'(`.+?`)'
# 0=img total group / 1=img text / 2=img link / 3=url group / 4=url text / 5=url link / 6=code text / 7=i text / 8=strong text / 
combine = img_pattern + r'|' + url_pattern + r'|' + code_pattern + r'|' + i_pattern + r'|' + strong_pattern
match = re.findall(combine, new_line)

if match:
    for m in match:
        if m[0]:
            img_alt = m[1][2:-1]
            img_src = m[2][1:-1]
            if img_src.startswith("http"):
                new_text = f'<img alt="{img_alt}" src="{img_src}">'
            else:
                new_text = f'<img alt="{img_alt}" src="images/{img_src}">'
            new_line = new_line.replace(m[0], new_text)

        if m[3]:
            url_text = m[4][1:-1]
            url_href = m[5][1:-1]
            new_text = f'<a href="{url_href}">{url_text}</a>'
            new_line = new_line.replace(m[3], new_text)

        if m[6]:
            code_text = m[6][1:-1]
            new_line = new_line.replace(m[6], f'<code>{code_text}</code>')

        if m[7]:
            i_text = m[7][2:-2]
            new_line = new_line.replace(m[7], f'<i>{i_text}</i>')

        if m[8]:
            strong_text = m[8][2:-2]
            new_line = new_line.replace(m[8], f'<strong>{strong_text}</strong>')

Rust Convert u32 to bytes. Convert bytes to u32.

2020-05-25

Here is how to convert u32 to bytes and then back again in Rust.

fn main() {
    let original_u32: u32 = 1048572;
    println!("{}", original_u32);
    
    let u32_as_bytes: [u8; 4] = original_u32.to_be_bytes();
    println!("{:?}", u32_as_bytes);
    
    let back_to_u32: u32 = u32::from_be_bytes(u32_as_bytes);
    println!("{}", back_to_u32);
}

/*
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0bc90209eb1df96ad76d23490d34e8be
to_be_bytes
https://doc.rust-lang.org/std/primitive.u32.html#method.to_be_bytes
from_be_bytes 
https://doc.rust-lang.org/std/primitive.u32.html#method.from_be_bytes
*/

On GitHub https://gist.github.com/Andrew-Shay/040fd84a9c557be8451e28e32784d8f4

Find Most Common Color in an Image with Python

2019-04-25

A while ago I wanted my Philips Hue light bulbs to change colors based on the most common color on my monitor.
I thought this would be cool when playing video games, which it is!

In this article I will explain a simple way for detecting the most common color in an image using Python.

Note: This article describes my method that I came up with for fun. It is not "the right way" to do this, the fastest, or most efficient. But it is simple!

Pixels

An image is made up of individual pixels. Each pixel is a color.
To find the most common color we can simply loop through every pixel, get the color, and keep a count of how many times the color appears.

I found an issue where for instance the color green and just barely slightly darker green are technically two different colors.
I couldn't see the difference between these two colors though, so I want them to be counted as a single color.
I need similar colors to be counted together because there are too many possible colors.

RGB

A color can be represented in RGB. RGB stands for RED GREEN BLUE.
An integer from 0 through 255 is used to represent R, G, and B. e.g. 255, 7, 125.
There are 16,777,216 total possible colors represented by RGB.
0 is no color (i.e. black) and 255 is all color.
0, 0, 0 would be black and 255, 255, 255 would be white.
So 0, 0, 255 would be no red, no green, and all blue.
0, 0, 254 would be just slightly darker blue, but not even perceptible.

Allowed RGB Values

What I decided to do was to create a set of "allowed" integers for R, G, and B.
I would then look at the actual integer, determine which allowed integer it was closest to, and change it to that.
This integer "snapping" technically changes the color of the pixel, but it greatly reduces the possible colors that can occur.
The pixels almost always still represent the correct color so determining the most common color is much simpler.

A decent set of evenly distributed allowed integers is [255, 223, 191, 159, 127, 95, 63, 31, 0].
This reduces our total possible color count to 729.
Let's say we have two pixels 254, 120, 5 and 250, 126, 20.
After we snap the integers we get 255, 127, 0 for both pixels.
We took two technically different colors and made them the same so they are counted together.

If you're looking for "color" you can also remove black, white and gray, from the count.

Python Script

Let's take a look at the code.
This is a simple straight forward script that can be read from top to bottom.
The Pillow library handles working with the images.
pip install pillow

import os

from PIL import Image

# Open image and get data
image_path = r"IMAGE_PATH.jpg"
img = Image.open(image_path)
width, height = img.size

# The RGB values we will "snap" to
colors = [255, 223, 191, 159, 127, 95, 63, 31, 0]

original_color_count = {}
color_count = {}
# Loop through every pixel in the image and modify it
for w in range(width):
    for h in range(height):
        current_color = img.getpixel((w, h))

        if current_color in original_color_count:
            original_color_count[current_color] += 1
        else:
            original_color_count[current_color] = 1

        r, g, b = current_color
        r_set = False
        g_set = False
        b_set = False

        #  Loop through our allowed values and find the closest value to snap to
        for i in range(len(colors)):
            color_one = colors[i]
            color_two = colors[i + 1]

            if not r_set:
                if color_one >= r >= color_two:
                    distance_one = color_one - r
                    distance_two = r - color_two
                    r = color_one if distance_one <= distance_two else color_two
                    r_set = True

            if not g_set:
                if color_one >= g >= color_two:
                    distance_one = color_one - g
                    distance_two = g - color_two
                    g = color_one if distance_one <= distance_two else color_two
                    g_set = True

            if not b_set:
                if color_one >= b >= color_two:
                    distance_one = color_one - b
                    distance_two = b - color_two
                    b = color_one if distance_one <= distance_two else color_two
                    b_set = True

            if all((r_set, g_set, b_set)):
                break

        # Set our new pixel back on the image to see the difference
        new_rgb = (r, g, b)
        img.putpixel((w, h), new_rgb)

        if new_rgb in color_count:
            color_count[new_rgb] += 1
        else:
            color_count[new_rgb] = 1

# Save our new image
filename, file_extension = os.path.splitext(image_path)
new_path = "{}_new{}".format(filename, '.png')
img.save(new_path)

# Count and sort the colors
all_colors = color_count.items()
all_colors = sorted(all_colors, key=lambda tup: tup[1], reverse=True)

# Print out the colors
print("## All Colors ##")
for i in range(10):
    print(all_colors[i])

# Remove black, white and gray
print("\n## All Colors Filtered ##")
filtered_colors = [color for color in all_colors if not color[0][0] == color[0][1] == color[0][2]]
for i in range(10):
    print(filtered_colors[i])

print("")
original_color_count = len(original_color_count)
print("Original Color Count: {}".format(original_color_count))
new_color_count = len(color_count)
print("New Color Count: {}".format(new_color_count))
color_diff = original_color_count - new_color_count
print("Reduced Color Count By: {}".format(color_diff))

Examples

Let's compare the changes using some images that I found on Unsplash.

Jungle

Photo by Jacob Plumb on Unsplash
https://unsplash.com/photos/UghHZmnJw58

Top 3 Colors
RGB: 0, 31, 0 Count: 61,167
RGB: 0, 0, 0 Count: 36,716
RGB: 31, 31, 31 Count: 23,208

Original Color Count: 62,963
New Color Count: 167
Reduced Color Count By: 62,796

Airplane Cockpit

Photo by JC Gellidon on Unsplash
https://unsplash.com/photos/7KFVkL1cV0w

Top 3 Colors
RGB: 31, 31, 31 Count: 166,090
RGB: 63, 63, 63 Count: 28,134
RGB: 255, 255, 255 Count: 4,943

Original Color Count: 21,216
New Color Count: 168
Reduced Color Count By: 21,048

Train Station

Photo by Patrick Schöpflin on Unsplash
https://unsplash.com/photos/BnlNjW3AQZ0

Top 3 Colors
RGB: 31, 31, 31 Count: 57,880
RGB: 63, 63, 63 Count: 29,510
RGB: 0, 0, 0 Count: 27,606

Original Color Count: 31,211
New Color Count: 117
Reduced Color Count By: 31,094

Summary

- Loop through every pixel
- Snap the RGB values to a smaller subset
- Count each color

Pillow Image docs https://pillow.readthedocs.io/en/stable/reference/Image.html

TDD vs BDD. Does TDD test implementation or behavior?

2020-01-01

About a year ago I wanted to learn more about good testing habits and specifically Test Driven Development (TDD) because I had heard about it so much. While reading about these topics I came across Behavior Driven Development (BDD). I came to the conclusion that there is confusion around TDD and especially when compared to BDD, however TDD is simple and a highly effective development strategy when understood correctly.
The first thing I did when trying to learn more about TDD was to of course Google it. I read the Wikipedia page, Stack Overflow questions, development/testing websites' explanations, and people's personal blogs. Someone looking up TDD as I did will quickly find many contradictions and inaccurate interpretations. Many of the inaccuracies are asserted as truth, which makes it hard to determine what is correct.

Common Statements

Listed below are the 5 most common statements that I found about TDD:

1) TDD tests implementation
2) TDD tests behavior
3) TDD means writing a test for every function
4) TDD tests implementation and BDD tests behavior
5) BDD is an improved version of TDD

I am going to go through these assertions and discuss where confusion around them comes from, why I think some are simply incorrect, and why some depend on the context of development.

After reading conflicting opinions on TDD, I decided the best thing to do is read the commonly recognized source on TDD, "Test Driven Development: By Example by Kent Beck". I felt the aspects of TDD became clear and obvious, after reading "Test Driven Development: By Example". My discussion and thoughts will be based on what I read in this book. This book is my "source of truth" on the topic of TDD.

Sources

Before getting into my discussion I will list my sources from the book that state TDD is for testing behavior.

1) page 4

What behavior will we need to produce the revised report? Put another way, what set of tests, when passed, will demonstrate the presence of code we are confident will compute the report correctly?

By page 4 Kent has explicitly stated we need to be testing behavior.

2) page 11 #1

Invent the interface you wish you had. Include all of the elements in the story that you imagine will be necessary to calculate the right answers.

We are thinking about the interface, the behavior. We are not thinking about the implementation.
We are thinking about how to "calculate the right answer", the behavior.

3) page 14

First we can talk about whether the system should work like this or like that. Once we decide on the correct behavior, we can talk about the best way of achieving that behavior.

Again stating we need to be thinking about the behavior and how the system should work.

4) page 16 paragraph 2
You aren't thinking about the implementation of equals(), are you?...I'm thinking about how to test equality.

Explicitly stating to not think about the implementation.

5) page 39 paragraph 2

How do we want to implement currencies at the moment?...I'll rephrase: How do we want to test for currencies at the moment?

Emphasizing to not think about the implementation. Think about how to test the behavior.

6) page 62 middle of page

The test above is not one I would expect to live a long time. It is deeply concerned with the implementation of our operation, rather than its externally visible behavior

Tests that test implementation do not last. The tests need to think about the behavior.

7) page 71 Bullet 2
Introduced a private helper class without distinct tests of its own.

We do not write tests for every function and class.

8) page 103
Now we are ready to implement tearDown(). Got you! Now we are ready to test for tearDown:

Test the behavior.

Now onto my discussion and thoughts...

Test Implementation or Behavior?

The biggest overarching question/concern around TDD is whether it's testing the implementation or behavior.
I think the word chosen to answer this question depends on the context of development, and TDD is usually discussed around developing unit tests.
If the context is around developer unit tests: The answer is behavior.
If the context is around higher level tests like system tests, end-to-end tests, etc.: The answer is implementation.

Here are some examples of the context to make sure I'm clear.
Let's say we are building a calculator application with a GUI, and let's only consider the following two sets of tests.
1) We will have some unit tests to test functions like add(x, y): x plus y, subtract(x, y): x minus y, multiply(x, y) and divide(x, y).
2) We will have some UI tests to test the GUI when the user is using the application (if the user presses "+" then they see the addition of the two numbers).

When working on our unit tests for add, subtract, multiply and divide, we want to make sure their behavior is correct.
Our test should be "When I call subtract(5, 1), I get 4".
We do not test "When I call subtract(5, 1), I get 4, and subtract() called add(x, negative y)."
With the above two examples, the first one tests the behavior of the subtract function. That is what we care about. We want it to be correct.
The second example is now dealing with the implementation. We do not care if subtract used "add(x, (y * -1))" or if it used "x - y", or "x + -y".

Tests give Confidence

One of the purposes of TDD is to give confidence that your code is correct after refactoring, because you will rerun your tests and if all your tests pass, then everything is still working correctly.
So if our test includes asserting "add() was called one time", what happens when we refactor subtract() to use "x + -y" instead of calling add()?
Our test fails! Well then subtract() is broken right? Our test failed, so it must be broken. Nope. subtract() still works 100% correctly. Now we waste our time examining why it failed and updating this failing test.
What happens if we update the test to reflect this new specific implementation (somehow), then we refactor AGAIN to "x - y"? The test breaks again, but subtract() is still working! subtract()'s behavior never changed.
When our unit tests are testing implementation, we end up testing what the system does, not what the behavior of the system should be.

The above makes a strong case for testing behavior and NOT implementation.
However, when we discuss the UI tests, I think TDD can be considered to be testing the implementation because TDD is usually discussed with respect to the developer's unit tests, and the code being tested by the unit tests are implementing the functionality of the UI.
This is where I think some of the confusion can arise. Even though the word chosen can change, how the code is tested, doesn't change.

So far the above has directly addressed the main statements #1 and #2.

How about #3, "TDD means writing a test for every function"? This is incorrect.
We care about testing the behavior. We just need enough tests, testing enough functions, that give us confidence that the behavior of our system is correct.
Let's say we remove all functionality from the calculator except multiply. So our UI no longer has +, -, and /.
Let's say we decide to implement multiply(x, y) by using a for loop and continually calling add().
If we wrote a test for every function we would have a test(s) for add().
Now we decide to refactor multiply() so it calls "x * y" instead, and we delete add() because we do not need it anymore.
We rerun our tests and what happens? Our tests fail, but the behavior of multiplying has not changed. It is the same scenario we saw above.
Simply writing a test(s) for every function is testing the implementation.
This does not increase our confidence that the behavior of the system has not changed.

Now we are on to main statements #4 and #5, "TDD tests implementation and BDD tests behavior" and "BDD is an improved version of TDD".
You can find different definitions of BDD, but these statements are comparing BDD to TDD so the context is around the developer unit tests (especially since some claim BDD is an improved version of TDD). These statements are incorrect.
TDD does test behavior. BDD is not improving anything. BDD certainly improves test development for people who do not understand TDD correctly.
Now, some people might define BDD differently. They might say BDD is not only the developer who is involved and BDD tests are not developer unit tests. That is not what I am comparing and discussing here.

Final Thoughts

Why is there confusion around what TDD is testing and how?
Maybe Kent should have called it Behavior Test Driven Development instead.

After I read Kent's book everything clicked. I had taken people's bad advice and wrote tests for every function and ran into problems and slow downs during refactoring. I questioned if asserting method call counts was really doing anything. Turns out I was wasting my time. I had been testing implementation instead of behavior.

I highly recommend "Test Driven Development: By Example by Kent Beck" to anyone who has not read it. He thoroughly covers TDD by building some example projects and it's very helpful to see the process. He also goes over general testing tips and good code quality tips.

How to create a gauge / speedometer graphic with Python

2018-06-30

This article discusses how to easily create a speedometer / gauge image in Python with only the Pillow library.

View source code and images on GitHub: python-gauge

I needed to create a gauge graphic in Python but I found many libraries did not have the graphic I needed or required that the gauge be rendered in a browser since the underlying library used was JavaScript. This script only requires the Pillow library since it is generated using two png images.

The gauge graphic is separated into two images. The first image is the needle that points to a number on the gauge. The second image is the gauge without the needle. The final gauge image is generated by rotating the needle image then pasting it on the gauge.

The script is simple. The gauge is half a circle therefore the needle needs to rotate between 0 and 180 degrees. The gauge goes from 0 to 100. So if you want the gauge to point to 20, the script just takes 20% of 180 degrees to rotate the needle.

python_gauge.py

"""
Copyright (C) 2018 FireEye, Inc., created by Andrew Shay. All Rights Reserved.
"""

import PIL

from PIL import Image

percent = 20  # Percent for gauge
output_file_name = 'new_gauge.png'

# X and Y coordinates of the center bottom of the needle starting from the top left corner
#   of the image
x = 825
y = 825
loc = (x, y)

percent = percent / 100
rotation = 180 * percent  # 180 degrees because the gauge is half a circle
rotation = 90 - rotation  # Factor in the needle graphic pointing to 50 (90 degrees)

dial = Image.open('needle.png')
dial = dial.rotate(rotation, resample=PIL.Image.BICUBIC, center=loc)  # Rotate needle

gauge = Image.open('gauge.png')
gauge.paste(dial, mask=dial)  # Paste needle onto gauge
gauge.save(output_file_name)

To modify the gauge image, modify gauge.psd. gauge.png should be the gauge without the needle (hide the needle layer). needle.png should only be the needle on a transparent background and should point at 50 (hide all layers except needle).

Device Checker: New project for detecting changes on Windows

2018-03-24

I recently wanted software to notify me when services, installed programs, and startup programs were added, removed, or changed.
I found some wmic commands to give me that information, as well as much more information.

I created a program to simply gather information about a Windows system and then notify the user when it has changed, and view the differences.

It is currently released in beta.

View on GitHub: shayConcepts/device-checker

How to get the data from msinfo32 with wmic commands

2018-03-23

I was recently looking to gather the same data that msinfo32 reports and I found a couple wmic commands that provided me with what I needed.

The first is a basic wmic command eg wmic service get /format:list. This will output the services on Windows.
Run wmic /? to see a list of other information you can gather.
Other formatting options /format:table and /format:csv

The second wmic command uses the CIM WMI Provider. eg wmic path CIM_LogicialDisk get /value. This outputs the logical disks that Windows detects.
To find a list of possible options see the MSDN docs https://msdn.microsoft.com/en-us/library/aa386541.aspx

Apache disable TLS_RSA_WITH_3DES_EDE_CBC_SHA after LetsEncrypt

2017-05-09

This guide is for Apache but might be similar for other web servers

If you've just installed Let's Encrypt certificates with Certbot on your web server and ran SSL Server Test, you might have warning saying your server supports TLS_RSA_WITH_3DES_EDE_CBC_SHA.

To drop support for this protocol you must

Open /etc/letsencrypt/options-ssl-apache.conf
Delete :DES-CBC3-SHA from the SSLCipherSuite line
Restart apache sudo service apache2 restart

Also, here is the default conf file https://github.com/certbot/certbot/blob/master/certbot-apache/certbot_apache/_internal/tls_configs/old-options-ssl-apache.conf