Mostly Software

    • BBQ
      • The smoker went for 8.5hrs unattended last night. When I woke up, the coal basket and water pan each had about 30% remaining? At 175 and you could go 10+ hrs. Great to know. Turned it hot and spritzed, will transfer to the oven in about an hour.
    • Supercontest
      • Python 3.
        • iteritems -> items
        • operator.itemgetter -> dict.get
        • dict.keys(), dict.values(), and zip() all return iterators now, not iterables. Cast to list when necessary. This is nice for memory saving, obviously.
        • import pdb; pdb.set_trace() -> breakpoint()
        • asyncio, obv
      • docker-compose restart <> is the same as start -> stop. Useful for reloading.
      • But there’s an even better way to iterate in development. Added –py-autoreload=1 to the uwsgi cmd call in the app container (same as flask’s dev server reload). Now when my files change (based on the mounted volume of my source), the uwsgi server reloads. Great!
      • Watch the app in realtime with docker-compose logs -f <> !
        • This confirms the app restart on change as well.
      • Effectively about 2.7k lines in the entire infrastructure of my application:
        • find -type f -not -path “*.venv*” -not -path “*.git*” -not -path “*.idea*” -not -path “*backups*” -not -path “*egg-info*” -not -path “*.pyc*” -exec wc -l {} +
          • 2743 total
      • There’s only 800 lines restricting it to actual application code (python extensions within the supercontest/ subdir):
        • find supercontest -name “*.py” -type f -not -path “*.venv*” -not -path “*.git*” -not -path “*.idea*” -not -path “*backups*” -not -path “*egg-info*” -not -path “*.pyc*” -exec wc -l {} +
          • 864 total
      • Pretty amazing. Under 1000 lines of custom code to produce a fully functional website.
      • Adding all the enterprise bells and whistles of dev, management, deployment, etc only adds about 2000 more lines.
      • Furthermore, the infra is twice the size of the source. Not surprising.
      • Ran pylint (via tox) for the first time in a longggg time.
        • Most common: no-member on all the usages of `db` (to commit, add, delete, etc).
      • Closed #19.
    • To search with ag but only for a specific filename regex (like an extension), use:
      • ag -G “.ext” keyword
    • JS. Class inheritance, templating. Prototypal inheritance (from other objects), more like composition. Composition is almost always favorable.
    • Angular has more of a pattern of two-way binding (frontend object updates, backend does as well, and vice versa). React is more one-way dataflow. The model is the only source of truth. The view just reflects it.
    • Game of thrones final season premiere!
    • My typing cursor had been jumping around a bit. It’s a little annoying. The browser typing isn’t as consequential, but it’s dangerous in vim when I’m actually writing code. The active line changes based on autocomplete and other things, and I don’t want any unintended visual shift to slip under the radar because of this.
      • I changed the cursor theme back to adwaita (default) in hopes this would help, based on this answer: https://askubuntu.com/questions/115349/why-does-my-cursor-jump-when-typing. Don’t think it’s gonna change much. Changing the movement speed just means you can’t move to a new line as quickly, which is a bandaid on the problem. A real solution would be changing the sensitivity. Maybe disabled touch clicks (and only registering push-clicks) would help. Otherwise, I just have to be very cognizant about my thumbs not touching the pad when typing (the setting for “disable touchpad when typing” is already on, but doesn’t work well).
      • Ya, that worked. Disabling tap-to-click means that errant touches don’t misplace my cursor. Well worth the trade of not being able to silently click on something.
    • Supercontest.
      • Continued work on the ever-helpful makefile. Added backup-db and restore-db. pg_dump defaults to output a sql text file, which pg_restore cannot read (lol). psql can, tho. Instead, I chose to have pg_dump output a custom file (which pg_restore CAN read, along with dirs and tars).
      • If just performing alterations, then you can connect to the db and modify as you’d like. If doing a DROP -> CREATE (which can include other things like schema and role changes), then there obviously can’t be other active connections to the db. Therefore, the flask container must be down before a restore runs.
      • Made the services and the containers the same name for convenience.
      • Got backup/restore working, but then it failed on a completely fresh postgres image because pg_dump does not include roles. Postgres keeps these at the cluster layer, which applies to all databases. I’ll need to change it to pg_dumpall.
      • Ended up actually not having to backup and restore the roles. Instead, I added docker/database/public.conf env_var back to the postgres image so that initdb would create the supercontest db and the supercontest role. Then my restoration just uses it to fill the data.
      • alter [table/database] [matchup/public.user/alembic_version/pick] owner to supercontest
      • drop role bmahlstedt
      • Added a healthcheck to the database container startup to make sure 5432 was ready. (pg_restore was running before postgres was actually up). JK, just added a sleep after the docker-compose up.
      • Ran some interesting queries on the data. Adding all teams picked by all players for the 2018 season, the top-picked teams were (1) chiefs (2) pats (3) rams. Shows that there’s subconscious bias for better teams, even when the lines are supposedly 50/50.
    • In a makefile, VAR=value doesn’t actually set it until used (lazy). VAR:=value forces it to evaluate and set upon declaration. 
    • I’d been using docker-compose up/down, which removes and creates the containers as well. If you just want to start and stop the existing containers, use start/stop.
    • Dad’s bday. Spoke on the phone for a while.
    • Started the big bbq for tomorrow. Brisket rubbed and put in the smoker overnight, lowest temp (175). Targeting 10pm-6am bark formation, then foil and oven.
      • HN/MD.
        • Read a bit about GraphQL again.
        • A curried function takes multiple arguments and processes/returns one at a time, sequentially.
        • Pretty awesome (but decently long) walkthrough of what a db is dong under the hood: https://cstack.github.io/db_tutorial/.
        • Amazon obviously listens to everything Alexa hears (anonymously). https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio.
        • Agile Lite. A more flexible, but still structure approach to sprinting: https://github.com/davebs/AgileLite.
        • Fascinating statistical snapshot of the whole software community: https://insights.stackoverflow.com/survey/2019.
        • Tested out a new stylesheet: https://github.com/kognise/water.css. Was pretty cool, but I kept my old CSS. This (and other just-add-css options) would be nice for future changes, to mix it up for the users.
        • Investments.
          • Nowadays, it’s not just a seed and venture round then an IPO 6 years later. There are many growth rounds, and companies can stay private for longer than a decade, continuing to increase value. Companies stay private longer because it makes the investors more money.
          • Founders and VCs have preferred stock. They have RSAs instead of ISOs. They can create liquidity events whenever they want with another growth round. They incur a lot less risk, while owning a substantially larger share of equity. Employees are just kinda along for the ride (which is not crazy, since employment is opt-in).
          • The liquidity goalposts have moved in the founder/VC favor, but the vesting goalposts for employees have not. Many believe that there should no longer be a vesting cliff because its original timeline was basically to finish at the IPO. Now, that’s no longer the case.
          • If a company does offer liquidity events for employees regularly, these arguments go away.
          • Dilution is real.
          • https://steveblank.com/2019/04/10/startup-stock-options-why-a-good-deal-has-gone-bad
      • Something weird happened with my docker-compose installation.
        • Ran apt install python3-apt because it was missing the apt_pkg. Strange.
        • Tried pip installing some stuff. ln -sf /usr/bin/pip3 /usr/bin/pip
        • Cat /usr/bin/docker-compose. No, docker-compose shouldn’t be a python entry script.
        • Ahhhh. docker-compose is installed from git, not apt. The package on apt is a python package. https://docs.docker.com/compose/install/. Reinstalled normally.
      • Picked up, trimmed, and brined 2 brisket packers and 2 pork butts for Sunday’s GoT party.
      • sudo apt autoremove killed a bunch of stuff again. I’m not going to run it anymore.
        • Some important packages: sudo apt install ubuntu-software update-manager ubuntu-desktop gdm3 gnome-control-center
        • Ugh aaaand it removed the display manager. Unbelievable.
        • Stuck in a login loop. ctrl-alt-f3 to enter terminal.
        • First, ubuntu is the dist based on debian. It’s the OS.
        • On top of ubuntu, there are desktop environments. Unity has been the primary for a long time, but now the default is Gnome for 18. Another option is Xfce (Xubuntu).
        • In addition, there are display managers (login managers). gdm is the default gnome one, and lightdm is a common alternative.
        • I had to reset everything back to gnome manually because autoremove removed it. Ridiculous.
        • `wmctrl -m` shows the desktop manager (window manager). Mine is back to gnome now.
        • `cat /etc/X11/default-display-manager` shows the login manager. Mine is back to gdm3 now.
      • A few more vim improvements.
        • Started using tabs instead of split windows. Just ctrl-t (like chrome) then a filename to open a new one. ctrl-w to close. Then and left or right arrows to move between them.
      • @ at the beginning of a command in a makefile makes it not echo the command itself. Unlike bash, it does print by default.
        • Updated default python to 3.7 (was 2.7) – `ln -sf /usr/bin/python3.7 /usr/bin/python`
        • Reconfigured the weekly backups to basically include my home dir but skip all subdirs. Ran my first backup.
        • More on the black hole image from yesterday; they stitched together a virtual aperture much larger than the individual imagers in order to achieve higher resolution, exactly like our work at Stanford with atmospheric lensing! The lead spoke about the process in a Ted Talk 2 years ago: https://www.youtube.com/watch?v=BIvezCVcsYs. It’s the same. We did some lunar photography by stitching together images (pixels) that refracted through various eddies in the atmosphere, creating a virtual aperture of a few kilometers. Their black hole target was much farther away. The simple diffraction limit required an aperture of about the size of the Earth (thousands of kilometers) in order to get the resolution to see the black hole so far away. So, the algorithm stitched together a TON of data from telescopes around the globe, over time, so the rotation of the Earth itself caused the observatories to cover more of the surface area of this virtual aperture. Same exact process. Reached out to Ved to revel.
        • Alt Shift X honestly has 50+ GoT videos, each 10-15m. I watched most on 2x speed.
          • The old Grandmaester Pycelle had a weird close relationship with Tywin.
          • Qyburn reanimated The Mountain. Cleganebowl would be soooo sick.
          • Lady Stoneheart is the reanimated Catelyn Stark.
          • The Valyrian empire held most of Essos until the Doom. All secrets about how to make Valyrian steel were lost, most of the dragons died, and the city of Valyria is basically nuclear fallout. The Targaryens were the only survivors because they left before. The Doom is the antithesis to the Long Night, the fire and ice contrast of the series.
          • The Arm of Dorne was a land bridge that the First Men used to cross from Essos to Westeros. It was destroyed by the Children of the Forest’s gods so no more colonizers would come.
          • The Horn of Winter brings down the wall and raises giants. It might be in the possession of Sam Tarly.
        • Placed Amazon Fresh order.
        • Removed two .desktop files in ~/.local/share/applications that were lingering but not installed. This was creating false icons in the applications menu (one for google play music and one for google remote desktop).
        • Added all common apps to dock (favorites) and cleaned the desktop background a bit.
        • Created symlink from ipython to ipython3.
        • Messed with my tmux conf a little.
        • Finally set up ctags (and uninstalled pycharm, the only real thing I used it for).
          • sudo apt install exuberant-ctags
          • Added tag location to vimrc `
          • Indexed the tags: `ctags -R -o ~/.tags $(~/code/supercontest/.venv/bin/python -c “import os, sys; print(‘ ‘.join(‘{}’.format(d) for d in sys.path if os.path.isdir(d)))”)`
            • Takes about 4s and generates a tag file of about 20MB for supercontest.
            • Added a bash script to wrap this with a venv, then an alias to wrap that so you can run it whenever.
          • ctrl-] to jump to, ctrl-o to jump back
          • Here is a neat way to reindex automatically with git: https://tbaggery.com/2011/08/08/effortless-ctags-with-git.html. The most common way is probably to reindex on a schedule with a cron job.
        • Added dockerignore (for the .venv right now)
        • Had to reinstall python3.7-dev to build the C extensions for psycopg and uwsgi.
        • To reload a config file, run `source .config_file`
        • Remember, the most portable shebang is `#!/usr/bin/env bash`
        • PW/MD
        • TDD = test driven development.
        • Thrust profiles usually decrease through a trajectory. This is seen commonly if you blow up a balloon and let it go. See that little kick at the end when it seems to go faster? F=ma. As mass decreases, and force stays constant, acceleration must increase. In order to keep the vehicle from speeding up, subjecting it to higher load stress, thrust can be decreased.
        • Vim
          • :Explore to open netrw (directory browser)
          • Added Vundle to my vimrc. Remember to call :PluginInstall after any changes.
          • Fugitive has a ton of super useful git integrations. You can blame, search history, checkout a file in mem, etc. I chose NOT to install this. I like keeping my text editing separate from my version control interface.
          • Use vim for multiple windows (instead of tmux). Use tmux for multiple processes. For multiple files being edited at once, use vim. This is a bit of shift from what I’m used to, even years into my career.
            • :sp file/path/blah to open a new file in a horizontal split. :vsp for vertical.
            • ctrl-w then an arrow to move between windows, just like ctrl-a for tmux.
            • Maximize is ctrl-w then _, go back with ctrl-w then =
            • :h or :h <plugin> for any help
          • Added the syntastic plugin for pylint/flake8 as well, but then removed it. Needs an active flake8/pylint on PATH, which I don’t wanna mess with since I develop in docker. Just ended up using the built-in python syntax highlighting. I’ve got enough experience with Python to not need realtime lint notification.
          • Added the Jedi plugin for autocompletion. ctrl-space to open autocompletion.
        • Summary:
          • You have ctags now: ctrl-] and ctrl-o
          • Split windows in vim instead of tmux: :sp file/path and ctrl-w arrow
          • Autocompletion: ctrl-space
          • Syntax highlighting.
          • The vimrc improved a ton of things for tabs, wrapping, color, etc.
          • TJ Dillashaw suspended two years for EPO! That’s the blood doping that Armstrong was involved with. It’s a severe performance enhancer for cardio. Brendan Schaub said before this breaking news that USADA had found a better way to screen for EPO, and fighters were going to start getting popped: https://www.youtube.com/watch?v=NABFMFHnstw&feature=youtu.be&t=3963. Amazing. Cody Garbrandt also specifically called him out for EPO before this.
          • Cleaned kitchen and organized utensils into a compartmental drawer finally.
          • Ran the SSL Labs tests against the encryption I set up for southbaysupercontest.com:
          • You can deduct state and local taxes (SALT) from federal taxes, up to the max (right now it’s 10k). TurboTax accommodates this, but you have to itemize. Worth looking at if you’re paying a lot in state taxes.
          • In docker-compose yamls, ‘expose’ means the port is exposed for linked services (like flask<->nginx) but not to the host computer. ‘ports’ are for exposing to the host.
          • Letsencrypt limits to 5 cert requests for the same domain per week.
          • FW.
          • Docker has the .dockerignore file available, just like git. This can be used to skip files/directories from COPY, etc.
          • Use apt-get and apt-cache in scripts instead of apt. Much more parseable output.
          • First ever image of a black hole! Good description here (and it preceded the image!) – https://www.youtube.com/watch?v=zUyH3XhpLTo.
          • Watched a tonnnn of alt shift x game of thrones videos to prepare for the premiere of the final season.
            • I always forget the history of the north. The first men came and attacked the natives, the children of the forest. The children needed help, so they captured a man and stuck dragonglass in his heart. This turned him into the first white walker, the night king. Eventually, the children and men came to a pact. Fast forward, and the army of the dead has grown uncontrollably, and it’s men+children vs them. This war was called the long night, or war for the dawn. The dead lost and were pushed back. Bran the builder (the first stark) and the children helped build the wall. The night’s watch was created to keep them north. Wildings then became a thing. They don’t have a historical significance like the dead/children/firstmen, they’re just free folk in the risky north.
            • While that account is the most historical, and is common in the north (called “old gods” with nature, weirwood, ec), there are other religious beliefs of the history. King’s landing obvious believes in the seven (Andals), hence the hesitance to believe in the white walkers. Melisandre believes in the lord of light, R’hllor, who use a sword called lightbringer. A prophecy says he’ll be reincarnated as the prince that was promised, who Melisandre believes to be Jon Snow.
            • The Andals came much later. They are from Essos and invaded Westeros, becoming the main culture we see in the show, with the exception of the north (they didn’t invade that far).
            • Wargs can possess animals. Bran is a warg. Bran also has greensight (visionary time travelling dreams).
            • White walkers are the captains. The hordes they convert are wights.
            • Dragonglass is obsidian. It is different than Valyrian steel, which is manmade. These two are the only materials that kill white walkers (+wights).
            • The three eyed raven is just a human. He was hand of the king back in the targaryan days, when his name was bloodraven, then left to become lord commander of the night’s watch, then disappeared north of the wall. He went to a cave at the base of a weirwood tree and basically mated his body with the trunk. He’s the last of the greenseers and can use visions to communicate (called greensight). This is how he finds and teaches Bran, ultimately bringing him to the cave. Then the night king killed him.
          • Supercontest.
            • Docker/letsencrypt
              • Remember, now that my uwsgi file is listening on a socket :8080, it can’t serve http directly. It needs to be wrapped with a web server. To run in develop mode locally, you need to forward to a tcp socket and specify the protocol as http.
              • I enabled this by rearranging the compose and uwsgi configurations. It’s now modular for prod and dev.
              • A service (or dockerfile) can only have one CMD. A RUN is called when the image is built. A CMD is something that’s called every time the container starts. You can have multiple RUN calls. The state of the container after they’ve finished will be built into the image.
              • Oh man, I spent a while trying to debug my services. It was just that LetsEncrypt was down, so my requests for certs (even dummy certs from staging) were getting ReadTimeout errors. You can check status here: https://letsencrypt.status.io/.
              • https://letsdebug.net/ is a great site for troubleshooting why you might be failing letsencrypt http challenges.
              • Another great site: https://transparencyreport.google.com/https/certificates. You can publicly view the certificates that have been granted to any site! Useful for debugging your own.
              • Certbot can either run an HTTP challenge or a DNS one to verify ownership of domain.
              • After a very long and frustrating (but educational!) sequence of troubleshooting, I made some progress. `docker-compose exec container_name command with no quotes` yielded some interesting info. My nginx container had an old conf file at /etc/nginx/conf.d. In all of my iterations on init-letsencrypt, the web_server container hadn’t been rebuilt to serve the webroot files at /var/www/certbot, so the challenges were failing when they requested them. Rebuild your images when you change anything in their dockerfiles!
              • This allowed my changes to ingest, isolating the problems faster. I rewrote nginx.conf with clearer simplicity, and removed the ‘www.southbaysupercontest.com’ from all entries. It’s just specified in the certbot command line call, not the nginx conf anymore.
            • Upgraded to Python 3.7 finallyyyyy
              • sudo apt list –installed *python*
              • sudo apt remove <the python packages you want>
                • You can accidentally remove wayyyy too much if you autoremove *python*
              • sudo apt update && sudo apt upgrade
              • sudo apt install python3.7
              • The service brought up and the main page was functional on the very first docker-compose up with the 3.7 python image, no problems! I’ll do more advanced digging soon.
          • Accidentally messed my machine up during some uninstalls. Would only boot into a terminal. Ended up being a reinstall of ubuntu-desktop.
          • Enabled automatic backups of my home folder on Google Drive.
          • `apt list –installed | wc -l` is 1782!
          • gnome-tweak is the package that offers extended settings (like removing the trash icon from the desktop).
          • Started the brisket. Went on at 845am. I’m getting the process down pretty smoothly now.
          • Listened to big bootie mix 15 for the first time.
          • Submitted my purchase offer election and emailed the subsequent transmittal form. Requested 900.
          • Ahhhh, this is what our dodgeball team is named after: https://www.youtube.com/watch?v=cVsQLlk-T0s.
          • Went to del amo to pick up the bar end that they forgot to replace originally. Took it home and installed it myself, just a #5 billet assembly for the bar end that extends about a half inch for the #3 mirror assembly to clamp onto.
          • You cannot use letsencrypt for local development. You cannot certify that you own an IP address, and you cannot certify that you own the generic “localhost” domain. Usually you can just provide yourself a self-signed certificate to make sure https works before you deploy to production.
          • Many benchmarks of the original py3 release were slower than py2. That’s no longer the case, they’re close for most (or advantage 3).
          • Worked an Autotest issue with briley, pointing toward the hyperion.shared.install_autotest module to show how to provision python workers with monorepo packages (to avoid conflicts with first-parties being used as third-parties).
          • Supercontest
            • Commented out the `COPY dump.sql -> docker-initd` in the database Dockerfile. If expects a db in the mounted volume, it shouldn’t rebuild an old one everytime. If you want to restore another database, do it manually.
            • Shifted the development vs production workflows. Dev will just use runserver and flask with a simple sqlite database, no containers, no nginx, no postgres, no https.
            • Set up automatic certification. Adds a 4th container, certbot, which exposes its certificates to the nginx container over a shared docker volume. It then runs certbot renew and nginx reload every few hours to ensure that they never expire.
            • Because the current (whole) source directory is mounted, you CAN just make a change to the app on the host (on the droplet) and it will be reflected immediately in the app container (because the supercontest package is installed editably).
            • Deployed to production
              • Followed the instructions (I wrote) in ADMIN.md. Installed docker, created the secrets files, ran the cert initializer, then started the services.
              • Copied the db backup over (from my laptop): scp -i ~/.ssh/digitalocean code/supercontest/backups/postgres/dump.sql southbaysupercontest.com:~
              • Restored the data (on droplet): cat ~/dump.sql | docker exec -i postgres psql -U postgres
              • Got https green, commited and pushed, closed ticket #27.
            • Fully done with all 4 docker containers, a wonderful composition.
            • Bought a whole packer. Prime. ~9lbs untrimmed. $3.50/lb (snow crab was almost $10/lb!). Trimmed and brined it. Will smoke brisket tomorrow. Separated the point and the flat.
              • My pitmasterIQ arrived as well! Pretty easy setup. Ran a power cable out of the window, connected the fan to the bottom vent, and snaked the RTD (with alligator clip!) to the top grate. Will use it tomorrow with the brisket.
            • Shaved head for the first time in (years?)! Easy, free, clean. Better for motorcycle helmet, gym, summer.
            • Watched GoT recaps for seasons 1-7.
            • `sudo -i` is used to enter a root shell. This is how you do certain things as root, like cd. Since you can’t `sudo cd <>` you have to change to root (you could also do this with su) then cd there in the root shell.
            • Docker volumes.
              • Docker mounts are stored on the host in /var/lib/docker/volumes by default, if you don’t specify a host location. This is useful if you don’t need host files to be mirrored to the container, you just want the data in the container to persist when the container is destroyed. This is common for databases.
              • If you specify a hostLocation:containerLocation volume, then it’s basically a bidirectional mount point. This is useful if you want changes on the host to sync into changes on the container, and vice versa. This is common for source (git repos).
              • You can share volumes across containers, btw, because they’re all persisted on the host.
              • You can do `docker volume ls` just like images or containers. You can `docker inspect <volume_name>` as well.
              • If volumes were created by docker-compose, you should delete them with docker-compose as well: docker-compose down –volumes (this will delete your data!)
              • You only need to pass `–build` to `docker-compose up -d` if….. you want to rebuild the images before starting the containers! If you don’t, just run UP. It will be way faster.
            • Supercontest.
              • Test. Changed my picks. docker-compose down, then back up. The picks were gone, as expected, because I didn’t set up the volume for them yet. Added the volume, then reperformed the test. They persisted.
              • Got all volumes working as expected.
              • The 3 main containers and docker-compose should be in a good place now. Moving on to the letsencrypt container.
            • NCAA national championship.
            • Last dodgeball game of the season. Playoffs begin next week.