• Apr 19, 2019

  • Supercontest
  • Scrubbed testing.postgresql. Wasn’t configurable enough for my tests. Couldn’t do stuff like changing the default encoding of the db, so collation in the static model failed. Instead, I just defined a separate uri for test (with proper cleanup) and piggybacked off the host postgres installation (which is obviously then required to run tests….system dep agh).
  • Removed the collation=NOCASE remnant from sqlite in my models. Then I was able to get a successful unnitest env working. Test app connected to test db.
  • The problem now is alembic. The migrations require a local postgresql instance, so I can’t run them nominally on the host – I need to run them in the context of the container. Alembic enters through flask-script, which runs on the app container. So you can run a migrate operation while just connected to one container.  But the upgrade operation needs the migrate.py from the app container and the actual db from the postgres container, so who knows how that will work. I’ll have to look up the right way to use alembic with docker.
  • I guess you’d run the manage.py migrate/upgrade commands through docker-compose exec, not on the host. Ok.
  • Didn’t end up having to migrate since postgres doesn’t need the collation, but for future reference: `python manage.py db migrate/upgrade` needs to be run through `docker-compose exec app_dev/prod` now that you’re working in a container setup.
  • Testing.
  • Added extra_config_settings to get_app(), allowing direct flask args to be passed from elsewhere (like tests, where I suppressed mail and csrf).
  • Added tests to make sure all the user routes (login/profile/logout/etc) work.
  • Debugging.
  • You can run `python -m pdb <myscript.py>` or just put `breakpoint()` in your code.
  • Then you can step through with s.
  • To print, just use `p <var>`.
  • You can also check types, specify new variables, etc! `pp locals()` or globals as well.
  • `l` or `where` will list where you are in the script.
  • `exit` finishes.
  • Remember, a lot of system binaries (like echo) are at /bin, not /usr/bin or /usr/local/bin.
  • Security.
  • Bandit is a cool tool I started playing with: https://github.com/PyCQA/bandit.
  • Added to supercontest tox. I had no issues, just two (example) passwords I had to ignore with # nosec.
  • Internet fiasco.
  • They wouldn’t activate the new account because the modem (which I own, not spectrum) is still attached (maybe mac address) to the old account. After calling them and getting transferred 10 times, and briley doing the same, we finally got it working.
  • In the meantime, I went to a few Hermosa cafes for wifi. The new yellow one doesn’t offer public internet (lol) and gum tree gives out temporary codes which provide access for an hour (lol). What happened to cafes?
  • Indian BBQ.
  • My two favorite cuisines on the planet. Had a thought the other day they I might try some fusion with my smoker at home. Did some research, and nothing came up. I don’t think it’s a very common idea, since the origins of BBQ are the south, and there’s not much overlap between the south and India, at least culinarily. High-level smoked meat with curry-based rubs….sounds delicious.
  • Started first batch today. Curry jerky! Thinly sliced cross rib roast with some premix curry powder. I can play with the levels and finetune later.
  • My xps13 ubuntu18 suddenly couldn’t play sound and just showed “dummy output”. Running `pulseaudio -k && sudo alsa force-reload` fixed it.
  • HN
  • Awesome statistical model of skittles: https://possiblywrong.wordpress.com/2019/04/06/follow-up-i-found-two-identical-packs-of-skittles-among-468-packs-with-a-total-of-27740-skittles
  • Mentor, don’t hire. Workplaces of growth are much better investments. https://sizovs.net/2019/04/10/the-best-developers-are-raised-not-hired.
• Apr 18, 2019

  • Exported my amazon data to google drive and ran some queries on it. Interesting.
  • 
  • 
  • Went to San Bernardino yesterday to hang out with Grandma and mom. Got Rosa Maria’s. Called Eric for his bday.
  • PW: GoT and Python https://medium.freecodecamp.org/how-i-used-python-to-analyze-game-of-thrones-503a96028ce6.
  • Guy on jeopardy has now won 10 games straight and holds the top FOUR single-day winning totals (from ~90k-130k). Total is 700k over 10 games. He’s got the second highest total now (non-tourney) after Ken.
  • Reapplied some thermal tape to the denali compressor. Noticed yesterday after the freeway riding that it was worn through.
• Apr 16, 2019

  • The line between functional programming and OOP is way grayer than most silly online articles try to make it seem. If you have mostly fixed data, and you think your project will scale by adding functionality, then functional programming is a good fit. If most of the functionality is already modelled and fixed, and you think your project will scale by adding more data, then OOP is a good fit. Most projects are combinations of both.
  • Supercontest
  • A good software design reminder: separate functions that read/write data from those that analyze the data. For example: the function that calculates if a pick is good. It queries the picks and matchups, does the math, then commits the results. Instead, it would be better as two functions: one that does the math, then a wrapper that does the query, calls the math func, then does the commit. This makes it a lot easier to unittest. You can write one just for the analytical part, then another that tests the db interfaces.
  • Webtesting options.
  • flaskr. http://flask.pocoo.org/docs/1.0/testing/.
  • flask-webtest. https://flask-webtest.readthedocs.io/en/latest/.
  • flask-testing. https://pythonhosted.org/Flask-Testing/.
  • They’re all pretty similar. Can create a client app connected to a db, get/post all routes, login/logout, context and sessions, sqlachemy support, json responses, selenium, etc.
  • I’m gonna use flask-testing.
  • The dev containers (flask and postgres) only consume about 10% memory (maybe 300MB). Not much cpu diff.
  • Unittests.
  • Changed get_app to accept a URI to make unittesting easier (creating a temp postgres db).
  • Remember the javascript analogs for python:
  • python=node, pip=npm, virtualenv=nodeenv, pylint=eslint, pytest=jasmine
  • Ran some delinting for javascript.
  • Did a little more bazel research. Still not the most useful thing for python, which doesn’t need to be built. Think of it like a make system. You have inputs, you have recipes that define what to do with the build, and then you have outputs (usually files).
  • The real power comes when everything in a highly complex system is built with bazel, because then you can define the deptree for the whole project/monorepo/whatever, even across languages.
  • Everything is a file, remember! This is why I think the python-package-support is a halfway solution.
  • Keanu Reeves founded a motorcycle company in 2011 named ARCH and it is headquartered in Hawthorne, CA!
  • Started using the whatsapp web app on my laptop. Convenient.
  • Also switched my default messaging (SMS) app to google, which has a web app. Texting from my browser on the laptop!
  • Changed the trackpad setting from area to fingers (1 anywhere for click, 2 anywhere for right click).
  • Reinstalled GNOME system monitor. Wasn’t working properly. Still amazed at how resource-hoggy chrome is.
• Apr 15, 2019

  • Supercontest
  • Ansible
  • Falls back on paramiko (python) if openssh fails.
  • Setup
  • sudo apt install ansible (this gives ansible-playbook and others)
  • added southbaysupercontest.com to /etc/ansible/hosts
  • ssh-agent bash
  • ssh-add ~/.ssh/digitalocean
  • ansible all -a “/bin/echo hello”
  • A lot of ansible’s power used to be the composition of various microservices, which docker-compose is a much simpler alternative for. It still has relevance in fleet management, however, and that’s what I’ll use it for.
  • Refamiliarized myself with their documentation.
  • I could just write a shell script to ssh, git pull, run the make target for docker-compose, then rsync dump files back and forth, but I’m gonna use ansible just for fun. It will enable scaling, load balancing, and more later (if need be).
  • Added playbooks to deploy the whole app remotely from local, as well as backup and restore the db. You simple enter through a oneliner make call, which farms out to ansible (which goes back and calls other make recipes).
  • Closed ticket #10.
  • Did a few test deploy/backup/restore operations between my laptop and the production server.
  • Upgraded docker-compose to 1.24 on southbaysupercontest.com. The py apt version was installed.
  • Unittests
  • Looks like there’s some nginx testing capability: https://github.com/nginx/unit/tree/master/test.
  • The module for a temp postgres db is: https://pypi.org/project/testing.postgresql/.
  • There are some docker unittest frameworks as well, but because it’s a generic tools, the test capabilities are generically shallow: https://github.com/GoogleContainerTools/container-structure-test.
  • Chromedriver
  • Tested the linefetching capability in the docker container. There were holes.
  • libnss3 was necessary for chromedriver.
  • You also need the chrome binary. Added.
  • Couldn’t verify end-to-end without some spoofing, because westgate has no lines during the offseason.
  • Here is another awesome script: https://gist.github.com/ziadoz/3e8ab7e944d02fe872c3454d17af31a5. If I have any problems at the start of the 2019 season, I can just add that to app/Dockerfile.
  • You can’t band-aid a poor engineering decision with software: https://www.youtube.com/watch?v=H2tuKiiznsY.
  • To rebalance panes in tmux so they’re equally spaced, used ctrl-b then alt-1 (vertical) or alt-2 (horizontal).
  • Python 3 natively keeps command history across interpreter sessions (like ipython does). Very convenient.
• Apr 14, 2019

  • BBQ
  • The smoker went for 8.5hrs unattended last night. When I woke up, the coal basket and water pan each had about 30% remaining? At 175 and you could go 10+ hrs. Great to know. Turned it hot and spritzed, will transfer to the oven in about an hour.
  • Supercontest
  • Python 3.
  • iteritems -> items
  • operator.itemgetter -> dict.get
  • dict.keys(), dict.values(), and zip() all return iterators now, not iterables. Cast to list when necessary. This is nice for memory saving, obviously.
  • import pdb; pdb.set_trace() -> breakpoint()
  • asyncio, obv
  • docker-compose restart <> is the same as start -> stop. Useful for reloading.
  • But there’s an even better way to iterate in development. Added –py-autoreload=1 to the uwsgi cmd call in the app container (same as flask’s dev server reload). Now when my files change (based on the mounted volume of my source), the uwsgi server reloads. Great!
  • Watch the app in realtime with docker-compose logs -f <> !
  • This confirms the app restart on change as well.
  • Effectively about 2.7k lines in the entire infrastructure of my application:
  • find -type f -not -path “*.venv*” -not -path “*.git*” -not -path “*.idea*” -not -path “*backups*” -not -path “*egg-info*” -not -path “*.pyc*” -exec wc -l {} +
  • 2743 total
  • There’s only 800 lines restricting it to actual application code (python extensions within the supercontest/ subdir):
  • find supercontest -name “*.py” -type f -not -path “*.venv*” -not -path “*.git*” -not -path “*.idea*” -not -path “*backups*” -not -path “*egg-info*” -not -path “*.pyc*” -exec wc -l {} +
  • 864 total
  • Pretty amazing. Under 1000 lines of custom code to produce a fully functional website.
  • Adding all the enterprise bells and whistles of dev, management, deployment, etc only adds about 2000 more lines.
  • Furthermore, the infra is twice the size of the source. Not surprising.
  • Ran pylint (via tox) for the first time in a longggg time.
  • Most common: no-member on all the usages of `db` (to commit, add, delete, etc).
  • Closed #19.
  • To search with ag but only for a specific filename regex (like an extension), use:
  • ag -G “.ext” keyword
  • JS. Class inheritance, templating. Prototypal inheritance (from other objects), more like composition. Composition is almost always favorable.
  • Angular has more of a pattern of two-way binding (frontend object updates, backend does as well, and vice versa). React is more one-way dataflow. The model is the only source of truth. The view just reflects it.
  • Game of thrones final season premiere!
  • My typing cursor had been jumping around a bit. It’s a little annoying. The browser typing isn’t as consequential, but it’s dangerous in vim when I’m actually writing code. The active line changes based on autocomplete and other things, and I don’t want any unintended visual shift to slip under the radar because of this.
  • I changed the cursor theme back to adwaita (default) in hopes this would help, based on this answer: https://askubuntu.com/questions/115349/why-does-my-cursor-jump-when-typing. Don’t think it’s gonna change much. Changing the movement speed just means you can’t move to a new line as quickly, which is a bandaid on the problem. A real solution would be changing the sensitivity. Maybe disabled touch clicks (and only registering push-clicks) would help. Otherwise, I just have to be very cognizant about my thumbs not touching the pad when typing (the setting for “disable touchpad when typing” is already on, but doesn’t work well).
  • Ya, that worked. Disabling tap-to-click means that errant touches don’t misplace my cursor. Well worth the trade of not being able to silently click on something.
• Apr 13, 2019

  • Supercontest.
  • Continued work on the ever-helpful makefile. Added backup-db and restore-db. pg_dump defaults to output a sql text file, which pg_restore cannot read (lol). psql can, tho. Instead, I chose to have pg_dump output a custom file (which pg_restore CAN read, along with dirs and tars).
  • If just performing alterations, then you can connect to the db and modify as you’d like. If doing a DROP -> CREATE (which can include other things like schema and role changes), then there obviously can’t be other active connections to the db. Therefore, the flask container must be down before a restore runs.
  • Made the services and the containers the same name for convenience.
  • Got backup/restore working, but then it failed on a completely fresh postgres image because pg_dump does not include roles. Postgres keeps these at the cluster layer, which applies to all databases. I’ll need to change it to pg_dumpall.
  • Ended up actually not having to backup and restore the roles. Instead, I added docker/database/public.conf env_var back to the postgres image so that initdb would create the supercontest db and the supercontest role. Then my restoration just uses it to fill the data.
  • alter [table/database] [matchup/public.user/alembic_version/pick] owner to supercontest
  • drop role bmahlstedt
  • Added a healthcheck to the database container startup to make sure 5432 was ready. (pg_restore was running before postgres was actually up). JK, just added a sleep after the docker-compose up.
  • Ran some interesting queries on the data. Adding all teams picked by all players for the 2018 season, the top-picked teams were (1) chiefs (2) pats (3) rams. Shows that there’s subconscious bias for better teams, even when the lines are supposedly 50/50.
  • In a makefile, VAR=value doesn’t actually set it until used (lazy). VAR:=value forces it to evaluate and set upon declaration. 
  • I’d been using docker-compose up/down, which removes and creates the containers as well. If you just want to start and stop the existing containers, use start/stop.
  • Dad’s bday. Spoke on the phone for a while.
  • Started the big bbq for tomorrow. Brisket rubbed and put in the smoker overnight, lowest temp (175). Targeting 10pm-6am bark formation, then foil and oven.
• Apr 12, 2019

  • HN/MD.
  • Read a bit about GraphQL again.
  • A curried function takes multiple arguments and processes/returns one at a time, sequentially.
  • Pretty awesome (but decently long) walkthrough of what a db is dong under the hood: https://cstack.github.io/db_tutorial/.
  • Amazon obviously listens to everything Alexa hears (anonymously). https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio.
  • Agile Lite. A more flexible, but still structure approach to sprinting: https://github.com/davebs/AgileLite.
  • Fascinating statistical snapshot of the whole software community: https://insights.stackoverflow.com/survey/2019.
  • Tested out a new stylesheet: https://github.com/kognise/water.css. Was pretty cool, but I kept my old CSS. This (and other just-add-css options) would be nice for future changes, to mix it up for the users.
  • Investments.
  • Nowadays, it’s not just a seed and venture round then an IPO 6 years later. There are many growth rounds, and companies can stay private for longer than a decade, continuing to increase value. Companies stay private longer because it makes the investors more money.
  • Founders and VCs have preferred stock. They have RSAs instead of ISOs. They can create liquidity events whenever they want with another growth round. They incur a lot less risk, while owning a substantially larger share of equity. Employees are just kinda along for the ride (which is not crazy, since employment is opt-in).
  • The liquidity goalposts have moved in the founder/VC favor, but the vesting goalposts for employees have not. Many believe that there should no longer be a vesting cliff because its original timeline was basically to finish at the IPO. Now, that’s no longer the case.
  • If a company does offer liquidity events for employees regularly, these arguments go away.
  • Dilution is real.
  • https://steveblank.com/2019/04/10/startup-stock-options-why-a-good-deal-has-gone-bad
  • Something weird happened with my docker-compose installation.
  • Ran apt install python3-apt because it was missing the apt_pkg. Strange.
  • Tried pip installing some stuff. ln -sf /usr/bin/pip3 /usr/bin/pip
  • Cat /usr/bin/docker-compose. No, docker-compose shouldn’t be a python entry script.
  • Ahhhh. docker-compose is installed from git, not apt. The package on apt is a python package. https://docs.docker.com/compose/install/. Reinstalled normally.
  • Picked up, trimmed, and brined 2 brisket packers and 2 pork butts for Sunday’s GoT party.
  • sudo apt autoremove killed a bunch of stuff again. I’m not going to run it anymore.
  • Some important packages: sudo apt install ubuntu-software update-manager ubuntu-desktop gdm3 gnome-control-center
  • Ugh aaaand it removed the display manager. Unbelievable.
  • Stuck in a login loop. ctrl-alt-f3 to enter terminal.
  • First, ubuntu is the dist based on debian. It’s the OS.
  • On top of ubuntu, there are desktop environments. Unity has been the primary for a long time, but now the default is Gnome for 18. Another option is Xfce (Xubuntu).
  • In addition, there are display managers (login managers). gdm is the default gnome one, and lightdm is a common alternative.
  • I had to reset everything back to gnome manually because autoremove removed it. Ridiculous.
  • `wmctrl -m` shows the desktop manager (window manager). Mine is back to gnome now.
  • `cat /etc/X11/default-display-manager` shows the login manager. Mine is back to gdm3 now.
  • A few more vim improvements.
  • Started using tabs instead of split windows. Just ctrl-t (like chrome) then a filename to open a new one. ctrl-w to close. Then and left or right arrows to move between them.
  • @ at the beginning of a command in a makefile makes it not echo the command itself. Unlike bash, it does print by default.