Mostly Software

  • Saturday

    • Some agile planning for the next few tickets.
    • Remember that alembic is a migration tool. It’s meant only for database revisions. Changes to the schema.
      • I somewhat abuse this for data changes as well. But I only really want this as a record of larger data changes: backfills, important deletions/additions, etc.
      • If starting over, I would probably just keep a separate directory for migrations/scripts alongside migrations/versions. Just like stored procedures.
      • I’ll try to start doing that now. It could be sql that I pass to a DO command. Or a python script that uses sqlalchemy (probably easier, since I have access to the app’s helper queries/commits, and more consistent with the migrations).
      • Yup, I ended up doing this. Added notes to ADMIN.md and created the migrations/data_changes folder with an example python script.
    • Subscribed to PR notifications in the graphql-sqlalchemy repo to check specifically when sa2 support is merged: https://github.com/graphql-python/graphene-sqlalchemy/pull/368. I don’t think you can subscribe to single PRs only.
    • Bought another ember mug – my current one is 2 years old and is very finicky when starting/stopping/charging. The new one is white/14oz vs the old black/10oz.
    • The 3 rise garden nutrients were Sprout, Thrive, and Blossom. You’d add sprout at the beginning, blossom and the end, and thrive throughout all stages. Thrive is a micronutrient blend. Sprout and blossom are macronutrient blends. In the new dry nutrient blends, they combined thrive into both sprout and blossom! So you only add sprout at the beginning (optimized for stems/leaves) and blossom at the end (optimized for fruits/flowers). And you only need to order two thirds as much, so 66% the costs. And they balanced the dry nutrients better for overall health as well as ph, so plants will grow slightly faster/fuller. All around win.
    • SBSC. New teams and statuses tables.
    • https://gitlab.com/bmahlstedt/supercontest/-/issues/175
    • Fixed all the sequences, their data types, and start values in postgres.
    • Added col defaults for all PK ID cols for my 8 tables in pg, set obviously to nextval of the corresponding sequences. 3 of the tables were missing this! Then I can remove it from models, where the sequences were explicitly defined in the ORM layer (saving my butt for those 3). Better to push this to the data layer.
    • Sqlalchemy will create the <table_name>_id_seq and column_default automatically for ID PK INT cols.
    • Also note – this is all because I didn’t really know what I was doing when creating the tables originally.
    • Some detail (a very good general reference for postgres sequence management) as well as explicit commands on this comment: https://gitlab.com/bmahlstedt/supercontest/-/issues/175#note_1292435219
    • Read through a bit of the alembic documentation. Remember to do alembic.op for the database interactions. You can use sqlalchemy for stuff like datatypes eg sa.Column('id', sa.Integer()). Try not to use db from the supercontest – that’s the flask scoped session manager.
    • Another bit of canon: No dependencies on your app within the migrations. It should only depend on the database. This means some convenience queries/commits/utils from your app are unavailable. You don’t have to follow this perfectly. You can use sqlalchemy to bind to the same alembic key and send ORM queries. Or you can use op.execute to run a select and use those results, if you want to stay 100% in the SQL layer.
    • Went through and rewrote the season rollover migration to be a lot cleaner.
    • Made the admin.add_view() for all AdminModelView models dynamic. It inspects the non-FK cols in a table and makes those cols searchable in the admin panel. I was manually maintaining those (breaking often), and copy-pasta of the longer add_view calls. Much cleaner now.

  • Friday

    • Scary – an intentionally-malicious chrome extension to demonstrate risks: https://mattfrisbie.substack.com/p/spy-chrome-extension
    • Lots of private work.
    • Hedgineer: https://www.hedgineer.io/
      • Followed the twitter (https://twitter.com/hedgineering) and subscribed to yt (https://www.youtube.com/channel/UCbulCxi0-MPPw5mHLLlN0Pg). Already following IG. Joined the slack too.
      • All the clips are up: https://www.youtube.com/@hedgineer/videos. Longform full interview soon I think.
    • Followup note from yesterday – I can now (and did) remove the google account from my macbook. It was just adding gcontacts, but now that ContactsSync is keeping icloud/gcontacts in sync on my iphone, the macbook just shows icloud (which is everything).
      • Web icloud and macbook contacts only show 596, but iphone shows 856 (correct, matching gcontacts). Not sure why.
      • Nope – after removing, that delta did result in some missing contacts. Everything fine on phone and google, but icloud/macbook missing some. Added google account back to macbook to be sure.
    • Never really used git stash heavily (weird, I know, it’s a helpful convenience) but played around with it today.
      • push = running git stash with no arguments (similar to create, but not exactly)
      • pop = apply (but pop removes it from the stash list, apply keeps it there)

  • Thursday

    • Amazon closed its acq of onemedical for $3.9B.
    • Lots of private work.
    • Finished the sqlalchemy2 upgrade from yesterday (notes on that post).
    • https://updraftplus.com/faqs/privacy-policy-use-google-drive-app/
    • Switched to the gitlab board view (from milestone) so that I can drag/drop tickets to rank them (order them for next tasks).
    • Read up a little bit about loan seasoning. Lending vs trading. Diff regulations.
    • WR Chess Masters is very even so far after 7 of 9 rounds. Of the 10-player field, 6 are on 3. 2 are on 4. 2 are on 4.5.
    • Moved the bmahlstedt.com domain from godaddy to route53. Unlocked, authed, transfered, approved, relocked, disabled dnssec, added A records to digital ocean droplet IP, confirmed e2e. Fully off godaddy now, all domains registered and DNSed on AWS.
    • Calendar / to-do / notes / lists.
    • Overall this was an exercise in separating my lists into timed events, timed tasks, and untimed tasks.
    • Added a bunch to my gcal so weekly planning is autonomous.
    • Also integrated calendars for niners games, ufc events, and warriors games. There’s always the risk of these losing their maintainers as well. Couldn’t find one for chess, so just created one for my own use.
    • Started using gtasks. Tasks are tasks. They show up individually to be completed. They allow descriptions. I don’t use reminders. Keep all in one place.
    • Took this opportunity to clean up gkeep a little bit as well. Synced the colors with gcal. Moved task lists in gkeep notes (list longterm, cka/cks, etc) to tasks in tasks (separate task lists). Then I can add dates to any individual task and it will automatically show up in gcal!
    • Also means that I switch from month view to week view (as default) on desktop and day view on mobile.
    • There’s also the “schedule” view which just shows the lists by day (instead of cal by time), which looks a lot like my old management of gkeep.
    • So events go in gcal (must have a date or datetime). Tasks go in gtasks (can have no date or datetime). And scratch goes in gkeep.
    • Remember gcal has the right side panel (on desktop) to show gkeep and gtasks. Gmail has the right side panel (on desktop) for gcal, gkeep, and gtasks. Gkeep and gtasks don’t have side panels. On mobile, none of these have side panels. So on desktop, just use gmail as the entry point! On mobile, use gcal and gmail and gkeep and gtasks as separate apps.
    • This also resulted in some cleanup of types. I did events for most things, even if tasks. Now I switched those to tasks.
    • Also I like having gym on gcal – there is the possible to shift the whole schedule (literally all future events) back a day if I miss/skip/etc, but I can also just move one event and keep the remaining schedule. Encourages me to compress and get back to the regular routine instead of pushing everything.
    • Disabled Google account from syncing contacts directly on the iPhone. Don’t need it now that Contacts Sync is manifesting them as icloud contacts.
    • Contacts.
    • Pruned/merged/edited/added/cleaned.
    • Added a bunch of birthdays during the cleanup.
    • Cleaned and exported icloud contacts too. Then deleted.
    • So google contacts is my (only) source of truth. 856 there. 0 icloud.
    • Note that there is no google contacts app for iphone (apple’s intention). So you have to add contacts to apple’s icloud and then sync as desired.
    • To avoid this, I’m gonna use an app. Looks like the canonical one is Contacts Sync.
    • Free tier only does 40 contacts. Paid $8 for lifetime access. Will now autosync all my contacts in both directions.
    • Note icloud autoinjects YOUR vcard from the icloud account into icloud contacts (even if you delete it).
    • Configured to autosync every 5 min. Or whenever I enter a new location.
    • Now I never have to deal with this again.

  • Wednesday

    • Filled both rise gardens. 68 plants growing. Added a little bit of sprout/thrive, ph, water. Removed the nurseries from the app so the “plant more seeds” tasks are disabled (and bc I’m skipping the nurseries).
    • Got some work done on the private company.
    • SQLAlchemy 2.0. General research. SBSC upgrade.
    • All details on the ticket: https://gitlab.com/bmahlstedt/supercontest/-/issues/199.
    • Don’t want to copy here, but there is a LOT of content on that ticket. I’ll link to the main comments below.
    • Deepdived flask-sqlalchemy: https://gitlab.com/bmahlstedt/supercontest/-/issues/199#note_1288605006
    • Review of the primary 2.0 changelog: https://gitlab.com/bmahlstedt/supercontest/-/issues/199#note_1288654017
    • Excellent general reference for sqlalchemy usage: https://gitlab.com/bmahlstedt/supercontest/-/issues/199#note_1288673543

  • Tuesday

    • Titled Tuesday is open to any Fide-titled player (>=2200). Every tuesday. There’s an early and late tourney, 11am ET and 5pm ET. Around 500 people play in each. 11 round swiss. 3+1 blitz. 1st place is $1k in each.
    • Lots of private work.
    • To show the views in a postgres db: select table_name from INFORMATION_SCHEMA.views WHERE table_schema = ANY (current_schemas(false));
    • Did some ticket planning. Brought the scorestreaming/tick task forward while I do this batch of table updates.
    • There’s a luka doncic bot on chess.com now! https://www.chess.com/news/view/play-luk-ai-bot-chesscom
    • Cooked another batch of liver. It smelled a little weird. Being extra sensitive after last week, decided to throw it away.
    • \o to switch output in psql.
    • SBSC. Prepped some data changes.
    • Removed scores.coverer and picks.points. Everything is recalculated from the data in 1st order normal form. Will do the change for gen cols (for results views) next.
    • The removal proved to take longer than expected, mostly due to data cleansing. The coverer column was directly populated from Petty’s gsheets in the backfill. The coverer col was correct, but the underlying scores were NOT mapped correctly (underdog and favorite switched). Works in all recent seasons, because the espn scorestrip maps correctly from home/visitor. But the backfill didn’t have that. So it got some wrong. And with coverer gone, the calculation from swapped source SCORES showed incorrect results.
    • Wrote a couple convenience views, matchups and allpicks.
    • Cleaned ADMIN.md.
    • Altered a bunch of col types from dialect-specific values (BIGINT, TEXT, postgresql.TIMESTAMP, etc) to sqlalchemy generics (sa.Integer/String/DateTime).
    • The frontend had a bug from the last couple releases. Was showing UTC timestamps. Fixed to show PT on the matchups view.
    • Also the asterisks for home teams were gone. Fixed the bug (was the hidden td).

  • Monday

    • https://www.ey.com/en_us/ipo/trends
    • Remember that pyproject.toml is not a poetry config. It’s a general python config, natively meant to define your project. It replaces setup.py, and should be considered at that level. Poetry is just one of the many tools configured in the toml.
    • Planted another round in the humidity caps. Everything is sprouting normally so far (but it’s fresh water, which is the same as the nurseries). I’ll do another humidity-cap-germinate round in the main garden once the water is nutrient-rich. We’ll see if the seeds have the same success rate. They don’t need the nutrients (can’t start absorbing until first set of true leaves), but it may be detrimental to even have nutrients present.
    • I don’t like the way the mona lisa is looking at me, can we give her a quick touchup: https://www.morningbrew.com/daily/stories/2023/02/19/roald-dahl-gets-a-controversial-rewrite
    • Remember poetry show --tree
    • Chamomile and sandalwood in the dehumidifier…amazing.
    • Remember the -C switch to grep for context, to show before and after. Pass a number. For just before or after, use A/B instead of C.
    • Evaluated grpc/protobufs for sbsc: https://gitlab.com/bmahlstedt/supercontest/-/issues/179. Not worth it for those APIs.
    • Looked at some REST extensions of flask. Overall, I just don’t get it. You can build a rest api with native flask. flask-restful certainly doesn’t help with much extra. flask-restplus at least gives you automatic swagger API docs. Both do some of the parsing/marshaling of request/response data as desired. But overall, not worth it imo.
    • Updated some of the rows in the doc Stack.
    • Don’t FK to another col for convenience on the join. It’s canon to just FK to the PK. It’s already unique, blah blah.
    • SBSC. Tests.
    • Finished https://gitlab.com/bmahlstedt/supercontest/-/issues/198.
    • Fixed the old app test. Removed flask-testing, hadn’t been updated since 2017.
    • The test target now produces junit and coverage xml.
    • Both become artifacts in gitlab, which then get shown conveniently on jobs and MRs, historical trends, analytics, badges, etc.
    • Added some unittests.
    • No integration tests for the app yet (including database, webscraping, etc).
    • Gitlab. CICD.
    • Just quick general shoutout: gitlab is an excellent product.
    • Plans.
      • I’m just on free tier right now, which comes with 400 shared runner minutes per month.
      • In order to actually use a plan, you have to associate yourself (as a seat) and your projects (repos) under a group. A free tier group has 5 seats (and unlimited repos, but there are storage limits).
      • It’s $19/user/mo for premium, and $99/user/mo for ultimate. You get more features, more CICD minutes, more storage, better support, etc.
    • I have 5 jobs that run in parallel right now (black, pylint, pyright, bandit, pytest). Each starts with a python image, installs poetry, then installs my project+deps into a testenv. That step is common to all, and takes about 4 minutes on the gitlab shared runners. Rather than spending time optimizing the cache for that right now, I’m going to let it be inefficient; a couple weeks from now I’ll change the pre step to be an image build and pass that to the test steps anyway, which will auto-resolve this.
    • But: These jobs have used a couple hundred minutes in the past few days alone. I’ll need to start a gitlab runner on my desktop, since I don’t have enough shared minutes to last until the month rollover (and even then, until the build change).
    • Did this, and disabled shared runners (and group runners). Instructions below.
    • Download the runner, create gitlab user to run it, run it, and register it with your project’s token.
    • gitlab-runner is managed by init (service <>) rather than systemd (systemctl <>) on my machine (although it supports both: https://docs.gitlab.com/runner/configuration/init.html). It should automatically start on boot.
    • Ended up removing the baremetal (well, WSL2) runner above and went with docker instead.
    • You just start the container with a local volume mount for config: docker run -d --name gitlab-runner --restart always -v /srv/gitlab-runner/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest
    • The restart policy means it relaunches when the docker daemon restarts (so on host startup).
    • And then register it: docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register
    • In order to achieve job concurrency: You don’t need to rerun the “register” portion (although you can, details here: https://docs.gitlab.com/runner/fleet_scaling/). Instead, just leave the one runner process, inside the one container, and have it spawn multiple OTHER containers for each job (obviously each job runs in a separate container, you define an image for each job as desired). Go into /srv/gitlab-runner/config on the host and modify config.toml for concurrent = <>
    • Google analytics. SBSC and Blog.
    • Protected the gtag inclusion from dev. Only prod now.
    • Some insights (for both sites): US #1, china #1. Mostly mobile. Chrome most popular browser. ~2m avg engagement time. Blog gets a little more traffic than sbsc (esp in offseason).
    • Enabled google signals.
    • For SBSC, I’ll have to add my custom user IDs soon (not too hard to integrate). Will wait for the cognito ticket.
    • I don’t really take advantage of many other features (conversions, purchases, user bucketing, etc).
    • And remember wordpress is via the monsterinsights plugin. And SBSC has the flask_monitoringdashboard with some redundant info (although FMD has a full profiler).

  • Sunday

    • Includes saturday.
    • Surprised there’s no way to auto-browser-open an index.html from vscode. At least a context menu option.
    • Remember python -m site to show path information, get location of site-packages, etc.
    • See below for significant comparisons between different docbuilding and dochosting approaches.
    • If there’s already a CNAME for a subdomain in route53, you can’t generate a cert in ACM for the toplevel domain. Delete the CNAME, create the cert, then recreate the CNAME.
    • Unrelated – It’s a little annoying that you can’t add a new name (subdomain) to an existing ACM cert (you have to create a new one).
    • Before you can delete an object in AWS (like a certificate), you have to remove its associations (like a load balancer). For that specific case, you can’t just update the LB to use a new cert. You have to actually go into the LB and disconnect the cert. EC2 -> Load Balancers -> Listeners -> View Details -> Certificates tab -> Remove.
    • SBSC. Docs. Nice: https://docs.southbaysupercontest.com/
    • https://gitlab.com/bmahlstedt/supercontest/-/issues/197
    • Autodocumented the whole supercontest. This involved writing the infra, moving some existing readmes around, as well as documenting every single member of the app.
    • Plugged into lint for autoverification / CI.
    • Switched from google style docstrings to sphinx style, normal rst. More compact.
    • Hosted on RTD. Configured it all. Integrated with gitlab, webhook on change, new version on MRs.
    • Switched to my domain, created DNS records (and certs) for subdomain docs.southbaysupercontest.com.
    • Ended up building with autosummary. No calls to autogen needed. And no apidoc. Wrote a custom template so that :members: of automodule are all populated (it’s just jinja, so you do some pretty flexible logic here). I don’t just want autosummaries, I want the actual autodocs (duh – this should be the default).
    • Now I can (and did) use auto references within docstrings (eg “this function has a param which is the return of this other func”).
    • Vscode ext “autodocstring – python docstring generator” for some easy automation / completion of docstring skeletons. Saves a little time.
    • Remember to use #: (with the colon) to document a module variable/attribute. Otherwise it will not be documented or included in the summaries.
    • SBSC. CI.
    • Made it so .gitlab-ci.yml calls the make targets for: black, pylint, pyright, bandit, and pytest.
    • Black is a special case – I don’t want the pipeline autoformatting changes (nor committing), so we run with --check which simply returns 0 if no changes or 1 if there would be changes.
    • Did a little research into sharing caches between jobs/stages/branches/etc. It’s all configurable. The caches are a little different than artifacts obviously. Artifacts persist. Caches are scoped to specific keys (in your control).
    • You use poetry config virtualenvs.in-project true to make the venv get created locally (which gitlab caches require, can’t go out of the build dir).
    • Implemented the extends keyword to reuse common sections in the CI yml.