Mostly Software

  • Tuesday

    • Titled Tuesday is open to any Fide-titled player (>=2200). Every tuesday. There’s an early and late tourney, 11am ET and 5pm ET. Around 500 people play in each. 11 round swiss. 3+1 blitz. 1st place is $1k in each.
    • Lots of private work.
    • To show the views in a postgres db: select table_name from INFORMATION_SCHEMA.views WHERE table_schema = ANY (current_schemas(false));
    • Did some ticket planning. Brought the scorestreaming/tick task forward while I do this batch of table updates.
    • There’s a luka doncic bot on chess.com now! https://www.chess.com/news/view/play-luk-ai-bot-chesscom
    • Cooked another batch of liver. It smelled a little weird. Being extra sensitive after last week, decided to throw it away.
    • \o to switch output in psql.
    • SBSC. Prepped some data changes.
    • Removed scores.coverer and picks.points. Everything is recalculated from the data in 1st order normal form. Will do the change for gen cols (for results views) next.
    • The removal proved to take longer than expected, mostly due to data cleansing. The coverer column was directly populated from Petty’s gsheets in the backfill. The coverer col was correct, but the underlying scores were NOT mapped correctly (underdog and favorite switched). Works in all recent seasons, because the espn scorestrip maps correctly from home/visitor. But the backfill didn’t have that. So it got some wrong. And with coverer gone, the calculation from swapped source SCORES showed incorrect results.
    • Wrote a couple convenience views, matchups and allpicks.
    • Cleaned ADMIN.md.
    • Altered a bunch of col types from dialect-specific values (BIGINT, TEXT, postgresql.TIMESTAMP, etc) to sqlalchemy generics (sa.Integer/String/DateTime).
    • The frontend had a bug from the last couple releases. Was showing UTC timestamps. Fixed to show PT on the matchups view.
    • Also the asterisks for home teams were gone. Fixed the bug (was the hidden td).

  • Monday

    • https://www.ey.com/en_us/ipo/trends
    • Remember that pyproject.toml is not a poetry config. It’s a general python config, natively meant to define your project. It replaces setup.py, and should be considered at that level. Poetry is just one of the many tools configured in the toml.
    • Planted another round in the humidity caps. Everything is sprouting normally so far (but it’s fresh water, which is the same as the nurseries). I’ll do another humidity-cap-germinate round in the main garden once the water is nutrient-rich. We’ll see if the seeds have the same success rate. They don’t need the nutrients (can’t start absorbing until first set of true leaves), but it may be detrimental to even have nutrients present.
    • I don’t like the way the mona lisa is looking at me, can we give her a quick touchup: https://www.morningbrew.com/daily/stories/2023/02/19/roald-dahl-gets-a-controversial-rewrite
    • Remember poetry show --tree
    • Chamomile and sandalwood in the dehumidifier…amazing.
    • Remember the -C switch to grep for context, to show before and after. Pass a number. For just before or after, use A/B instead of C.
    • Evaluated grpc/protobufs for sbsc: https://gitlab.com/bmahlstedt/supercontest/-/issues/179. Not worth it for those APIs.
    • Looked at some REST extensions of flask. Overall, I just don’t get it. You can build a rest api with native flask. flask-restful certainly doesn’t help with much extra. flask-restplus at least gives you automatic swagger API docs. Both do some of the parsing/marshaling of request/response data as desired. But overall, not worth it imo.
    • Updated some of the rows in the doc Stack.
    • Don’t FK to another col for convenience on the join. It’s canon to just FK to the PK. It’s already unique, blah blah.
    • SBSC. Tests.
    • Finished https://gitlab.com/bmahlstedt/supercontest/-/issues/198.
    • Fixed the old app test. Removed flask-testing, hadn’t been updated since 2017.
    • The test target now produces junit and coverage xml.
    • Both become artifacts in gitlab, which then get shown conveniently on jobs and MRs, historical trends, analytics, badges, etc.
    • Added some unittests.
    • No integration tests for the app yet (including database, webscraping, etc).
    • Gitlab. CICD.
    • Just quick general shoutout: gitlab is an excellent product.
    • Plans.
      • I’m just on free tier right now, which comes with 400 shared runner minutes per month.
      • In order to actually use a plan, you have to associate yourself (as a seat) and your projects (repos) under a group. A free tier group has 5 seats (and unlimited repos, but there are storage limits).
      • It’s $19/user/mo for premium, and $99/user/mo for ultimate. You get more features, more CICD minutes, more storage, better support, etc.
    • I have 5 jobs that run in parallel right now (black, pylint, pyright, bandit, pytest). Each starts with a python image, installs poetry, then installs my project+deps into a testenv. That step is common to all, and takes about 4 minutes on the gitlab shared runners. Rather than spending time optimizing the cache for that right now, I’m going to let it be inefficient; a couple weeks from now I’ll change the pre step to be an image build and pass that to the test steps anyway, which will auto-resolve this.
    • But: These jobs have used a couple hundred minutes in the past few days alone. I’ll need to start a gitlab runner on my desktop, since I don’t have enough shared minutes to last until the month rollover (and even then, until the build change).
    • Did this, and disabled shared runners (and group runners). Instructions below.
    • Download the runner, create gitlab user to run it, run it, and register it with your project’s token.
    • gitlab-runner is managed by init (service <>) rather than systemd (systemctl <>) on my machine (although it supports both: https://docs.gitlab.com/runner/configuration/init.html). It should automatically start on boot.
    • Ended up removing the baremetal (well, WSL2) runner above and went with docker instead.
    • You just start the container with a local volume mount for config: docker run -d --name gitlab-runner --restart always -v /srv/gitlab-runner/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest
    • The restart policy means it relaunches when the docker daemon restarts (so on host startup).
    • And then register it: docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register
    • In order to achieve job concurrency: You don’t need to rerun the “register” portion (although you can, details here: https://docs.gitlab.com/runner/fleet_scaling/). Instead, just leave the one runner process, inside the one container, and have it spawn multiple OTHER containers for each job (obviously each job runs in a separate container, you define an image for each job as desired). Go into /srv/gitlab-runner/config on the host and modify config.toml for concurrent = <>
    • Google analytics. SBSC and Blog.
    • Protected the gtag inclusion from dev. Only prod now.
    • Some insights (for both sites): US #1, china #1. Mostly mobile. Chrome most popular browser. ~2m avg engagement time. Blog gets a little more traffic than sbsc (esp in offseason).
    • Enabled google signals.
    • For SBSC, I’ll have to add my custom user IDs soon (not too hard to integrate). Will wait for the cognito ticket.
    • I don’t really take advantage of many other features (conversions, purchases, user bucketing, etc).
    • And remember wordpress is via the monsterinsights plugin. And SBSC has the flask_monitoringdashboard with some redundant info (although FMD has a full profiler).

  • Sunday

    • Includes saturday.
    • Surprised there’s no way to auto-browser-open an index.html from vscode. At least a context menu option.
    • Remember python -m site to show path information, get location of site-packages, etc.
    • See below for significant comparisons between different docbuilding and dochosting approaches.
    • If there’s already a CNAME for a subdomain in route53, you can’t generate a cert in ACM for the toplevel domain. Delete the CNAME, create the cert, then recreate the CNAME.
    • Unrelated – It’s a little annoying that you can’t add a new name (subdomain) to an existing ACM cert (you have to create a new one).
    • Before you can delete an object in AWS (like a certificate), you have to remove its associations (like a load balancer). For that specific case, you can’t just update the LB to use a new cert. You have to actually go into the LB and disconnect the cert. EC2 -> Load Balancers -> Listeners -> View Details -> Certificates tab -> Remove.
    • SBSC. Docs. Nice: https://docs.southbaysupercontest.com/
    • https://gitlab.com/bmahlstedt/supercontest/-/issues/197
    • Autodocumented the whole supercontest. This involved writing the infra, moving some existing readmes around, as well as documenting every single member of the app.
    • Plugged into lint for autoverification / CI.
    • Switched from google style docstrings to sphinx style, normal rst. More compact.
    • Hosted on RTD. Configured it all. Integrated with gitlab, webhook on change, new version on MRs.
    • Switched to my domain, created DNS records (and certs) for subdomain docs.southbaysupercontest.com.
    • Ended up building with autosummary. No calls to autogen needed. And no apidoc. Wrote a custom template so that :members: of automodule are all populated (it’s just jinja, so you do some pretty flexible logic here). I don’t just want autosummaries, I want the actual autodocs (duh – this should be the default).
    • Now I can (and did) use auto references within docstrings (eg “this function has a param which is the return of this other func”).
    • Vscode ext “autodocstring – python docstring generator” for some easy automation / completion of docstring skeletons. Saves a little time.
    • Remember to use #: (with the colon) to document a module variable/attribute. Otherwise it will not be documented or included in the summaries.
    • SBSC. CI.
    • Made it so .gitlab-ci.yml calls the make targets for: black, pylint, pyright, bandit, and pytest.
    • Black is a special case – I don’t want the pipeline autoformatting changes (nor committing), so we run with --check which simply returns 0 if no changes or 1 if there would be changes.
    • Did a little research into sharing caches between jobs/stages/branches/etc. It’s all configurable. The caches are a little different than artifacts obviously. Artifacts persist. Caches are scoped to specific keys (in your control).
    • You use poetry config virtualenvs.in-project true to make the venv get created locally (which gitlab caches require, can’t go out of the build dir).
    • Implemented the extends keyword to reuse common sections in the CI yml.

  • Friday

    • Very common to write custom sort functions as lambdas (even if many levels of sort), but remember you can go even further if your custom logic has function wrappers or other inputs you need to consider (sorted‘s key param just passes the element of the iterable you’re sorting, one argument). Or if you need to type annotate in a clear way, a custom function is better. Remember: you just need to return a tuple of the numbers that matter, in order. Negate the ones you want to sort in the opposite direction, for that level.
    • Equinox massage. Better quality than massage envy. I got the master therapist session, although not sure the diff between master and normal (assume training/licensing).
    • Lots of private work.
    • typing.cast for list comprehensions and other places where normal : <> annotations do not work.
    • Rise Gardens came out with a membership plan now.
      • Analysis of current costs for 1 triple garden:
        • My current subscription for nutrients is $78 every 4 mo (all-in, including tax+shipping). 200mL pH, 500mL blossom, 500mL sprout, 1000mL thrive. So that’s $234/yr.
        • Then I’d buy seeds as necessary. Say 8x 4-packs, at $12 apiece, every 3mo. Plus (assume) 10% for tax+shipping. So that’s $422/yr.
        • Total $656/yr to stock a single triple rise garden. Then you’d double for 2 triple gardens, to $1312/yr.
      • Perks of membership plan:
        • Membership is $360/yr (I got it for $300/yr with LOYALGARDNER bc first signup).
        • Free shipping on orders >$30.
        • 15% off sitewide.
        • Reedeemable loyalty points.
        • Plus a bunch of others: VIP sales, early access, welcome gifts (microgreens+nursery for me), events, birthday gifts, merch, more.
        • You can add to upcoming boxes at will (or change seeds, etc), just like butcherbox.
        • You get 6x 4-packs every 3 mo. Then you’d add 2x more (to equal the above), at $10 apiece (15% off), every 3mo. Free shipping. Ignore tax (say that’s equal to loyalty point redemption). So that’s $80/yr.
        • You get these nutrients in every pack: 100mL pH, 150g dry nutrient, 150g dry nutrient. Not sure exactly how the dry nutrients convert, or only 2/3, but assume I have to manually double this. So that’s another $78*0.5quantity*3times/yr*0.85discount = $100/yr.
        • Total $480/yr to stock a single triple rise garden. Then for the second garden it’s $320 for seeds (incl15% discount) and $200 for the nutrients. So total $800/yr.
      • So I’m saving >$500/yr, and getting a bunch of other perks for early access, welcome gifts, cheaper spot orders, birthday gifts, loyalty points, etc. Well worth it.
    • Quick sqlalchemy refresher:
      • db.session.query() returns a query
      • query.join() returns a query
      • query.filter() returns a query
      • query.all() returns a list of results
      • query.one() returns exactly one result, or raises an error if there’s 0 or multiple results
      • query.first() returns the first result of all results, or None if no results
      • query.scalar() returns the first element of the first result, or None if no results, or raises an error if multiple rows
    • “Result” above means an object (instance) of the class which defines the model for that result (whatever table you queried). If the query/results span multiple tables/models, then instead of single objects, it will be tuples of objects of the respective tables/models in the query/join.
    • “Element” is just the object if you’re querying a table. If you’re querying a specific col, scalar() will return the value of that col.
    • SBSC. Misc.
    • Cleaned up a ton of the ORM layer.
    • Removed the joins module, made explicit in the queries module.
    • Removed all type:ignore instances except the flask_user addition of user_manager to flask.current_app. Will be fixed later when I undepend on the yanked flask_user package.
    • Updated all my stubs. Just : Any specs for any objects (vars, classes, functions), structured in the module structure of the package. I’m not managing any members, args, params. Not worth it.

  • Thursday

    • Some private work.
    • Reorganized all the cabling in the bedroom, ~10 connections. Much cleaner now and much stabler for the remotes.
    • Did some elimination testing for the food poisoning over the last couple days. Not smoothies or my other usuals. Was either the batch of liver or the new hot sauce (clark+hopkins florida). Threw away both in an abundance of caution.
      • This has happened ~twice in two months now. If it happens again, then I’ll be safe and cook the liver more (right now it’s around medium rare).
    • Desktop/WSL2 tasks.
      • Windows cumulative update patches as well as Corsair iCUE.
      • Created a few custom murals and device layouts for iCUE. Simple watercolor (dynamic) is probably my favorite. The audio lighting is pretty cool tool, following your music.
      • Uninstalled sonic studio. The only programs that should be running in the tray are iCUE and docker desktop (both started automatically at startup as well).
      • Full startup list: spotify, chrome, icue, docker desktop. Others: chrome (not technically on the list, but I always leave it open so it reboots), expressvpn, sshd/agent, etc.
    • Planted 2 chrysanthemums and 2 lettuces directly in the rise gardens, skipping the nursery (like aerogarden). I bought separate humidity caps. Water level looks good, and this obviously keeps fresh water (vs stagnant nursery requiring weekly change). Only difference (and remaining concern) is nutrients. The garden has higher levels, the nursery has none. AFAIK, seeds simply don’t absorb nutrients, rather than being harmed by them. So I’m hopeful. Overall, if this works, it should be much faster/easier/cheaper/simpler without nurseries.
    • Poetry version updating (comparing HEAD to your lockfile).
      • poetry show -o
        • Prints all the versions that are behind latest.
        • Does NOT respect resolution. Ie if one of your deps pins lower than latest, this command will still show latest.
      • poetry update --dry-run
        • Prints all the versions that are behind latest.
        • DOES respect resolution. Will show what your lockfile updates to (exactly) with poetry update or poetry lock.
    • SBSC. Misc.
    • Typing. Fully finished.
    • Snuck in the change to color percentage cells above/below/equal 50.
    • Updated deps. Only minor and patch semvers, no major.

  • Wednesday

    • 5/7 days into the airthings calibration. CO2 modulation is good with the window. Low humidity though. Bought a humidifier.
    • Still feeling terrible after yesterday.
    • Little bit of private work.
    • Spacing guide for rise garden: https://support.risegardens.com/en_us/how-to:-space-out-your-gardens-B1yWSGyqc
    • Great little Shel Silverstein story: https://www.slideshare.net/cyaneum/the-missing-piece-meets-the-big-o
    • For gunicorn python autoreload in dev mode, I notice that often I have to restart twice (just save twice) for changes to take effect, not just once.
    • operator.itemgetter and attrgetter are excellent for basic sorts, but if you need to negate values (or do any calculation of any kind on any sort level), then just build a custom lambda.
    • A note on general building blocks, dict vs defaultdict. Use dict when you have expected keys, of course. Don’t just use it for coding convenience, because you can have unexpected results from the parents/callers.
      • Eg for sbsc game statuses, use a dict. For something like a user_id key (unknown, could be many in the future, unbound), use a defaultdict. Even for weeks, which can change / be dynamic.
      • Also: if the defaultdict has an value lambda of something like a list or a dict, that’s safer. A lambda of an int (which defaults to 0) or another value might be more misleading. An empty list will rarely break the caller silently (who would have gotten a keyerror). If the caller gets a 0.0 instead of a keyerror, that might break silently.
    • My sally lightfoot crab (adult) caught and ate my red fairy anthias (~3”) alive! I reached in with tongs to separate them, but the crab was able to scurry away to safety under a rock with the wriggling fish still in its front claws.
    • SBSC. More restructuring of the results module.
    • Very clean now.
    • Separated by week_results, season_results, and alltime_results. Each managing multiple views, as appropriate. Each following the same flow: query -> organize -> calculate -> sort -> inject -> toprow.
    • Did a little deepdive into conditional typing. You can use union | to indicate that a function can return multiple types, but what if the expected return is directly linked an input param? Eg return_email=True vs the default of returning user IDs. Use typing.overload! Details on ticket.

  • Tuesday

    • Food poisoning again. Bed at 130 last night, woke up at 330 to puke (and stayed in there straight until ~830). Couldn’t do much today.
    • Little bit of private work.
    • Finished the setup of the v2 charchoal. Deep cleaned both triple families. Planted all nurseries.
    • Pro chess league started today. And warriors’ last game before allstar break. We’re finally getting into the paint.