• Friday

    • Added vscode shortcut to shell:startup.
    • Bought an air purifier (levoit core 300). I have the airthings for sense, now this provides a control (along with window).
    • Private work.
    • Airthings masters finished. Winners bracket winner won the grand final in all 3 divisions: magnus, fabi, sevian.
    • Received and setup the wave plus. Pretty cool. Air quality is good to start, but sensors need some time to calibrate.
      • Global radon map is here: https://radonmap.com/
      • To check air quality: open the mobile app, wave your hand in front of the device (only red/yellow/green response), or the web dash: https://dashboard.airthings.com/
    • SBSC. Finished the typing ticket from yesterday.
    • Split the “by user and by week” defaultdicts in the results module into separate dicts, one for pick status counts and one for teams (they were joined before, which just complicated things – readability, default behavior, types, etc).
    • Use dict subscription when all keys have the same type and all values have the same type, eg myDict: dict[str, int]
    • Use typing.TypedDict when known keys have known types. Use typing.NotRequired to specify keys as optional.
    • Black defaults to 88 line length. Pylint defaults to 100. I changed both to 100, and synced my vertical rulers in vscode.
    • Good reference on writing type stubs: https://typing.readthedocs.io/en/latest/source/stubs.html
    • More details on ticket.
  • Thursday

    • Lots of private work.
    • NBA trade deadline. All official transactions here: https://www.nba.com/players/transactions
    • Laser tag at Area 53 in DUMBO.
    • Updated my address (nyc) and payment method (zelle) in the class action for: Yahoo Data Breach Settlement.
    • Watched knock at the cabin. I love single setting movies.
    • Keyboard was working ok today after ~24hrs of drying. I still wanted a new one anyway, so I’ll leave the V2 brown as my backup and set V3 blue as my primary.
    • SBSC. Typechecked and annotated the whole package.
    • https://gitlab.com/bmahlstedt/supercontest/-/issues/183
    • Compared mypy, pytype, pyre, and pyright. Ultimately chose the latter. Has inference, supports py3.11, doesn’t ship with a bunch of deps, and very popular.
    • Used pylance vscode extension (pyre-vscode for pyre).
    • All of these use python’s native typeshed under the hood: https://github.com/python/typeshed
    • All speak LSP (language server protocol), the bridge between the IDE and the running code (provides autocompletion, typechecking, etc). Eg between vscode and a python executable.
    • The pyright cli uses Node.
    • Type stubs are .pyi files. They’re the public contract for a python file.
    • You may run pyright --createstub <import> to generate the stubs, or in vscode just hover->quickfix->createstub. They’re organized by module.
    • The default stubpath is ./typings. If that doesn’t exist, it will try to infer from code usage (if useLibraryCodeForTypes = true). This should be committed to VCS.
    • Some third-party packages ship with them, some don’t. Some have a separate <pkg>-stubs (I don’t like that, just ship with the pkg), some have no typings at all (and you have to generate them for the interfaces that you use, if desired).
    • For first-party packages, you shouldn’t need stubs. Your code should be directly annotated with types. You may do this with the same process as the above, by using pyright to generate the stubs, and then using another tool like merge-pyi or libcst to autoannotate your code from the types in the stubs. But you have direct access to the code – it’s usually best to go through and just incrementally add annotations.
    • Went through and started with very lenient scope, then slowly ratcheted the strictness while adding annotations and fixing bugs.
  • Wednesday

    • Went through google password manager and did a cleanup. 195 passwords. 0 compromised, 0 unused, 7 weak (all from other people).
    • Lots of private work.
    • Looked at AWS devtools:
      • https://gitlab.com/bmahlstedt/supercontest/-/issues/194
      • CodeCommit, CodeBuild, Code Artifact, CodeDeploy, CodePipeline.
      • CodeStar is the quickstart for all of those.
      • Just like gitlab, which has a full source->prod flow (for free).
      • Just like github actions.
      • Just like bitbucket+bamboo in the atlassian world.
      • What AWS devtools does not have is a ticket system. Gitlab/github/jira all do.
    • Whitney testified in congress (the House Committee on Energy and Commerce) for legislation around satellite comms technology: https://www.youtube.com/watch?v=Btx6FUP24jM. David Goldman (Senior Director of Satellite Policy, SpaceX) testified as well.
    • In Biden’s SOTU last night he asked for congress to increase his IRA’s 1% buyback tax to 4%. Absolutely insane.
      • Of the two primary liquidity opportunities for shareholders (dividends and tenders), the act is meant to discourage buybacks and push volume over to dividends as that generates more tax income for the govt (as Levine posted about today).
      • That’s ridiculous though – dividends are taxed as income, buybacks are taxes as cap gains. Both are taxed appropriately, and according to first principles already. To double tax tenders is just silly.
    • Received replacement parts for rise garden and the charcoal v3 extension. Now just missing a single screw (gonna skip this one, the top is stable with the other 3), hole covers, and pump inlet foam. Ordered the latter 2.
    • Poured an entire cup of coffee on my keyboard. I had a Code V2B 104-key, Cherry MX Brown. Don’t need the numpad at all, I can go back down to 87-key.
      • Of the Cherry MX switches (gold standard), first decide linear, tactile, or clicky (order of sound/force).
      • Linears: silver -> red -> nature white -> black -> grey.
      • Tactiles: brown -> clear -> grey
      • Clicky: blue -> green -> white
      • There’s a new brand Zeal that makes a few switches called Zealios. They’re supposed to feel really good.
      • Ended up going with the Code V3, 87-key, and cherry MX blue switches.
    • Could do some customization like the below (lol) if you order directly from the Code (WASM) website.
    • SBSC. Networking.
    • Created a single A record to point to my EC instance’s elastic IP, so there’s a single source of truth. Then any other record that wants to forward to the machine (directly, like for ssh) instead of the root domain name for website access (via the ELB) can just use an A record to point to the IP’s A record.
    • And an A record to point from the www subdomain to the root domain (in supercontest, can do this directly in route53; in privateer, do this through amplify – it creates the route53 A record to amplify’s cloudfront for you).
    • Explicitly added another listener to the load balancer which redirects all http/80 traffic to https/443 (the other listener, which forwards to my target group, which forwards to the EC2 instance). Can’t do this through DNS, of course.
    • Could just have the ELB continue to forward to 80 on the EC2 instance, but remove the nginx container and have the gunicorn app container ports: 80:8000
    • But again, better to front gunicorn with a proxy (other than ELB, which is handling the load balancing and the cert/decryption). Even though I’m not serving static files through nginx (using cloudfront instead), it is still good for buffering and other custom proxy configs.
    • AWS (all accounts). Did some billing analysis.
    • Here’s an average 3 months before the network changes
    • EC2 and route53 are expected and fine.
    • Cloudwatch seems a little expensive. And not sure why it’s only happening in the second half of every month. Maybe it’s free up to a certain number of metrics transmitted, then starts charging.
      • It charges for custom metrics (30c/mo), dashboards ($3/mo), alarms (10c/mo), logs (including the logs from just running the cloudwatch agent), more. Pretty expensive (relatively). Mine is about $2.60/mo.
      • Deleted the dashboard and changed the disk_used_percent to only push for / (it was pushing for all mount points, which is 29 custom metrics!
    • The other surprise is ELB. Over the past two days since starting, it’s charging about 45c per day. It’s the offseason with no traffic. This should be pennies.
      • This is just to HAVE the load balancer active. It’s ~$18/mo just to run one in us-west-1, then traffic costs on top of that.
      • ELB is crazy expensive if you’re just using it for https. It’s just a cert. It only really makes it worth it if you’re serving multiple sites with the same cert, and/or using the actual load balancing functionality to proxy to many many instances.
      • Therefore – if you move the app to serverless (lambda per request, fronted by API gateway) or (eks/ecs with nginx ingress), you don’t have to pay for the load balancer anymore.
    • There are some pretty cool cost anomaly detection and budging-reaching notification services.
    • Updated my root aws account’s default console layout.
  • Tuesday

    • Placed all positions from the td deposits last week, not that they’ve cleared.
      • Remember that (even after a cleared deposit) you go on margin until the trades settle (<=2d), but you don’t accrue interest on that margin.
      • And remember that at the end of every day, they’ll sweep cash automatically into an FDIC insured deposit account which accrues interest for you.
      • Although this account is not covered by SIPC (Securities Investor Protection Corporation), which helps recovery/return of assets when a broker fails.
      • Enabled futures and forex.
    • Bought an airthings device, help support the chess sponsor.
      • And since I work from home every day, would give good insights for window/sleep/etc.
      • Between the wave plus and view plus.
      • The view adds a display, an internal hub (wifi instead of just bluetooth, and connect multiple devices), and particulate matter sensing.
      • You can buy the hub separately, but I have a small home and only need to check it when I’m there, so I don’t need a hub.
      • And I don’t need a display, I’ll just use the app.
      • And PM is less important than the other metrics.
      • Wave plus is $130 after $100 airthingsmasters discount, view plus is $300.
      • So overall, went with wave plus.
    • Lots of private work.
    • SBSC.
    • Lots of cleanup of SBSC from the past 2 days. Finished a ton.
    • Created milestone to finish before the offseason pause: https://gitlab.com/bmahlstedt/supercontest/-/milestones/7. Organized all tickets.
    • Created board to track as well: https://gitlab.com/bmahlstedt/supercontest/-/boards/5334642?milestone_title=2023%20Q1%20Part%202
    • Boards in gitlab are nice because you can order (the hidden “rank” propery in jira). But they don’t have good “status” columns. It’s just open and closed. The “labels” are meant for static ticket properties (like “this is a bug” – basically a jira ticket type).
    • Created milestone for 2023-2024 season: https://gitlab.com/bmahlstedt/supercontest/-/milestones/8
    • Created a bunch of tickets with AWS plans for these cloud splitouts.
  • Monday

    • “Today, Gemini reached an agreement in principle with Genesis Global Capital, LLC (Genesis), Digital Currency Group, Inc. (DCG), and other creditors on a plan that provides a path for Earn users to recover their assets. This agreement was announced in Bankruptcy Court today. This plan is a critical step forward towards a substantial recovery of assets for all Genesis creditors. In addition, Gemini will be contributing up to $100 million more for Earn users as part of the plan, further demonstrating Gemini’s continued commitment to helping Earn users achieve a full recovery.”
    • There are 18 EGOT winners.
    • YouTubeTV is $65/mo.
    • Remember by default, containers build/run as root, and execute processes within as root. You can change these to a specific user for even more control/security.
    • Temporary issues with local servers for archive.ubuntu.com:http:* today, apt was installing packages incredibly slowly (or failing).
    • Couldn’t find a way to unsubscribe from uber email notifications (they duplicate all the time).
    • “Google Invests Almost $400 Million in ChatGPT Rival Anthropic”
    • The BMW M1000RR is so sick. 205hp, 420lbs, 4cyl inline, 1000cc, $38k MSRP.
      • For reference: The HP4 was 215hp, 320lbs, 4cyl inline, 1000cc. It was around $85k IIRC. Fully carbon. Not street legal, track only. They only made 750 units in 2017.
    • You can nest executions within an f-string, like f"{foo:{bar}}" (eg dynamically padding one variable with spaces according to a width set by another variable).
    • Airthings masters kicked off today.
    • Champions chess tour page got a rewrite, looks better (much more like chess.com now that it bought chess24/CCT): https://championschesstour.com/schedule/
    • They changed the rules *substantially* this year: https://championschesstour.com/format-regulations/
    • 6 events the tour finals, each having a play-in and then a knockout. $2m total fund.
    • Play-ins are 10+2 9-round swiss, knockouts are 15+3 double match elimination (loser bracket winner must beat winners bracket winner twice to win).
    • There are 3 completely separate divisions in knockouts. Div 1 has 8 players and knockout matches are 4 games in the winners bracket and 2 games in the losers bracket. Div 2 has 16 players and knockout matches are 4 games in the winners bracket and 2 games in the losers bracket. Div 3 has 32 players and knockout matches are 2 games in both brackets, with the exception of the final being 4 games.
    • Placement in knockout divisions is according to rank in last event’s knockout as well as rank in this event’s play-in. There are weekly qualifiers for 3 non-GMs to get into the play-in.
    • SBSC. Switched from uwsgi to gunicorn and moved the proxy infra to ELB.
    • https://gitlab.com/bmahlstedt/supercontest/-/issues/186
    • Requested a new certificate from ACM, verified via DNS (you have to click the button to create the CNAME records in route53 to prove you own that domain).
    • ACM public certs are free. Target groups are free, obviously. The load balancer itself charges on use/traffic.
    • Sidenote – accidentally deleted privateer’s cert, had to reissue from amplify (remove-readd domain).
    • No longer need the infra containers. nginx-proxy and letsencrypt are gone.
    • Python apps speak wsgi (web server gateway interface), of course, but most proxies nowadays can speak this as well (including nginx). They’re not just static file servers nowadays.
    • Remember sockets are less overhead than ports.
    • uwsgi has a stats server but I don’t use it: https://uwsgi-docs.readthedocs.io/en/latest/StatsServer.html
    • You can actually embed your entire app in the uwsgi binary! https://uwsgi-docs.readthedocs.io/en/latest/Embed.html. But still easiest to have the single artifact be a docker image.
    • Docker comes with 3 networks predefined that can’t be removed: bridge, host, none.
    • Was getting 502 bad gateway in prod. Traced to uwsgi segfaults. Tested and isolated to SC_DEV. In flask debug=False mode, something was breaking with uwsgi. Hence the switch to gunicorn.
    • Also from the uwsgi docs: “The project is in maintenance mode (only bugfixes and updates for new languages apis). Do not expect quick answers on github issues and/or pull requests (sorry for that) A big thanks to all of the users and contributors since 2009.” And gunicorn is more popular, more support.
    • I ended up still fronting gunicorn with an nginx proxy, but now it’s a simple nginx base image that I copy a custom config in for. No more third-party dependencies on jwilder and letsencrypt.
    • SSL is now handled by ELB, which has a cert from ACM attached to the HTTPS listener. You can’t just associate a cert with an EC2 instance. You also can’t copy a cert generated from ACM into your nginx config. You just front with ELB, hence the change. Autorenewal.
    • Workers and threads (I’m using gthreads in gunicorn, not sync) are 2*NUM_CORES+1. This is set dynamically, and can read fine in both WSL2 and EC2.
    • Remember that although each gunicorn worker starts my flask app in a separate process, instantiating a new apscheduler, they all share a centralized store (no duplicate scheduled jobs).
    • Here are the default variables that flask injects into the global jinja context: config, request, session, g, url_for(), get_flashed_messages(). I use all.
    • Docker creates a bridge network by default, and exposes it to the internet. The external key is not for connectivity. It just means that the network is managed outside of that compose file.
    • expose means it’s available to other containers in docker-compose. ports means it’s available to the host (and therefore anything that can access that host, which for a webserver, is the whole internet).
    • Best toplevel docs for nginx: https://nginx.org/en/docs/dirindex.html and https://nginx.org/en/docs/varindex.html
    • Nginx returns a 302 redirect for my reverse-proxy-to-gunicorn-container config, so I had to tell AWS that the target group considers both 200 and 302 as successful. Otherwise the health check will declare failure, and the load balancer won’t forward requests to an unhealthy target.
    • Full path:
      • Route53 sees the domain request and has an A record to send to ELB.
      • ELB has a listener on HTTPS/443 and an associated SSL cert from ACM.
      • The listener forwards the decrypted requests to a target group configured to receive HTTP/80.
      • The target group forwards to port 80 on the EC2 instance (within the VPC).
      • That EC2 instance is running an nginx container, which forwards 80 from the host to 80 within the container.
      • Nginx proxy passes the traffic to the exposed port 8000 on the gunicorn container.
      • The gunicorn container handles the request (passes to my application, defined in the flask microframework).
    • The load balancer is governed by a security group that allows all HTTP/S traffic from all IPv4 addresses.
    • The EC2 instance is governed by a security group that allows HTTP traffic from only the load balancer (security group), as well as ssh (from anywhere) for dev.
    • Kept an A record in route53 to the elastic IP so that I could ssh to a static address for convenience (my manual work, ansible, etc).
    • Remember docker compose will add the name of the services to the network so they’re accessible by other containers in the same network. Eg: My compose file defines supercontest-app, and the nginx container can reach it at http://supercontest-app.
    • Health checks to a target group do not count as load balancer capacity units. You’re not charged for these.
    • SBSC. Redid all stream management into a proper logging implementation.
    • All print() calls gone.
    • Every sbsc module logs individually.
    • Inspects the logging namespace and handles other loggers too.
    • Strips the gunicorn loggers, which have a stream handler which swallows everything, and instead has them propagate up to the root logger (like everything else).
    • I then manage the streamhandler and formatter on that, giving full control.
    • SBSC. Other.
    • Sped up the dockerfiles. It now copies the pyproject toml and poetry lock ONLY into the container before install and the whole source dir, so that docker can cache it and only pip reinstall (the long part) when the lock/toml change.
    • Reorganized the dockerfiles and server configs. All are in their appropriate subdirs now, and build contexts are relatively defined.
    • Significantly cleaned up the app factory pattern of init.create_app(). It now splits by cli-mode, dev-mode, and prod. Lazy importing only when necessary. Dynamically logging.
    • This cleanup fixes a lot. Example: the flask cli runs your factory to access your app. So it prints everything, sets up caching/minification/scheduler/etc – all the stuff it doesn’t need. That’s now fixed.
    • This means flask shell and flask db <> are both faster, and don’t log garbage to console.
  • Sunday

    • Saw wicked on broadway.
    • My ranked order of all the ones I’ve seen so far:
      1. Book of Mormon
      2. Moulin Rouge
      3. Harry Potter
      4. Rock of Ages
      5. Lion King
      6. Hadestown
      7. Wicked
      8. Aladdin
      9. Music Man
      10. Six
    • Remember intel is arm and apple is amd.
    • tail -f /dev/null as docker entrypoint/command to keep container running.
    • Properly ran pyenv on wsl2. Installed the python build deps for a clean pyenv python install. Did some cleaning with apt autoremove. Now running python 3.11 on wsl2, macbook, and in all containers running sbsc. For all in both pyenv and poetry env, not system env.
    • Installed the Better TOML and Mako vscode extensions for syntax highlighting on those configs.
    • Remember that docker down removes volumes as well. So you’ll have to restore the database if you down. Instead, it’s better (in most cases) to just use docker stop.
    • poetry export can be used to output a standard requirements.txt from the lockfile.
    • SBSC. Upgraded ALL package versions.
    • https://gitlab.com/bmahlstedt/supercontest/-/issues/180
    • https://gitlab.com/bmahlstedt/supercontest/-/issues/184
    • Python 3.8 -> 3.11.
    • Flask 1 -> 2.
    • SQLAlchemy 1 -> 2.
    • Upgraded literally every other package. All flask extensions, all utility libraries, everything. Removed bs4 and decorator. Everything is unpinned in pyproject.toml (locked by lock, of course).
    • Removed flask-script and moved over to flask’s (now native) CLI.
    • Got flask-monitoringdashboard working again.
    • Took advantage of the new py features: enum, walrus operator, switch statements, more.
    • Removed the dependence on joyzoursky/python-chromedriver, which stopping uploading a couple years ago (since py3.9). Built my own image, basing from selenium. I then layer pyenv, poetry, and my packages in.
    • This was quite the chore. Chrome is not seamless to install (tried to base from py3.11 first). Details on ticket.
    • So multiple layers of isolation are now working: host machine -> isolated docker env (with chrome/chromedriver/selenium) -> isolated python env (via pyenv) -> isolated project env (via poetry).
    • The client js reimplemented some logic for arePicksOpen() and unstartedStatus/finishedStatuses. These are now ONLY serverside-defined. Single source of truth.
    • Exposed week/season as an arg for the scrapers to test them outside the current week. Looks good for next season.
    • Selenium has a cool element.submit() which walks up the dom and clicks the form’s submit button. Better than finding the el by xpath/whatever and .click(), which has “Element is not clickable” errors sometimes (viewpoint, scroll, elements over it, etc).
    • flask-user is yanked. The maintainer is no longer developing on it, but I’m not sure why the yank. flask-security was forked in 2020 and is maintained by someone else now. flask-login doesn’t have everything I want. So I just kept flask-user for now.
    • Made some lint and style improvements, corresponding to newer versions of python/stdlib/pylint/black.
    • Fixed all logos in the stack description. Got off clearbit. Made the readmes linked, so you don’t have to keep 2 sources of truth.
    • Wrote some super helpful new gnumake targets to drop into various shells with various contexts (active flask app, env vars, etc). This organized things a bit better too; docker compose is now only used for targets that manage container state. Everything else just uses docker.
    • Note: upgrading the version of flask-migrate works on its own if you don’t already have a migrations/ folder. If you do, the site-packages update is not sufficient; your local env.py and alembic.ini and script.py.mako will be out of date. I updated them manually from github src.
    • Moved all constants into supercontants.core.constants. Much better now.
    • Cleaned the explicit template argpassing, jinja globals, and jinja exposing vars to js.
    • Changed DB statuses to new format.
    • Linted all js.
    • Made version management of the sbsc package (itself) single-source-of-truth, instead of duplicating for sentry-sdk and flask-monitoringdashboard. Now, just pypoetry.toml need bumping.
    • Cleaned some custom functions, just used configparser from stdlib.
  • Saturday

    Python version history comparison today. Scraped the releases and decided what version to run my apps on, and what new features to add to them.

    3.0 (2008-12-03)
    • print statement -> function
    • dict.iteritems() -> dict.items()
    • raw_input() -> input()
    • many things return iterators instead of lists (zip, map, keys, etc)
    • unicode string no longer necessary, assumes it
    • string formatting, starting to deprecate % and use format
    • using with for context management
    3.1 (2009-06-27)
    • collections.OrderedDict added
    • collections.Counter added
    • unittest can be marked skip and expectedFailure
    • .format() can automatically add commas in the thousands places
    3.2 (2011-02-20)
    • Stable ABI (easier to write extensions across versions)
    • argparse replaced optparse, which only had support for positional args
    • logging config by json file
    • concurrent.futures for thread/process management
    • pyc moved to __pycache__
    • wsgi headers need to be native strings, no encoding
    • @functools.lru_cache added
    • itertools.accumulate() added
    • contextlib now allows contextmanagers to be used as function decorators as well
    • can now use a .pdbrc
    3.3 (2012-09-29)
    • added venv as a stdlib module, for programmatic access from python, as well as command line access via pyvenv (and then python -m venv)
    • added mock to unittest in stdlib
    • added py launcher for windows (double clicking .py automatically runs them)
    • yield from to delegate generators
    • __qualname__ for fully qualified path to that object
    3.4 (2014-03-16)
    • asyncio added
    • enum added
    • pathlib added
    • statistics added
    • tracemalloc added
    • pip is shipped with python now, always
    3.5 (2015-09-13)
    • typing (could use comments and annotation before, but this formalizes)
    • coroutines can be used with async / await syntax (diff syntax before)
    • can multiply matrices with A @ B instead of dot(A, B)
    • much of stdlib reimplemented in C, improving perf significantly
    • os.scandir() added (better than the os.walk function)
    • math.inf and math.nan constants were added
    3.6 (2016-12-23)
    • f strings
    • dicts use 25% less mem
    • underscores in numbers
    • async generators and async comprehensions
    • typing for variable annotations
    • secrets module added to stdlib
    • asyncio was introduced in 3.4 as provisional but is now declared stable
    3.7 (2018-06-27)
    • types now have postponed evaluation; basically annotations allow hoisting
    • breakpoint() added (internally, this calls sys.breakpointhook(), which calls pdb.set_trace())
    • the time module now supports nanoseconds (for most functions)
    • typing was introduced in 3.5 as provisional but is now declared stable
    • added dataclasses
    3.8 (2019-10-14)
    • walrus operator
    • / in function param means it MUST be positional (can’t be keyword)
    • using = in an fstring will insert not just the value of the var but also the name of the var (really helpful for printing and debugging)
    • async mocks were added to unittest
    3.9 (2020-10-05)
    • dict merge | (same as {**d1, **d2}) and update |= (same as dict.update) operators
    • graphlib added to stdlib
    3.10 (2021-10-04)
    • much better error messages in standard exceptions (more specific syntax)
    • structural pattern matching (switch statements!)
    • type union operator: you can do int | float instead of Union[int, float]
    • distutils is deprecated, and will be removed in 3.12 (just setuptools now)
    3.11 (2022-10-24)
    • all around 10-60% faster, average 1.25x, uses slightly more mem tho
    • added tomllib, can parse toml now with stdlib
    • even more specificity in tracebacks, pointing to exact problem
    • multiple exceptions can be raised simultaneously in groups
    • ton of deprecated stdlib modules and functions removed (none that I really use)