-
- Private work.
- Back to nyc.
- JS notes in gdoc.
- Link another credit card save/connect tool.
- Naiad, canid, canna, dacha.
- Linkedin premium is ~$30/mo (individual).
- Leetcode premium is ~$11/mo (annual sub, cyber monday deal). Dynamic layout, autocompletion, debugger, access to premium problems, access to some editorials, faster execution.
React.memo.- Supercontest.
- AWS Billing and Cost Management.
- Enabled cost optimization suggestions. Creates a role to access your other services, collects usage, gives realtime updates.
- Opted into and played with Compute Optimizer.
- Created a cost monitor and attached subscription (my email). Sends daily summaries and alerts if >50% above expected (a single day, I believe).
- Created 4 budgets: 1 for monthly spend under a certain amount (actual AND projected), and 3 for each RI ensuring that utilization of the RI is 100%. All 4 email me if triggered.
- VSCode SSH Remote.
- Installed all extensions in the sbsc remote. Remember this is an installation on the remote.
- It bricked the EC2 instance. Not sure which extension. Maybe one of the windows-specific ones?
- Rebooted from EC2, connected, removed all vscode extensions (kept them local, uninstalled from ssh:sbsc).
- Remember the disconnections could have been from my poor internet too. Just wait a few.
- Remember you can
make build-start-prod without losing any data, even if you restart or stop-prod manually, and that’s how it’s always been. BUT it USED to be the case that make down-prod would wipe the DB volume. Now that’s clean too – it persists in RDB, regardless if prod docker composition (just app+nginx) is downed.
- ElastiCache.
- Went through the app and added some view function
cached and regular function memoize decorators.
- Played with an initial memoization from flask-cli. Nothing was persisting. Ah – it’s because CLI uses NullCache. Changes so the prod flask cli connects to the prod elasticache (like it already does with rds).
- Note: you cannot traverse sqla relationships from a cached object. The parent obj is not bound to a session, the dependents are dangling.
- You can change this to subquery or selectin loading (rather than the default lazy, which queries at ACCESS time)(https://docs.sqlalchemy.org/en/20/orm/queryguide/relationships.html) or manually load all children on the dangling object with new queries, but then we defeat the whole purpose of a cache.
- Memoized most of the functions in
queries (as long as they return something, and something other than direct sqla objects). This should speed up some stuff like get_current_week(), is_regseason(), etc. TTL set to 30, since the intention for the memoization is intra-request optimization, not inter. Will do that with view caching next.
- Also – you don’t really want to cache anything outside of query libs. I have lots of functionality in utils and other libs. But the point of the cache is to optimally save a db trip. If the other logic is a local compute op, up to a reasonable ceiling of time complexity, it’s faster than network-hitting redis anyway.
- Added caching for the main view functions. Set TTL to 60 to match the frequency of the score fetch, the primary cache invalidator.
- Skipped caching for API endpoints, stuff like
/email-all-picks and /commit-scores.
- Created followup ticket for event-driven cache invalidation (based on
supercontest.dbsession.commits), rather than blanket time-driven TTLs right now.
- Deployed to prod, tested, ready for MNF.
- Created a new cost report for Amortized blending (takes the RIs and extends their costs in the daily/usage expenses). And then filtered RIs out of the main cost report for daily service totals.
- Added paid users, now to 20. Expect 27 total.
- There was an infinite recursion in the function that adjusts purses for ties. This occurred when there were only 4 people in the paid league. Not sure which ranks were tied.
- Will debug this next time I see it.
- Enabled some aws cost mgmt prefs: multi-year lookback at monthly granularity (up to 38mo), daily granularity for all services up to 14 day lookback (from 7, default).
-
- Private work.
- Some more refreshers on (pandas, psql, dsa, js). Remember postincrement
i++ (eval to val then increment) and preincrement ++i (increment then eval to val). And == vs === for strict type. Error. new.
- AWS merge Billing with Cost Explorer -> Billing and Cost Management.
- In redis, just
EXPIRE <mykey> <ttl>. You can check with TTL <mykey>. Then flask-caching obviously allows you to plug into the redis api.
- Supercontest. Found a bug in the parsing of the espn score API. Added to the ticket where I’m swapping those out.
-
- Private work.
- Gitlab-runner runs on macbook even when vscode is closed; it’s running the container whenever the host is up. Remember it directly runs – not through a ubuntu container on wsl2 for my desktop.
- Christmas shopping. Went through amazon (+other) black friday and cyber monday.
- No-rush shipping options gets 6% back an prime visa (vs 5).
- Played with mint a little. Remember moving to credit karma. Will fix solium connection etc at that time.
- Increased credit limits.
- Money moves.
- AWS is paid by CSR. Remember auto/condo insurance are on doublecash.
- There was an errant instacart+ subscription being charged to my citi card (my actual account is the chase one) since May.
- Reached out to instacart help. Was transferred over 15 times lol. Syed A, Neelima P, Bishal D, Anthony M, Carol V, Tabrez P, Sumera S, Swetha S, Neelima P again, Amanda D, Khanyisile M. At this point, they turned off the notifications.
- Then the fraud team just said “fill out this unrecognized-charges form and we’ll get back to you in 7-10 days” – unacceptable. https://www.instacart.com/secure_forms/unrecognized_charges.
- Didn’t fill it out, instacart didn’t deserve the courtesy. Went to cc to dispute.
- Citi’s dispute interface was “temporarily unavailable” and I couldn’t submit. Both great systems! Gonna cancel both citi and instacart after this trash.
- Amazon reload $100, get 10% back, once.
-
- Private work.
- Remember gnumake’s variables are accessed with
$(var). To grab shell variables, use $$var.
- So you don’t have to set upstream on every new git branch:
git config --global --add --bool push.autoSetupRemote true
- Redis $5.62 on equitybee.
- Added gitlab runners to my macbook so I can pass CICD while working remotely.
/srv doesn’t exist by default, so sudo mkdir /private/srv and chown it to you then run the usual two commands:
docker run -d --name gitlab-runner --restart always -v /private/srv/gitlab-runner/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latestdocker exec -it gitlab-runner gitlab-runner register
- Then enter all the info. Token is from the “create new runner” screen on gitlab’s UI. Default image usually
python:3.12. Executor type is docker.
- Then just up the concurrency by editing
/private/srv/gitlab-runner/config/config.toml on the host.
- Pandas
group_by, agg, more.
- Supercontest.
- Fixed the auth for redis-cli and the private conf file. It’s still clunky (sourcing secret then passing var). Redis-cli should absolutely provide native support for reading auth from a file.
-
- Private work.
- Pandas
.rank() method. Could be used for leaderboard stuff. .melt() is useful too.
- Biggest spelling bee I’ve seen. 74 words, 462 points.
- Amplify and Cognito getting a lotttt of hate: https://www.reddit.com/r/aws/comments/182kunk/which_is_the_most_hated_aws_service/. CodeCommit too, although no one uses.
- 100% draws after 3 rounds in sinquefield.
- Supercontest. Purchased RI for elasticache, 3yr upfront. A little over $7/mo, not bad.
-
- Private work.
- Some regex refreshers. Remember
\b = word boundary (basically all characters except other letters/numbers/underscore.
- More pandas.
iloc, str.match
- Sleeping poorly for about a week.
- I should probably set up a gitlab-runner on my macbook? The desktop is not always running.
- You can read variable definitions from a conf file and pass as env vars to a command.
- Example in bash:
(source <path/to/conf> && echo $<var_from_conf>)
- The parentheses are obviously for a subshell. Then the env vars are defined ONLY in the subsequent command, not the host shell you’re running everything from.
- For redis it’s more efficient to use
SCAN than KEYS to list all keys.
- Supercontest.
- More with elasticache today. Played with
flask-caching and the redis cluster.
- Tested
make sync-db-dev for the first time. Worked well.
- Added a gnumake target to drop into the redis-cli.
- Deployed all infra changes (deps, redis, flask-caching init, auth, app init, etc), as well as the queries/selects splitout.
- Now just ready to ADD caching wherever appropriate (using
flask-caching‘s decorators). And test.
- Connect to the sbsc EC2 with vscode’s remote explorer for ide benefits (vs vim). Can have both windows open (wsl2 and remote), with separate terminals, etc.
- Was getting gunicorn timeouts in prod after deployment.
- The flask-caching default timeout is 300s. Gunicorn is 30, hence dying first.
- No long running queries or anything. Happening on boot.
- Confirmed redis-py is threadsafe, since I use gthread workers with gunicorn: https://redis-py.readthedocs.io/en/stable/advanced_features.html
- So it’s a connection issue. Then simply times out.
- Wrote a little script and debugged till I found it. Unlike host/pw/etc (directly exposed by flask-caching), redis-py expects
ssl=True (which flask-exposes through the generic CACHE_OPTIONS).
- My elasticache redis cluster requires tls, so it was quietly rejecting connections. Added the kwarg. Fixed.
- Tested the PROB restore for the first time, from
pg_restore. Still haven’t restored from an RDS snapshot.
- Was failing at first. “relations/seq/PKs/etc already exists”.
- Confirmed that
--clean --create was being passed, as usual. So it should drop everything before restoring from a dump.
- Worked fine in dev.
- Maybe something in RDS doesn’t allow drops?
- Nope. It was just that you can’t drop a db if there are open connections to it, so
--clean was failing. In dev, I stop the app container before restoring. I had deleted that from prod. Added it back.
- The line fetch was broken because (for the first time in 10 years) westgate added spaces between the team name and the asterisk for home. Fixed.
-
- Private work.
- More pandas. For join, use
merge. Left, right, inner, outer. .unique(), .sort_values(), .drop_duplicates, more.
- Remember to not use truthy operators (
or, and, not) like usual in python. Use bitwise (|, &, ~).
- Changed trays.
- Beside Altman, Greg, Ilya – the other 3 board members were Adam D’Angelo, Tasha McCauley, and Helen Toner. They don’t work at OpenAI (Quora, Rand, and Georgetown). Tasha is also Joseph Gordon Levitt’s wife. I don’t think any have tweeted since? Although some private. Ilya/Altman/Greg have.
- CZ pled guilty for money laundering, 50M fine, step down as binance CEO.
- Gardening.
- Had to relaunch the gitlab runner, and with
--restart always so it boots when wsl2 boots (anytime the docker daemon restarts).
- Registered for re:invent (next week).
- Stacked 10-player sinquefield cup started today.
- Supercontest.
- All elasticache all day today – the implementation on the app side.
- Remember in the old flask-caching infra: You query the db once. It stores the resultant objects in the cache. A second request for the same data in. The cache returns the objects. But the objects are dangling (not associated with a session), and could be wrong, so sqla rejects them. I no longer have to worry about this with the redis cache.
- Made sure ALL reads were from the
queries module and ALL writes were from the commits module. There is one exception: flask-user. This is already scheduled for cognito replacement.
- Split into
queries (wrappers, returning python types) and _queries (direct cache/db reads, returning sqla objects).
- This whittles down the surface area of (direct data access) vs (the app’s manipulation of data in python). I’ll only cache the former, for now. Maybe later I can cache commonly accessed views / whatever.
- Used
flask-redis which plugs directly into the app factory pattern instead of managing my own redis object.
- Hmmm, nope – not needed.
flask-caching (the standard, what I was using before) supports this. It can connect to a redis cluster. And it can cache view functions, custom functions, memoize, cache jinja snippets, auto-create keys for all of those based on specific request paths, args, etc – and exposes direct set() and get() calls from redis.
- Updated banner.
