-
- Private work.
- Pandas
.rank() method. Could be used for leaderboard stuff. .melt() is useful too.
- Biggest spelling bee I’ve seen. 74 words, 462 points.
- Amplify and Cognito getting a lotttt of hate: https://www.reddit.com/r/aws/comments/182kunk/which_is_the_most_hated_aws_service/. CodeCommit too, although no one uses.
- 100% draws after 3 rounds in sinquefield.
- Supercontest. Purchased RI for elasticache, 3yr upfront. A little over $7/mo, not bad.
-
- Private work.
- Some regex refreshers. Remember
\b = word boundary (basically all characters except other letters/numbers/underscore.
- More pandas.
iloc, str.match
- Sleeping poorly for about a week.
- I should probably set up a gitlab-runner on my macbook? The desktop is not always running.
- You can read variable definitions from a conf file and pass as env vars to a command.
- Example in bash:
(source <path/to/conf> && echo $<var_from_conf>)
- The parentheses are obviously for a subshell. Then the env vars are defined ONLY in the subsequent command, not the host shell you’re running everything from.
- For redis it’s more efficient to use
SCAN than KEYS to list all keys.
- Supercontest.
- More with elasticache today. Played with
flask-caching and the redis cluster.
- Tested
make sync-db-dev for the first time. Worked well.
- Added a gnumake target to drop into the redis-cli.
- Deployed all infra changes (deps, redis, flask-caching init, auth, app init, etc), as well as the queries/selects splitout.
- Now just ready to ADD caching wherever appropriate (using
flask-caching‘s decorators). And test.
- Connect to the sbsc EC2 with vscode’s remote explorer for ide benefits (vs vim). Can have both windows open (wsl2 and remote), with separate terminals, etc.
- Was getting gunicorn timeouts in prod after deployment.
- The flask-caching default timeout is 300s. Gunicorn is 30, hence dying first.
- No long running queries or anything. Happening on boot.
- Confirmed redis-py is threadsafe, since I use gthread workers with gunicorn: https://redis-py.readthedocs.io/en/stable/advanced_features.html
- So it’s a connection issue. Then simply times out.
- Wrote a little script and debugged till I found it. Unlike host/pw/etc (directly exposed by flask-caching), redis-py expects
ssl=True (which flask-exposes through the generic CACHE_OPTIONS).
- My elasticache redis cluster requires tls, so it was quietly rejecting connections. Added the kwarg. Fixed.
- Tested the PROB restore for the first time, from
pg_restore. Still haven’t restored from an RDS snapshot.
- Was failing at first. “relations/seq/PKs/etc already exists”.
- Confirmed that
--clean --create was being passed, as usual. So it should drop everything before restoring from a dump.
- Worked fine in dev.
- Maybe something in RDS doesn’t allow drops?
- Nope. It was just that you can’t drop a db if there are open connections to it, so
--clean was failing. In dev, I stop the app container before restoring. I had deleted that from prod. Added it back.
- The line fetch was broken because (for the first time in 10 years) westgate added spaces between the team name and the asterisk for home. Fixed.
-
- Private work.
- More pandas. For join, use
merge. Left, right, inner, outer. .unique(), .sort_values(), .drop_duplicates, more.
- Remember to not use truthy operators (
or, and, not) like usual in python. Use bitwise (|, &, ~).
- Changed trays.
- Beside Altman, Greg, Ilya – the other 3 board members were Adam D’Angelo, Tasha McCauley, and Helen Toner. They don’t work at OpenAI (Quora, Rand, and Georgetown). Tasha is also Joseph Gordon Levitt’s wife. I don’t think any have tweeted since? Although some private. Ilya/Altman/Greg have.
- CZ pled guilty for money laundering, 50M fine, step down as binance CEO.
- Gardening.
- Had to relaunch the gitlab runner, and with
--restart always so it boots when wsl2 boots (anytime the docker daemon restarts).
- Registered for re:invent (next week).
- Stacked 10-player sinquefield cup started today.
- Supercontest.
- All elasticache all day today – the implementation on the app side.
- Remember in the old flask-caching infra: You query the db once. It stores the resultant objects in the cache. A second request for the same data in. The cache returns the objects. But the objects are dangling (not associated with a session), and could be wrong, so sqla rejects them. I no longer have to worry about this with the redis cache.
- Made sure ALL reads were from the
queries module and ALL writes were from the commits module. There is one exception: flask-user. This is already scheduled for cognito replacement.
- Split into
queries (wrappers, returning python types) and _queries (direct cache/db reads, returning sqla objects).
- This whittles down the surface area of (direct data access) vs (the app’s manipulation of data in python). I’ll only cache the former, for now. Maybe later I can cache commonly accessed views / whatever.
- Used
flask-redis which plugs directly into the app factory pattern instead of managing my own redis object.
- Hmmm, nope – not needed.
flask-caching (the standard, what I was using before) supports this. It can connect to a redis cluster. And it can cache view functions, custom functions, memoize, cache jinja snippets, auto-create keys for all of those based on specific request paths, args, etc – and exposes direct set() and get() calls from redis.
- Updated banner.
-
- Private work.
- Altman / Shear / Ilya / Satya, the whole story crazy.
- IRA reposition. It’s the same investment mgmt fees for MS, retirement or non-retirement accounts (total AUM). Fees drop with more AUM (~1.5 -> 1). Externally-managed positions may have additional fees.
- Adobe’s free svg convertor will turn backgrounds black if transparent. Use source images with white backgrounds.
- Started process for Clear’s NextGen Identity+.
- Supercontest.
- I don’t need a general sbsc KMS key.
- Right now it’s only used to encrypt/decrypt RDS exports to S3.
- That’s a rare requirement, and I can create the key when needed for that.
- Every AWS KMS key (that you custom-manage, not aws-managed) costs $1/mo, or 3c/day. Then there are usage costs, but they’re small – 3c for 10k requests.
- Scheduled deletion. Min holding period of 7 days, can inspect logs for usage, then all data from it is un-decrypt-able.
- I still have the 6 aws-managed keys.
- Elasticache.
- Upgraded elasticache cluster to redis 7.1.
- 1 shard, 0 replicas. Inherits backup+maintenance schedule from RDS.
- Implemented with lazy-loading and write-through. No TTL.
- ElastiCache AUTH has 3 options: IAM, password, none.
- Redis offers RBAC. You can define users and usergroups in elasticache. Perms are defined by “Access Strings” – eg
on ~* +@all, which gives everything.
- Remember for connectivity: keep resources for (EC2, Lambda, RDS, ElastiCache) in the same VPC.
- Redis CLI syntax and Python Client syntax for connection/interaction here: https://gitlab.com/bmahlstedt/supercontest/-/issues/225#note_1659547726
- My redis cache requires encrypted connections (TLS).
- Remember there’s no connection between cache and db. The app manages access to both.
- Created a security group to allow TCP traffic on 6582 into the cache.
- “The system automatically creates a default user that is associated with all user groups when a default user is not provided. This user does not provide secure access control to clusters and is intended only for compatibility. Please replace with a default user that is disabled or uses a strong password.”
- Added access control to the cache (was none before). Controlled by User Groups and Users. Added the default and supercontest users.
- Created supercontest user. Important to make the NAME
default (bc you must have a default user, and the default default has no pw, and you want auth) and the User ID custom, like supercontest.
- Created a user group for access to this cache. Added
supercontest as the sole user. You must have one user in the group named default.
- Fully connected from CLI and Python, with auth, with TLS. Played with both a little.
- Thought a bit about the key design. I’ll probably clean up the data API first so the access patterns are clear. Then easier to cache.
- Thought a bit about using redis sorted sets for the leaderboards.
-
- Private work.
- Starship.
- “Altman’s firing followed an intensifying dispute with his fellow co-founder, OpenAI chief scientist Ilya Sutskever, over the speed and safety of the startup’s product rollouts, according to people close to the company, who asked not to be identified discussing private information. The pair and their respective allies on the board also disagreed over Altman’s campaign to raise funds for a separate company to make AI chips to compete with Nvidia Corp., and another project to produce AI-related hardware in partnership with former Apple chief designer Jony Ive. Sutskever and his friends on the OpenAI board may have also been put off by Altman using OpenAI’s name to raise capital, and by the proposed new companies not sharing the same capped-profit governance model as OpenAI, according to one of the people.”
- Friendsgiving!
- Fabi wins st louis rapid blitz. 2 more days of etcc.
- Pandas refresher. Basics. Drop duplicates, add columns, reshape,
dropna, renaming cols, astype, querying, fillna, etc.
-
- Private work.
- Chess24 went down, nginx 404.
- Updated next rise garden subscription box (feb). All seeds, no nutrients/ph.
- Distilled water, refilled 8sleep, primed.
- Repositioned IRA into large cap growth.
- Pinpoint split the SMS interface into a new app: https://us-east-1.console.aws.amazon.com/sms-voice/home
- Actually not too worried about the cognito elements (I think it was SES? Or pinpoint? Or SMS?) being in diff regions. Where possible, everything is us-west-1. Pinpoint (I think) didn’t support us-west-1, so it’s in east.
- Crazy how anti-elon bb and nyt are. Hard to read.
- https://openai.com/blog/openai-announces-leadership-transition
- “Mr. Altman’s departure follows a deliberative review process by the board, which concluded that he was not consistently candid in his communications with the board, hindering its ability to exercise its responsibilities. The board no longer has confidence in his ability to continue leading OpenAI.”
- MVL dominated blitz today. 9 final rounds tomorrow and he has a 2pt lead.
- Remember GPT-4 Turbo is up to April 2023.
- Supercontest.
- Some planning for the big changes coming next.
- ElastiCache.
- Redis and memcached.
- Control size, charged for storage.
- Like RDS, it’s $X/hr. There are on-demand and reserved instances.
- It’s $16/mo for the smallest cache, on-demand. 3yr RI for the same is a little over $8/mo.
- Note that RIs = not fully serverless/scalable. They’re locks. Each one you add makes the whole stack a little less flexible.
- SQS.
- Remember common uses: data buffer btw applications, batching tasks, storing notifications.
- Created a queue.
- Can do standard (at least once, not ordered) or FIFO (exactly once, ordered).
- Configs: msg expiration (up to 14 days), max size (up to 256KB), more.
- Remember dead-letter-queues (DLQs) temporarily stored failed/unprocessable messages (alongside main queue). Also called a redrive policy.
- Messages can have a generic body, whatever you want. And then structured metadata – field:value as desired.
- You can manually send msgs, and manually receive msgs (just poll and inspect them).
- Then they can be deleted after they’ve been accessed. If you don’t, all other consumers will see the message (and likely try to process it).
- Inputs: ALL sorts of services can write to an SQS queue. A popular one is SNS – if a topic on the SNS gets a message, you can forward it to SQS.
- Outputs: When a new msg appears in the queue, you can trigger all sorts of other services. A popular one is Lambda – receive an SQS message -> run a lambda. You can also pipe to eventbridge.
- Note too; if your use case is purely for notifications, SNS is better. It has its own topics, producers, subscriptions, etc.
supercontest-alarms currently does this; cloudwatch sends events to an SNS topic (FIFO, exactly once), then my personal gmail is a subscriber to that topic.
- Thought through an event-driven rewrite of some pieces of the app. Will use SQS for scores and picks.
- Thought a bit about about the 5 BIG changes: Amplify, Lambda, API Gateway, Cognito, and SES/SNS.
- I wish there was a way to chunk it up? But I’ll probably just do it all in one fell swoop. Linked the tickets.
- The lambda change will probably come first (although created from amplify). API will switch to just data-returns (the app will handle the formatting / other logic).
- API gateway will front each of these (again, managed by amplify). This should be a pretty simple change.
- Then cognito will auth each of them (again, managed by amplify).
- Then I’ll go back to the app and rewrite it all in tsx/react. This will be fun.
- At that point, everything will be fully deployable with amplify.
- And then at the end, you can plug into SES/SNS for the current email functionality swapout (and new text).
-
- Private work.
- Not sure why 8sleep was flashing. Water is half full – will fill next aquarium RODI cycle.
- AWS Secrets Manager. Will move privates there once I lift EC2.
- AWS KMS = Key Management System.
- For some reason, docker was wiped on my ubuntu wsl (maybe from docker desktop update on the host?). Reinstalled.
- That was easy. For some reason, my gitlab runners died with this. Ran a new one in a docker container, registered it (have to create a new token in gitlab). Default image
python:3.12. Remember it stores config locally in /etc/gitlab-runner/config.toml. You might have been able to restore it from there.
- Disabled resource saver on docker desktop. You’ll get errors like this when it’s in that mode:
error getting credentials - err: exec: "docker-credential-desktop.exe": executable file not found in $PATH, out:
- Segmented, tenderized, injected, and started the cure for both friendsgiving turkeys.
- Supercontest.
- All RDS today.
- Finished the cloud DB deployment.
- Yep, left docker-compose with an app and a db container for local dev. Prod is EC2+RDS (soon to be serverless for both).
- Remember you have the useful
migrations/data_changes folder. Cleaned it up a bit today.
- Because migrations are with
flask-migrate (via alembic), and flask-migrate goes through the flask CLI (flask db <>), it piggybacks on whatever DB connection your app has. I have to update that anywhere for the prod config, so migrations should work with no code changes beyond that.
- Removed the flask-monitoring-db volume and other minor infra. It was deleted a while ago.
- Database URLs (for sqla):
dialect+driver://username:password@host:port/database
- Removed flask-debugtoolbar.
- Added
.pgpass for abstraction in the makefile (to connect to prod).
- You still have to pass host/user/pw to the psql/pg_dump/pg_restore commands.
- And in the
PGPASSFILE, specify db as * so that it works with the supercontest (main) db as well as the postgres db (target of pg_restore).
- Created KMS key to encrypt rds exports to s3. Also a new IAM role with s3 write perms for this.
- Took a manual snapshot and exported to s3. Snapshot takes a minute or two, export takes longer (it sat in “starting” for 27min and then the actual export took ). Restored it in my local db to test.
- To update the db in your local dev env, you have two options:
- SSH into the EC2 instance, run a pg_dump of the RDS instance, then copy it to your local box and run pg_restore.
- Take a snapshot (manual or automatic) in RDS, export it to S3, download, and restore from it.
- RDS snapshots are parquet files, which can’t be imported with pg_restore. So the former is almost always better.
- Less load on the EC2 now (small benefit).
- Added a convenient gnumake target for the above dev-syncing-method. SSHs into EC2 (using existing SSH profile), runs a backup against the cloud db, SCPs it to the local machine, restores the local db against it and restarts everything.
