-
- Private work.
- Altman / Shear / Ilya / Satya, the whole story crazy.
- IRA reposition. It’s the same investment mgmt fees for MS, retirement or non-retirement accounts (total AUM). Fees drop with more AUM (~1.5 -> 1). Externally-managed positions may have additional fees.
- Adobe’s free svg convertor will turn backgrounds black if transparent. Use source images with white backgrounds.
- Started process for Clear’s NextGen Identity+.
- Supercontest.
- I don’t need a general sbsc KMS key.
- Right now it’s only used to encrypt/decrypt RDS exports to S3.
- That’s a rare requirement, and I can create the key when needed for that.
- Every AWS KMS key (that you custom-manage, not aws-managed) costs $1/mo, or 3c/day. Then there are usage costs, but they’re small – 3c for 10k requests.
- Scheduled deletion. Min holding period of 7 days, can inspect logs for usage, then all data from it is un-decrypt-able.
- I still have the 6 aws-managed keys.
- Elasticache.
- Upgraded elasticache cluster to redis 7.1.
- 1 shard, 0 replicas. Inherits backup+maintenance schedule from RDS.
- Implemented with lazy-loading and write-through. No TTL.
- ElastiCache AUTH has 3 options: IAM, password, none.
- Redis offers RBAC. You can define users and usergroups in elasticache. Perms are defined by “Access Strings” – eg
on ~* +@all, which gives everything.
- Remember for connectivity: keep resources for (EC2, Lambda, RDS, ElastiCache) in the same VPC.
- Redis CLI syntax and Python Client syntax for connection/interaction here: https://gitlab.com/bmahlstedt/supercontest/-/issues/225#note_1659547726
- My redis cache requires encrypted connections (TLS).
- Remember there’s no connection between cache and db. The app manages access to both.
- Created a security group to allow TCP traffic on 6582 into the cache.
- “The system automatically creates a default user that is associated with all user groups when a default user is not provided. This user does not provide secure access control to clusters and is intended only for compatibility. Please replace with a default user that is disabled or uses a strong password.”
- Added access control to the cache (was none before). Controlled by User Groups and Users. Added the default and supercontest users.
- Created supercontest user. Important to make the NAME
default (bc you must have a default user, and the default default has no pw, and you want auth) and the User ID custom, like supercontest.
- Created a user group for access to this cache. Added
supercontest as the sole user. You must have one user in the group named default.
- Fully connected from CLI and Python, with auth, with TLS. Played with both a little.
- Thought a bit about the key design. I’ll probably clean up the data API first so the access patterns are clear. Then easier to cache.
- Thought a bit about using redis sorted sets for the leaderboards.
-
- Private work.
- Starship.
- “Altman’s firing followed an intensifying dispute with his fellow co-founder, OpenAI chief scientist Ilya Sutskever, over the speed and safety of the startup’s product rollouts, according to people close to the company, who asked not to be identified discussing private information. The pair and their respective allies on the board also disagreed over Altman’s campaign to raise funds for a separate company to make AI chips to compete with Nvidia Corp., and another project to produce AI-related hardware in partnership with former Apple chief designer Jony Ive. Sutskever and his friends on the OpenAI board may have also been put off by Altman using OpenAI’s name to raise capital, and by the proposed new companies not sharing the same capped-profit governance model as OpenAI, according to one of the people.”
- Friendsgiving!
- Fabi wins st louis rapid blitz. 2 more days of etcc.
- Pandas refresher. Basics. Drop duplicates, add columns, reshape,
dropna, renaming cols, astype, querying, fillna, etc.
-
- Private work.
- Chess24 went down, nginx 404.
- Updated next rise garden subscription box (feb). All seeds, no nutrients/ph.
- Distilled water, refilled 8sleep, primed.
- Repositioned IRA into large cap growth.
- Pinpoint split the SMS interface into a new app: https://us-east-1.console.aws.amazon.com/sms-voice/home
- Actually not too worried about the cognito elements (I think it was SES? Or pinpoint? Or SMS?) being in diff regions. Where possible, everything is us-west-1. Pinpoint (I think) didn’t support us-west-1, so it’s in east.
- Crazy how anti-elon bb and nyt are. Hard to read.
- https://openai.com/blog/openai-announces-leadership-transition
- “Mr. Altman’s departure follows a deliberative review process by the board, which concluded that he was not consistently candid in his communications with the board, hindering its ability to exercise its responsibilities. The board no longer has confidence in his ability to continue leading OpenAI.”
- MVL dominated blitz today. 9 final rounds tomorrow and he has a 2pt lead.
- Remember GPT-4 Turbo is up to April 2023.
- Supercontest.
- Some planning for the big changes coming next.
- ElastiCache.
- Redis and memcached.
- Control size, charged for storage.
- Like RDS, it’s $X/hr. There are on-demand and reserved instances.
- It’s $16/mo for the smallest cache, on-demand. 3yr RI for the same is a little over $8/mo.
- Note that RIs = not fully serverless/scalable. They’re locks. Each one you add makes the whole stack a little less flexible.
- SQS.
- Remember common uses: data buffer btw applications, batching tasks, storing notifications.
- Created a queue.
- Can do standard (at least once, not ordered) or FIFO (exactly once, ordered).
- Configs: msg expiration (up to 14 days), max size (up to 256KB), more.
- Remember dead-letter-queues (DLQs) temporarily stored failed/unprocessable messages (alongside main queue). Also called a redrive policy.
- Messages can have a generic body, whatever you want. And then structured metadata – field:value as desired.
- You can manually send msgs, and manually receive msgs (just poll and inspect them).
- Then they can be deleted after they’ve been accessed. If you don’t, all other consumers will see the message (and likely try to process it).
- Inputs: ALL sorts of services can write to an SQS queue. A popular one is SNS – if a topic on the SNS gets a message, you can forward it to SQS.
- Outputs: When a new msg appears in the queue, you can trigger all sorts of other services. A popular one is Lambda – receive an SQS message -> run a lambda. You can also pipe to eventbridge.
- Note too; if your use case is purely for notifications, SNS is better. It has its own topics, producers, subscriptions, etc.
supercontest-alarms currently does this; cloudwatch sends events to an SNS topic (FIFO, exactly once), then my personal gmail is a subscriber to that topic.
- Thought through an event-driven rewrite of some pieces of the app. Will use SQS for scores and picks.
- Thought a bit about about the 5 BIG changes: Amplify, Lambda, API Gateway, Cognito, and SES/SNS.
- I wish there was a way to chunk it up? But I’ll probably just do it all in one fell swoop. Linked the tickets.
- The lambda change will probably come first (although created from amplify). API will switch to just data-returns (the app will handle the formatting / other logic).
- API gateway will front each of these (again, managed by amplify). This should be a pretty simple change.
- Then cognito will auth each of them (again, managed by amplify).
- Then I’ll go back to the app and rewrite it all in tsx/react. This will be fun.
- At that point, everything will be fully deployable with amplify.
- And then at the end, you can plug into SES/SNS for the current email functionality swapout (and new text).
-
- Private work.
- Not sure why 8sleep was flashing. Water is half full – will fill next aquarium RODI cycle.
- AWS Secrets Manager. Will move privates there once I lift EC2.
- AWS KMS = Key Management System.
- For some reason, docker was wiped on my ubuntu wsl (maybe from docker desktop update on the host?). Reinstalled.
- That was easy. For some reason, my gitlab runners died with this. Ran a new one in a docker container, registered it (have to create a new token in gitlab). Default image
python:3.12. Remember it stores config locally in /etc/gitlab-runner/config.toml. You might have been able to restore it from there.
- Disabled resource saver on docker desktop. You’ll get errors like this when it’s in that mode:
error getting credentials - err: exec: "docker-credential-desktop.exe": executable file not found in $PATH, out:
- Segmented, tenderized, injected, and started the cure for both friendsgiving turkeys.
- Supercontest.
- All RDS today.
- Finished the cloud DB deployment.
- Yep, left docker-compose with an app and a db container for local dev. Prod is EC2+RDS (soon to be serverless for both).
- Remember you have the useful
migrations/data_changes folder. Cleaned it up a bit today.
- Because migrations are with
flask-migrate (via alembic), and flask-migrate goes through the flask CLI (flask db <>), it piggybacks on whatever DB connection your app has. I have to update that anywhere for the prod config, so migrations should work with no code changes beyond that.
- Removed the flask-monitoring-db volume and other minor infra. It was deleted a while ago.
- Database URLs (for sqla):
dialect+driver://username:password@host:port/database
- Removed flask-debugtoolbar.
- Added
.pgpass for abstraction in the makefile (to connect to prod).
- You still have to pass host/user/pw to the psql/pg_dump/pg_restore commands.
- And in the
PGPASSFILE, specify db as * so that it works with the supercontest (main) db as well as the postgres db (target of pg_restore).
- Created KMS key to encrypt rds exports to s3. Also a new IAM role with s3 write perms for this.
- Took a manual snapshot and exported to s3. Snapshot takes a minute or two, export takes longer (it sat in “starting” for 27min and then the actual export took ). Restored it in my local db to test.
- To update the db in your local dev env, you have two options:
- SSH into the EC2 instance, run a pg_dump of the RDS instance, then copy it to your local box and run pg_restore.
- Take a snapshot (manual or automatic) in RDS, export it to S3, download, and restore from it.
- RDS snapshots are parquet files, which can’t be imported with pg_restore. So the former is almost always better.
- Less load on the EC2 now (small benefit).
- Added a convenient gnumake target for the above dev-syncing-method. SSHs into EC2 (using existing SSH profile), runs a backup against the cloud db, SCPs it to the local machine, restores the local db against it and restarts everything.
-
- Private work.
- Python’s
sort() function uses Timsort, which is (worst case) time O(nlogn) and space O(n).
- Reset zoom on vscode. I had been zoomed IN (just one click) for years, but it’s unnecessary strain on the eyes.
- Macbook launchpad keyboard shortcut continues to be broken. Pinch with thumb and three fingers instead. You can also spread with thumb + 3 fingers to go to desktop. F4 is spotlight, but full launchpad view is better.
- To list versions of installed pkgs on ubuntu:
apt list -a <>
- Updated docker desktop on macbook.
- Order dutch crunch rolls online! https://dutchcrunch.com/collections/all
- Windows cumupdate.
- You can export a backup from rds to s3.
- Got jury summons. Requested first-time postponement, 6 months (max).
- Supercontest.
- Uninstalled postgres14 on the ec2 instance (came native with ubuntu). Also did some apt purge/autoremove/upgrade. Then reinstalled postgres15 and rebooted. Couldn’t find JUST the client (psql/pg_dump/pg_restore) in v15.
- Reserved Instances.
- “Reserved DB instances are not physical instances, but rather a billing discount applied to the use of certain on-demand DB instances in your account.” Your usage on an instance comes in. You get charged for storage, backups, other side items. And then you’re left with the hourly rate for the instance. If you have an RI that matches the specs of that instead, your hourly rate is at the lower RI price. If not, you pay full on-demand price.
- For example: Take a single az, rds (not aurora), t3.micro. Storage and backups are a couple dollars a month each, and those are the same in both cases.
- On Demand: 0.024/hr, or 17.52/mo. No upfront.
- RI: 0/hr. Upfront 294, or 8.17/mo.
- This is all the same as EC2. My RI was 315 upfront (8.75/mo), 0/hr usage charge. And then for storage etc, there’s a little extra.
- So every RI for both EC2 and RDS is about $8/mo (all upfront), and then maybe $4/mo for the peripheral usages.
- Created case to remove the second RI: https://support.console.aws.amazon.com/support/home#/case/?displayId=14308599831&language=en. I’ll just use local docker-compose with pg for dev. He was able to cancel it for me, but this will occur at the end of the billing cycle (EOM).
- I’d like to use RDS’ blue/green capability. It handles the syncing for me. Upgraded the prod rds from 15.3 to 15.4 and created parameter group with
rds.logical_replication=1 and associated it with the instance. All required for blue/green. Gonna test it for a bit, check costs to ensure green instances are covered under RIs, and decide to keep (or just create another dev instance and manage it directly).
- Accidentally deleted the prod ec2 db container. Last backup was thursday of last week. Had to manually recover picks and scores.
-
- Private work.
- Browsers do not count for netflix devices.
- Finished The Fall of the House of Usher. Flanagan is so good.
- Garden maintenance.
- AWS product refresher.
- App Runner is kinda like Elastic Beanstalk. It’s a all-in-one service for “I have an app and don’t care about the infra, just deploy/manage it for me.” EB is comprehensive. AR is only for containers.
- SAM = Serverless Application Model. It’s a template for building a serverless stack on aws.
- There’s also SAR = Serverless Application Repository. Integrates with Lambda.
- Remember storage options: Glacier, S3, EFS, EBS.
- Boto is the python SDK.
- Chalice is AWS’ platform for python serverless applications. It’s an aggregator, like the others. With SDK/CDK integration. Define your web routes, your schedules, your s3 triggers, your lambda functions, whatever. https://aws.github.io/chalice. Pretty cool tool, although I don’t think I’ll keep the API backend in python for much longer, so I don’t want to marry to a python platform.
- DevOps guru does some pretty cool ML-based anomaly detection for services like RDS. Costs a few dollars a month.
- Lambda dive.
- Most stemming from the large tree of https://docs.aws.amazon.com/lambda/latest/dg/welcome.html.
- Entry point is always
lambda_handler(event, context). Event is the (json) data passed by the caller. Context is env, arch, logs, etc.
- You can do cool stuff like trigger from s3; every time an image is uploaded, it creates a thumbnail and adds that to s3 as well. I could use this for the sbsc banner.
- You can do some advanced stuff later. Testing functions in the cloud. Added traces.
- The default artifact of a lambda created from the UI is a zip archive. You can include OTHER libraries, dependencies. And you can organize them in layers for efficiency. Example on https://docs.aws.amazon.com/lambda/latest/dg/python-package.html.
- If you want to bypass all that, you can just deploy one of your containers as a lambda. Then it’s not
lambda_handler, it’s the docker ENTRY_POINT.
- If you’re not plugging a lambda into another service and triggering on some event-driven flow, you can invoke the lambda directly via the UI or CLI. You can ALSO assign a function URL and call it to invoke a lambda.
- Use CDK + CloudFormation to define lambdas from source and autodeploy them. Fun fact: cloudformation is declarative (as you’d expect) but CDK is imperative! It converts your instructions to a finish line.
- There are tons of third-party extensions for lambda: datadog, splunk, sentry, etc (if you don’t want to use internal services like cloudwatch for these)
- Lambdas don’t have a connection pooling mechanism and make lots of short connections. This is exactly what RDS proxy was built for.
- Your lambda functions can use IAM permissions to connect to the DB. Much better than UN/PW, etc.
- Full RDS example: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-lambda-tutorial.html.
- Another example with Aurora: https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/cross_service/aurora_rest_lending_library
- Supercontest.
- Probably the best E2E tutorial for a webapp: https://aws.amazon.com/getting-started/hands-on/build-serverless-web-app-lambda-apigateway-s3-dynamodb-cognito/. Amplify, API Gateway, Lambda, Cognito, Dynamo. Pretty close to my supercontest stack.
- Closed docker image build ticket (https://gitlab.com/bmahlstedt/supercontest/-/issues/165) – I’m gonna do serverless. Frontend = amplify hosted. Backend = lambdas. Will handle the remainder on the respective tickets for react and compute.
- Cognito and SES (and most of the rest of the infra) is in us-west-1. Pinpoint and SNS are in us-east-1. Hopefully this doesn’t cause any issues.
- When I sell the EC2 RI later (once fully serverless), aws marketplace takes 12%.
- Since the lambdas hit the db through rds proxy (not connection-pooling through flask anymore), I have to finish #188 first.
- RDS.
- Current EC2 node is t2.micro. And it runs 3 containers, only 1 being postgres. So t3.micro for the db is plenty.
- RDS Postgres calculator: https://calculator.aws/#/addService/RDSPostgreSQL
- Aurora Postgres calculator: https://calculator.aws/#/addService/AuroraPostgreSQL
- Scenarios (order of modern-ness)
- Aurora serverless v2, 0.5ACU/hr (min), rds proxy, 10GB storage, 10GB backups = $175/mo
- Aurora serverless v1, 2ACU/hr (min), 10GB storage, 10GB backups = $115/mo
- Aurora non-serverless, 1 instance, t3.medium, RI 3yr upfront, rds proxy, single az, 10GB storage, 10GB backups = $60/mo (35 from the upfront, only 25/mo recurring)
- RDS, 1 instance, t3.micro, RI 3yr upfront, rds proxy, single az, 10GB storage, 10GB backups = $35/mo (10 from the upfront, only 25/mo recurring)
- RDS proxy is more than HALF the cost for the prices above. For aurora serverless v2, RDS proxy is ~115/mo (at min of 0.5ACU/hr, min of 8ACU charge). For the non-serverless solutions, it’s ~$25/mo (at min of 0.5vCPU/hr, min of 2 vCPU charge).
- Serverless WOULD be cheaper than my RDS because 0.5ACU/hr is a pretty small workload. BUT, supercontest is even less than that, so the serverless minimum is greater than an always-on RDS instance. This is the only piece of the app that isn’t scalable at the end of the cloud migration. As traffic increases, the db should be converted to aurora serverless v2. https://gitlab.com/bmahlstedt/supercontest/-/issues/226
- Bought the RI, 3yr, upfront, single az, $294. Created the DB as above. Non-aurora, non-serverless, postgres15, db.t3.micro, existing master un/pw, 20GiB gp3 SSD, auth via pw and IAM, rds proxy.
- Will add elasticache later: https://gitlab.com/bmahlstedt/supercontest/-/issues/225
- Backups daily (6am ET). Maintenance weekly (tuesdays 7am ET).
- There’s DMS = Database Migration Service. Designed for tasks like this, getting another DB into RDS. I wanna see the lowlevel cloud db details, so I’ll do it all manually.
- Actually no – deleted the RDS proxy. I get low traffic for this site, not worried. Postgres default connection limit is 100. I’m not gonna have 100 concurrent lambdas until the userbase substantially increases. Added to #226 for future.
- The version of psql/pg_restore/pg_dump must match between server (rds) and client (where you’re running the above, either localhost or ec2 instance). Therefore, you’ll need pg15 installed.
- You can attach a security group which allows postgres traffic inbound (tcp, port 5432), as opposed to an EC2 or Lambda connection, but it still won’t work. You have to change the db config in rds to be publicly accessible.
- All explicit targets should go to lambdas, and have explicit lambda connections in rds. Open-ended psql can be accessed from EC2, for now – when I’m fully serverless later, I can open up public access to the DB (at least dev).
- Got
psql / pg_restore / pg_dump working with the cloud db.
- Updated banner.
-
- Private work.
- Dot product in python is easy:
sum(a*b for a, b in zip(a, b))
- Hawaii NYE planning.
- Paperless post has SUCH bad interfaces. Surprising for one of the top premium invite services? Best way to get sharing is just add yourself as a guest, then use the texted link to forward around. Then you don’t have to generate the universal share link for 200 coins.
- Charge kindle from the desktop, not the free micro usb cable.
- Aquarium maintenance. Some meal prep.
- Created gitlab project and ticket for the app catalog: https://gitlab.com/bmahlstedt/app-catalog/-/issues/1
- Supercontest.
- Lots more work on api gateway.
- Played with passthroughs of path params, query params, and payload JSONs.
- Diff stages, manifesting as paths, abstracted since you’ll put CF in front of the full API Gateway URL.
- Built a full example of private access. API Gateway -> VPC Link -> NLB -> ALB -> EC2. Any path in apigw can use this. Just very clunky and double price (both ELBs).
- I don’t need the EC2 public IP. It’s just a convenience (
ssh.<domain> -> ec2.<domain> -> elastic IP) to ssh sbsc from the command line. Come next year (before they start charging), it’ll be gone anyway as I’ll have migrated all of compute from EC2 to Lambda.
- Cleaned some old resources. RDS subnet group. Unused security groups and network interfaces (largely autocreated by RDS).
- Ultimately decided to migrate everything to Amplify/APIGateway/Lambda/Cognito.
