-
- Private work.
- != and <> are the same in most sqls. Also remember “
case X then Y else 0 end“.
- Updated macos to 14.2.
- https://www.forbes.com/sites/alexkonrad/2023/12/11/armada-edge-computing-ai-spacex-starlink/?sh=6988e7386a45
- Some lawyer calls.
- Garden maintenance. The little bumps at the bottom of the tomato vine = root initials. If in soil, they’ll develop roots. If not, they’ll just stay as nodules. Usually happens when there’s a blockage in the main roots – the plant is trying to create more. Totally normal though.
- Use common table expressions (CTEs) instead of conditions against subqueries results. It’s just more logical to think about. WITH my_new_table AS (SELECT * FROM table1) SELECT * FROM my_new_table JOIN table2
-
- Private work.
- The eagles have the worst point differential of any 10-3 team in NFL history: https://www.statmuse.com/nfl/ask/worst-point-differential-of-a-team-with-10-or-more-wins-and-3-or-fewer-losses-in-a-season-nfl
- More re:invent breakout sessions. Almost all serverless content.
- Remember eventbridge can handle routing. Send events to lambda, trigger step functions, whatever.
- SAM Accelerate (
--sync) deploys your code to a dev env in the cloud for testing. That way, you don’t have to mock everything (serverless def requires more mocking that a full local app).
- Application Composer – a UI that you can use to diagram your full stack. It will then autogenerate the templates for you.
- Isolate your lambda code that touches the db, just like you do in the flask app.
- Testing in the cloud against a full dev stack is very very valuable.
- Remember aws itself uses an eventbridge event bus to communicate between aws services.
- Eventbridge’s event bus is the most generic pubsub interface. It has routing rules, filter polices, archive history, etc. It can push, get polled, whatever. It’s the central hub of complete decoupling. SQS is a very specific service for a single queue. Most reliable for FIFO, a bit more overhead. It’s a pretty common model to have eventbridge be the primary bus and messages continue on to specific SQS queues for consumer processing.
- Changed trays. Finished harder gum (probably won’t buy again, just chew regular). Meal prep (smoothies powders liver). Aquarium maintenance.
- Linear = github jira.
- Fireflies is pretty sweet. Played with it a little more today.
- Rogo = llm for business use cases.
- Remember that
where operates before groups/aggregates in sql. having operates after (so you can condition on counts, etc).
-
- Private work.
- CCT Finals started today.
- Mint alternatives.
- Empower.
- Linked all accounts.
- Got mobile app. All set up.
- Empower uses envestnet and yodlee to connect to accounts.
- Shareworks was down for maintenance, but addition to empower was seamless when it was restored.
- Only account addition hiccup was citi. Failed through desktop. Worked through mobile.
- For crypto, you can enter specific holdings and it will autoupdate values. Won’t link to metamask or coinbase or anything. Most of the names are only the bigger ones on eth. So not worth it – I just put total value.
- Good overall interface. Separation of retirement. Solid account list. Perf comparison to primary ETFs.
- Some lender analysis (big banks) and REBNY.
- Supercontest.
- VPC endpoints are not cheap.
- $0.011/hr in us-west-1, per endpoint.
- I had an endpoint in us-west-1a and us-west-1c for my SAM stack. Deleted.
-
- Private work. Meetings today.
- Remember the real AoC: https://adventofcode.com/
- AWS step functions for automating workflows. ETL, lambdas triggering other things, any pipeline really. Has a visual diagramming tool.
- There’s always Serverless Framework (node module) to compete with SAM. It’s more generic, less catered to aws.
- Distilled the ~700 re:invent breakout sessions into ~35 videos I want to watch.
- Mealprep, amazon fresh, haircut, gym, holiday-tipped building+barber, chores.
- Instacart replied with … nothing. They asked to fill out the SAME form again. Great! After two weeks!
- CDK is written in an imperative language like python, but remember that its output is a declarative cloudformation template.
- Did some research on AI notetakers.
- Butter, Fireflies, Otter, Fathom, more.
- There’s also openai Whisper, but I’m not aware of any direct meeting integrations.
- Most are ~$10-20/user/month.
- Moved from mint to credit karma.
- Shareworks integration still broken. Didn’t migrate the MAJORITY of my accounts. Terrible.
- The mint interface is 1000% better than CK.
- Downloaded mobile app.
- Verified email. Played with some settings. Opted out of creditworthiness/other sharing.
- Tried to submit a support case to reinstate my mint. Failed. Multiple times.
- Started chat. Got transferred the usual few. They were not able to recover my account data.
- I was with mint for 2 decades.
- Going to move to an entirely new platform. This one is trash. Terrible strategic decision on Intuit’s part.
-
- Private work.
- Nearly 700 videos on the list of breakout sessions from reinvent. Will go through and make a list of the ones I’d be interested in.
- Gemini released by Google: https://blog.google/technology/ai/google-gemini-ai/#sundar-note
- More AWS Q problems in vscode:
- Sometimes, it just doesn’t answer (replying it “can’t”) – and then if you resubmit with one word different, it answers perfectly.
- UI is messed up. Sometimes will render left of the window (cut off by sidebar). Sometimes will NOT render code snippets, instead showing plaintext.
- AWS secrets manager is 40c/secret/month, and then access costs.
- Standard # for comments in SAM templates. Full line OR inline.
- You can also compose template.yaml by importing other yaml files. Helpful to split large SAM configs by resource: one file for your lambdas, one for your security groups, etc.
- Updated vscode to 1.85.
- The Amex platinum benefit is every 6mo. You get $50 from jan-jun, and another $50 from jul-dec. Created recurring calendar event. Created online account. Easy.
- Mint still can’t connect to solium.
- Supercontest.
- Thought a bit about the hardest part of SAM: my existing rds db. Config is sensitive, there’s an RI around it, etc – just more sticky to move to IaC. I think I’ll use regular creds for now. Then during the serverless ticket, I’ll move RDS to SAM, and will create a lambda that runs pgrestore after db creation.
- Remember you have to configure the lambda to access rds. This is in the vpcconfig in sam: set the subnet IDs to those of the RDS DB, and the security group which allows RDS access.
- This obviously does not happen in
local invoke.
- Secrets Manager vs Systems Manager Parameter Store. Overall: use secrets manager for secrets, and systems manager parameter store for any other configuration.
- Secrets manager is more expensive.
- Secrets manager does automatic rotation.
- Secrets manager only supports strings.
- Systems manager supports all configurations and types, not just secrets and strings.
- Systems manager is more generic.
- Note that the apigateway resource changes when you (only) change the lambda function behind it. This is just to reconnect to the lambda, since the rest api depends on the backend function.
- Gave the lambdas vpc access to the rds db (via sam policies) to get past the “can’t resolve address” error.
- Then started seeing auth errors where boto3’s token generation didn’t work. Couldn’t get it to auth properly. Ended up just passing password (via Secrets Manager). Remember you can define multiple KV pairs in a secret, so I included username/engine/dbname/etc.
- Works from the command line, but remember you have to give the lambda perms to pull the secret:
SecretsManagerReadWrite.
- Another hangup: SecretsManager is internet-facing. If the lambda is running in a VPC, then you need to add a VPC endpoint to your SecretsManager resource in order for the lambda to be able to reach it (connectivity is diff than authorization!).
- Sidenote: you can only have one endpoint per subnet in each availability zone. Since I have two subnets each in us-west-1a and us-west-1c, I can’t include all 4 subnets in the vpcendpoint for secrets manager. Just choice one from each AZ.
- And: make sure your security groups allow ingress/egress appropriately. The lambda needs to be able to reach the internet (for secrets manager) and the vpc endpoint for the secrets manager needs to allow the lambda to reach it.
- E2E.
- The old path: eventbridge -> lambda -> cloudfront -> loadbalancer -> nginx -> flask -> rds
- The new path: eventbridge -> lambda -> secretsmanager -> rds
- We lose a little time on serverless cold start, but the network path is actually a little shorter.
-
- Private work.
- Finished the re:invent innovation talks.
- Overall, not a lot gained. The keynotes have the valuable releases. The breakouts have the detail to learn/deepdive. These innovation talks are a weird middle ground.
- Multiple genAI talks started with the same proof: “AI, write me an intro”
- Developer experience. Q, codewhisperer, codecatalyst, java upgrade tool, etc.
- Codecatalyst is like an AI-managed suite of source management, issues, CICD, templates – basically gitlab but with Q.
- You can’t really access any of the cool codecatalyst features on the free/individual tier, like assigning issues to Q.
- Paid ortho.
- Streamlit is another pure-python-to-webapp tools.
- Obviously w2 = employees, w9 = contractors.
- Remember both aws codewhisperer and ms copilot are doing autocomplete/autosuggest inline in vscode.
- Prequalification is less than preapproval. It’s usually self-reported -> quick estimate Preapproval is more – the lender actual verifies your financial statement.
- 1031 exchange. Selling an investment property and buying another similar. Defers capital gains tax.
- Zoom webinar hosted by datadog: how to monitor a genAI stack.
- Remember engines are sqla’s lowlevel. You then create a connection from an engine. A session is the higher-level object that uses the ORM to interact via an engine/connection.
- For this most recent leetcode contest, 1st place was all 4 problems in 16min.
- One of the worst contaminants to look for in tap water is fluoride. Affects thyroid function. Checked a few repos, nyc is ok.
- Supercontest.
- Updated banner.
- Added multiple lambda functions (and corresponding IAM roles + API gateway endpoints) and played with them. Interactions, imports. CLI commands, generic invokes or specific calls to a func.
- Remember to
sam build before sam deploy.
AWS::ApiGateway::RestApi is from cloudformation. Use it to directly manage an API gateway. AWS::Serverless::RestApi is from SAM. It manages a lot of the config for you. Use that one.- The
Outputs section of template.yaml doesn’t feed directly to anything on the deploy of the cloud formation. It just exposes dynamic content from the stack build. You can export things like the API gateway URL, the ARNs of the created resources, etc. These print to the command line for immediate testing, and they export to the actual CF template – allowing subsequent use in other SDKs, CLIs, whatever.
- If you want to share code across multiple lambdas:
- One option is S3. Simply tell SAM that “this lambda needs this S3 bucket”, and then tell SAM which files (shared libs) to upload to that S3 bucket on every deploy. You’re just using S3 as your common-code host.
- Another best practice is Lambda Layers. SAM supports this natively. Layers are a little better because Lambda’s execution engine understands how to cache layers. S3 has caching, but it’s cross-tool instead of native.
- Successfully did #2 with
AWS::Serverless::LayerVersion. Builds the shared lib and makes it avail to the lambda.
- Added
boto3 to pyproject. Ultimately I’ll probably just maintain one large set of reqs for the whole project, create and version a single layer from that, and all serverless functions wrap around it. Then I can point my IDE at that (in vscode you can point at a requirements.txt rather than a python interpreter) for highlighting/resolution/dev.
- Added stubs to boto3, and the specific services (sqs, eventbridge, etc) that I need. Just for typechecking in serverless dev.
- Remember this error is from
psycopg2 (switched to psycopg2-binary long ago): [ERROR] ImportError: libpq.so.5: cannot open shared object file: No such file or directory.
- Now that these snippets are running in lambda, I can use
sqlalchemy directly – no need for flask-sqlalchemy since it’s not running in an app context.
artifact_store_retention = 10 in samconfig.toml to control how many deployments are stored in s3 (remember this is the template and the code and everything, so all resources like lambda/apigateway/etc can read from this).”- Local
sam build and invoke do not pass the Outputs section of the template. This is because the local workflow doesn’t deploy anything; there are no ARNs/URLs/etc to reference, since CF has not pushed any. It’s just all local.
- I’ve noticed that copilot is much better than Q so far. Q had wrong information multiple times today:
AWS::Serverless:Queue (AWS::SQS::Queue).- SAM does not auto-inject Outputs into the envs of lambda functions. You have to add it to the env section of the func resource in the template.
- A few others.
-
- Private work.
- ios from 17.1.1 to 17.1.2.
- 6 more re:invent innovation talks.
- AWS in financial services. JP morgan chase, nasdaq, others.
- Lots of genAI.
- RAG = retrieval augmented generation. Customizing a model with private data, like querying an internal db.
- This uses a lot of the new vector db capabilities.
- Would love love love if google tasks adds the ability to order within a day, while the overall list is ordered by date.
- Finished typing – done with daily tasks for now.
- New york housing. Median days on market at sale: 73 days. Normal range is 1-3mo. Going slightly down.
- Garden maintenance.
- Raw cranberries are delicious. I’ve never tried them raw in my entire life.
- Supercontest.
- Made the gitlab repo private, as I’m about to make some substantial changes to upgrade the app.
- Fixed the perm issues from yesterday. Wasn’t macbook vs desktop, it was just s3. Disabled
resolve_s3 in samconfig.toml, instead specifying s3_bucket directly.
- Remember S3 is necessary to support all your templates. This gets around the cloudformation max template size – basically it will upload all assets as necessary to s3, then the templates will reference those addresses. Not necessary for smaller serverless apps, but definitely good practice for larger apps. If you don’t specify an s3 bucket,
sam deploy will simply deploy your template to cloudformation directly.
AWS::Serverless::Function is a wrapper around AWS::Lambda::Function, which is lower level. The higher one also handles roles, APIs, etc.- Did some cleanup on the samconfig, template, events, unit tests, integration tests, and function structure itself.
- Modified the lambda name, moved some files around, explored.
- It gracefully handled the changes. Deleted the old function/apigateway/role/etc associated with the old name, added the new.
- Confirmed
requirements works in SAM. For both local invoke and cloud invoke.
- SAR is just like pypi or dockerhub for SAM apps.
- When you push a deployment from sam cli, that exact stack is what populates “Application” in lambda. You can see resources, endpoints, dashboards, more.
