Mostly Software

  • Sunday

    • Private work.
    • Installed java8 on macbook.
    • AppSync.
      • AWS’ serverless graphql api tool. Like apigateway for rest.
      • Define schema, test queries in the interactive console, custom domains, everything you’d expect.
      • Really nice for cases where you’re combining data from multiple sources: rdb, lambda, http apis, etc.
    • Played with graphql.
    • Ordered all friendsgiving stuff.
    • Amplify E2E.
      • Install amplify cli (npm pkg in your app). Created IAM user, added amplify-admin policy, created access key, configured CLI to auth with that user. These are stored in ~/.aws/credentials. You can have multiple local profiles. ~/.aws/config is there too, which defines zone and such.
      • Use the cli to init. Deploys resources to support amplify. Like creating a new app in amplify studio.
      • Install npm pkg aws-amplify for frontend libs.
      • amplify add api, define type (graphql/rest), define auth (key, cognito, etc), define schema. It can also autogenerate code for you (CRUD operations, like graphql mutations/queries, rest gets/posts, etc).
      • amplify push for just backend, amplify publish for both backend and frontend.
      • CloudFormation is obviously the default provider that amplify’s CLI uses for creating/managing the cloud resources. For the API, AppSync is used for GraphQL and APIGateway is used for REST.
      • amplify console api opens up your API’s console in the browser (the aws ui), appsync for graphql and apigateway for rest. amplify console opens up the full console (amplify studio).
      • amplify mock api will spin up a local graphiql server to test your api (interactively). I think it spins up a local db too so you’re not writing to the cloud db.
      • Testing the app locally (npm start) will still connect to your cloud db! Make sure it’s in the env you desire. For dev, have a dev setup (duh).
      • amplify add auth. Define some cognito basics (federation/socials, un/ph/em/pw, etc). Then you just add some components / auth wrappers to your frontend.
      • Then users can create accounts, confirm email, login, etc. Everything you wanna do. These changes are synced with cognito. Played with this a bit.
      • There are many other components in @aws-amplify/ui-react. You can use these, style them, customize them, combine them with yours, whatever.
      • Add hosting. You can use amplify native to host a node env where the app runs, or you can do a custom setup with cloudfront and s3. Amplify just speeds a lot of this up for you (while still using cf and s3 under the hood), and allows CD by plugging into your git repo, etc.
      • publish will gzip the assets and deploy an optimized production build to your hosting env. I’m sure you can customize this to use whatever bundlers/minifiers you want.
      • REST.
        • You basically add paths and corresponding serverless functions. These go to API gateway and lambda.
        • You can separately have unauthed paths and authed paths. The authed paths integrate with cognito.
      • For amplify add storage, you only have s3 and dynamodb options.
    • Lambda layers are reusable bits of code/assets available to other functions. Imagine a python module you wanna import to two lambdas. Or a node_modules dir you wanna use across multiple functions.
    • Supercontest.
      • Pinpoint’s tollfree registration finally came back (after 2 weeks) and said “company website is not accessible” (bullshit). Updated/resubmitted the registration to /user/register to make it obvious…that of course the site requires login. That’s literally the point of the registration process, to affirm user flows before comms are provided.

  • Saturday

    • Private work.
    • UFC 295 at MSG.
    • Bowery Hotel (Only Murders event was a scam).
    • Saw Jason Momoa.
    • Remember: Pointers in your brain are efficient. If someone has a billion verbosities committed to memory (or full algos memorized for DS&A, creative arts, whatever), retrieval for other RAM is slower. Store a pointer. Look it up later.
    • Amplify charges in two segments. Build/deploy time, 1 cent per minute. My build is only a few seconds, and I’ll probably only do a couple rollup commits per day. The second segment is hosting. 20c/GB stored (build output size, times number of builds, factor in expiration), 15c/GB served, 30c/1M requests, 20c/hr request durations.
    • Did a ton of planning for parents trip.
    • Ordered friendsgiving food.
    • Escape Room.
    • ABC kitchen.
    • Cloudshell is aws’ cloud CLI. Injects creds and a lot of other things for simplification.
    • A bit more on Amplify.

  • Friday

    • Private work.
    • Remember (cassandra java) (scylla cpp). Both nosql.
    • Anthony Jeselnik at Carnegie Hall.
    • OpenAI seeking data partnerships: https://openai.com/blog/data-partnerships
    • NY map: https://www.nytimes.com/interactive/2023/upshot/extremely-detailed-nyc-neighborhood-map.html
    • The longspine urchin ate the molt of the sally lightfoot crab!
    • I get so many spam calls nowadays.
    • Ordered fresh. Turkeys were anywhere from $2-5/lb at morton williams. $1/lb for butterball on fresh / whole foods.
    • Next.js useful reminders. Most are obvious.
      • SSR is still valuable if your app is purely a client that hits other servers/APIs you don’t own. SSR runs wherever next.js runs; in my case, the amplify hosted env. That’s where we get the SEO/Perf/Fetching benefits. Not the data server / API server.
      • You can host html/css/js in s3. The client browser knows how to render it. You can also return it directly from the api server. Next is much smarter – the client browser requests, the app hosting env is running next/node and can do a bunch of serverside actions. Fetching. Caching. Rendering. Whatever. This speeds up the client. 
      • App router (new) vs page router (old). It’s filesystem based. You can also have dynamicism (eg `=[routeFolder]).
      • By default everything from fetch() (server-side only) is stored in a data cache (in amplify, vercel, wherever it’s hosted). If you want to fetch from the client, use Route Handler (or third party libs like swr https://swr.vercel.app/). Try to lean toward doing datafetching serverside though. Then you can store db connections, api tokens, etc, there and not expose to the client. Also, serverside means you can cache responses across users, which you cannot do clientside (single user).
      • “Deploying” is pushing the app to that serverside env (not api server, not client). It’s running next and node.
      • Remember the browser immediately shows your html (non-interactively) on initial page load, then hydrates everything with js instructions.
      • Serverside rendering can occur in a few places. Static rendering is at build time (for pages like docs, feedback forms, etc). Dynamic rendering is at request time (eg for user-specific views). Next will automatically choose the best rendering method for your pages (based on the functions used). There’s also Streaming rendering, which is basically just where the data is chunked.
      • Clientside rendering is best for interactivity. You can useState(), useEffect(), event listeners. I probably don’t need any client components for sbsc. Maybe for something like statistics, where it fetches everything at once then the client chooses the dropdown filters. I might need a client component for the matchups view too, where onClick() is needed for picks.
    • Supercontest.
      • Looks like most of the amplify cost is in the build time (for my relatively low request service). In steady state, I’ll just have builds on master and staging (not feature branches? or just on pull requests).
      • More tinkering in amplify studio.

  • Thursday

    • Private work.
    • GitHub Universe Keynote: https://www.youtube.com/watch?v=NrQkdDVupQE
      • Of course MS, same OpenAI umbrella.
      • Focusing mostly on AI pair programming with Copilot, coming December.
      • Plugs into github itself too; eg asking chatgpt to write a PR summary for you.
      • Automatic scans for security reviews, plaintext passwords.
      • Regex assistant.
      • You can train copilot on your company’s private data. Copilot Enterprise. Example: if you have an internal service to do something, copilot can suggest that instead of a public solution. Example: explain code XYZ to me.
      • Copilot Workspaces. For every issue filed on github, it will analyze it (and the repo, and the internet) and propose a solution and PR the fix.
    • Copilot.
      • Unistalled tabnine and installed copilot (+chat, +labs) for vscode. Signed up for individual ($100/yr, 1mo free trial). Labs requires an extra term agreement (free).
      • Autocompletion is great. Has multiple suggestions you can scroll through (alt and a bracket, left or right)
      • ctrl-i for (editor-inline) copilot chat. Ask it to create all sorts of code. Instead of inline, you can have a full chat window open in the sidebar (basically chatgpt in ide).
      • You can also just write comments and tab through the autocompletions to generative code step by step.
      • The Labs sidebar has 4 features.
        • Explain highlighted code.
        • Translate highlighted code (to another language).
        • Generate tests for highlighted code (only available in js/ts for now).
        • And a bunch of “brushes” – add docstrings, add types, chunk into smaller functions, clean, make readable, more.
      • After 1 day of use – this is extremely helpful in speeding up development times. Especially for less-familiar areas, like my tsx sbsc translation.
    • NYT definitely uses wordpress for their site. After looking it up, so does Forbes, CNN, many more.
    • Updated iCUE.
    • Can’t simply fetch() in useEffect to an address with a CORS policy that hasn’t whitelisted your requested origin (like localhost for testing -> sbsc). Must set the response headers on the server itself (not a clientside fix, obv). Postman doesn’t hit these errors bc it’s just sending the request. Browsers enforce CORS policies (and requests submitted from apps running within them). Exception: the navigation bar doesn’t enforce CORS. These are considered basic requests, unlike XMLHttpRequest/AJAX/FetchAPI.
    • Remember server components can’t useState or useEffect.
    • Monthly water change AND biweekly maintenance. Replaced trident reagent A.
      • You can’t fill the sump all the way up to the top of the white divider. It will overflow when the main pump is off.
    • The AWS vscode plugin allows you to interface with all your aws resources. You can modify files in s3, invoke lambdas, check cloudwatch logs, more.
    • Omegle was forced to shut down (mostly child abuse).
    • Updated vscode.
    • Mint is being shut down.
      • Moving to Credit Karma.
      • Another product under the Intuit umbrella.
      • Not sure why. Mint is such a popular aggregator.
    • Disputed the “undeliverable” charges on the prime card. Amzn hadn’t issued a refund, and the charges are still posted to the account.
    • Amplify.
      • Amplify has been costing the privateer site less than a cent per month (low traffic, no big changes to rebuild).
      • There’s Amplify Studio and Amplify Hosting. The latter is for hosting the frontend, which I need. Studio is huge:
        • Assists with building the fronted. Library of UI elements.
        • Defines data model.
        • Storage.
        • Managers users / authentication.
        • Analytics.
        • API definition (REST and GraphQL).
      • Amplify automatically creates a CF dist and DNS records to front your app.
      • While Studio is basically the IDE for the full stack; you can also use the console, `amplify-cli`.
      • You can’t migrate an amplify app across regions (easily).
    • Supercontest.
      • Worked on Amplify all day today.
      • Dumb refresher. In the early years, my sbsc site was basically SSR. Flask would get the requests and respond with the rendered templates. But you can just as easily deploy a static site (SPA or otherwise, it’s just a collection of html/css/js files) to s3/cloudfront. Nowadays, Amplify supports SSR via next.
      • Currently using flask-user, and migrating to aws-cognito, but I could also move to flask-jwt-extended. You just make a /login route, take the un/pw, use the jwt lib to create a token from it, then return the token to the user. Subsequent async requests from your client app to the server can use the jwt.
      • Versioning.
        • This was entirely too difficult for such a commonly used modern stack of amplify+next. You can’t use node 20, which is LTS.
        • At first (node 20, npm 10, next 14). Ran into too many issues with the default build image, pkg overrides, GLIBC_2.27 not found, etc.
        • Switched to (node 18, npm 9, next 14). Still facing issues.
        • So just used the build image public.ecr.aws/docker/library/node:20. Build passed, failed on deploy (for the hosting env, not backend).
        • Found this: https://github.com/aws-amplify/amplify-hosting/issues/3773#issuecomment-1783092708. The fix lies on the amplify side. As of 2 weeks ago, amplify only supports node 16 (!!).
        • Then it looks like the devs added support for node 18 (which is compatible with next 14) last week. Node 20 is still not supported.
        • Honestly considering moving to vercel purely to avoid this.
        • Eventually got it working with node 18. Can’t add package overrides (since still missing glibc in amazon linux 2), so set custom build image to public.ecr.aws/docker/library/node:18.
      • Next obv ships with ts, eslint, tailwind.
      • Yep, Amplify has UI components (and cognito integration) to replace everything from flask-user.
      • https://aws-amplify.github.io/amplify-js/api/classes/authclass.html

  • Wednesday

    • Private work.
    • WP.
      • Updated wp to 6.4 (and plugins).
      • Updated theme 2015 -> 2024. FInally got the editor indentation back.
      • Remember the primitives: posts, pages, media, comments. I don’t use comments. I use posts and media. And now I split segments into pages for ease of use.
      • Created pages: About, Archive. Then there’s the default “Blog Home” which shows recent posts as the homepage.
      • Edited footer (template) to just be copyright.
      • Customized header. Site logo, site title, pages, search.
      • Removed author name, category, etc from the Post Meta template. Just Date and content. Title goes around the meta (the calling template).
      • Remember – you have to specifically link a link in the post editor, you can’t just copy a URL. Just highlight the link and click the paperclip. Example: https://gitlab.com/bmahlstedt
      • Created new template part Posts Query Loop for use on Blog Home and Search Results (note this fixed the search inconvience, now shows full posts on results page).
      • Also added this new part to the other templates that show posts: Archive (day, month), Index.
      • Edit button on individual posts is gone? Just go through admin interface.
      • One last little nit: the bullets in editor mode are normal and the bullets in the viewer (published) are reversed (clear is level one, solid is level two, etc).
      • Media library does not allow ico or svg – use jpg or png.
      • Installed a vanilla plugin to highlight search terms in the results.
    • You can obviously connect a webapp directly to a db. Faster to prototype. But at scale (and for safety), the frontend should talk to the backend via a rest api, and the backend maintains the db connections. You can pool, make queries efficient, cache, check auth – many things that are harder / impossible / spoofable on the frontend.
    • AWS.
      • Remember AWS Pinpoint tollfree registration can take up to 15 business days. That ends Nov 24 for me.
      • Bunch of research on AWS API Gateway.
        • Swagger, methods, CORS, massive amounts of customization, deployment stages, custom domains, integration with cognito, integration with lots of compute backends (lambda is most supported), ability to generate SDKs from the API definitions, testing, versioning.
        • Great choice for API dev/mgmt. Overall; the value is truly in the definition of the API. Right now, it’s just living in the flask routes. API gateway gives it structure, standards, stages, versioning. All the desires of an at-scale API management process.
        • This is also deeply tied to compute. The value of APIGW is extracted much more greatly if the microservice model is employed with Lambda functions as the serverless backend. It’s more cheap and atomic than ELB -> EC2.
        • Note on cloudfront: if you build a REST API in APIGW and choose edge, it will create a managed CF dist for you. Both work in tandem.
      • Outposts = on-prem deployments.
      • Compute. ECS is much cheaper than EKS (which costs $70/mo just for control plane). ECS is cheaper than ELB/EC2 ($18+/mo). Fargate is the serverless deployment (for both).
      • Elastic Beanstalk.
        • It’s an aggregator. Can define your backend server, what env/lang/version, attached RDS, nginx revproxy, load balance, logs to s3, maintenance windows, CW integration, security groups / IAM users / policies.
        • This is very nice for users who don’t want to learn the details of the aws stack/offerings. It’s a single entry point to deliver a full-stack application, and EB handles the infra under the hood for you.
        • Since I’m already managing all those components, I’ll stick with Amplify/Vercel for the frontend deployment.
    • Iphone’s “repeated calls” setting is >= 2 calls from the same origin in 3min. Disabled this for DND and Sleep.
    • Supercontest.
      • Currently, compute is ELB/EC2. Considering moving to Lambda/APIGW. Or ECS/Fargate.
      • The pricing for ELB is fine after the scoreticking change. It’s not the traffic that is the expensive piece (although that scales), it’s simply the $18/mo for the ALB.
      • Did some planning. Order the remaining tickets, gates, and do-togethers.
      • Updated banner.
      • Genericized banner (without extension) so that I can upload gifs or jpgs or pngs or whatever week-by-week format I want.
      • Started designing the backend/frontend splitout.
      • SELECT CURRENT_TIME; in psql to verify the week rollover is good after DST.
      • Made it so the table-specific actions (ranks, highlightUserRow, setInterval for scoreticking, etc) only occur on (1) the matchups view when lines are committed and there’s an actual matchups table and (2) the allpicks view when picks are closed and there’s an actual allpicks table.
      • Fixed some /pick functionality (mostly from jquery removal).
      • Verified that the scoreticking changes were good on week rollover as well. If you leave a tab open before week rollover, it will keep fetching the old week after rollover (as intended).
      • The metric filter for app container log alarms was triggering on a gunicorn log msg: [WARNING] Exception managing chrome: Unable to discover proper chromedriver version in offline mode. This doesn’t matter. Changed the filter to only trigger on my errors, ?ERROR ?CRITICAL ?EXCEPTION.
      • The commitScores lambda failed during redeploy, that’s fine (remember the logic to NOT commit scores is serverside, the lambda fires 24/7).
      • The commitLines lambda failed. The function behaved normally (lines committed, email went out), but the lambda logs showed timeout. It defaults to 3s. My server doesn’t return that quickly for the commit-lines route (webscrapes, writes, emails, more). Increased the timeout for all 4 eventbridge lambdas appropriately.

  • Tuesday

    • Private work.
    • SES Trust/Safety team approved the production request. Likely the scathe yesterday. Still waiting on the tollfree registration of the pinpoint origination number, then will submit to take it out of the sandbox, then all clear on the cognito changes.
    • Put the ducati on the tender.
    • OpenAI devday keynote: https://www.youtube.com/watch?v=U9mJuUkhUzk. Release of GPT-4 Turbo. 128k context (300 pages). JSON compatibility. Trained on data up to Apr 23. Custom GTPs = biggest change. You and modify/focus/customize a GPT for a specific purpose, then rerelease that to the GPT store for others to use.
    • Roasted 6lb pork butt to make carnitas burritos.
    • Google invested another 2B in anthropic.
    • Sandhill has some spacex. Selling at 86. 50k min check. Curious how Bret is handling/approving this (if at all?).
    • Supercontest.
      • Cloudwatch.
        • Finished the logging change. Piped from app container on EC2 to cloudwatch. Added alarms. Removed sentry.
        • First attempted with a cloudwatch agent config before finding the awslogs docker driver. Easy.
        • CW can do a lot. I use for metrics/logs/alarms/dashboards. But you can also do RUM, traces, canaries, A/B, more.
        • Seamless handling between EC2<->CW, from both a credential and a config perspective. Also datetime formatting (from your custom logs), how to manage the timezone when tailing in the CW UI, etc.
        • Retention policies. Starting at 30d for now.
        • You can query logs like: filter @message like /commit-scores/ | sort @timestamp desc | limit 20
        • The alarms work as follows: A custom metric scrapes the logs and counts errors. An alarm is created from that metric. The alarm sends violations to the SNS topic. The topic emails me.
        • The log filter checks for ERROR, CRITICAL, EXCEPTION (case insensitive).
        • Also added an alarm for any lambda errors (to email as well).
        • Seeing a bit weird behavior with insufficient data for the metric filter on the logs. It counts regex matches, and default value IS value, but the metric is not reporting the 0 datapoints, so the alarm says insufficient data.
          • Ended up finding this bug. It was that all timestamps were being pushed as the same timestamp (ie not updating with the log message, just repeating the same startup time).
          • So the metric filter (and corresponding alarm) didn’t have any data. It thought it was all flattened into one datapoint.
          • Not sure why it was doing this. Because it was freezing at the startup ts, I looked at the beginning of the logs. Gunicorn was printing slightly diff format.
          • So I standardized everything (gunicorn, flask, my loggers, etc) to ignore ms and include tz.
          • This fixed it. AWS showed properly parses datetimes.
      • There are many trash requests from scrapers checking wordpress things and assets to steal info (/wp-includes/*, /uploads/, /admin/, etc).
      • LB links.
        • https://gitlab.com/bmahlstedt/supercontest/-/issues/222
        • Looks clean.

  • Monday

    • Private work.
    • Ducati died in uws. 3 days, 50deg. Did not think that was cold enough (although the battery is close to EOL). Geico app, takes 1 min to submit request. Took a little over 2hrs for service provider to arrive. Originally 1hr but then delayed, so waited, locked outside, with bags. Overall door to door, 4 hours.
    • Difference between SES and Pinpoint: SES is basic functionality. Pinpoint is the more comprehensive (SMS and other comms methods, marketing campaigns, etc). SNS requires pinpoint for the number origination.
    • If everything finishes now, the 2 final candidates spots are to Alireza (highest rated, not already in) and Anish (most points on FIDE circuit, not already in).
    • Manually updated clocks for DST. Just oven, coffee maker, and autofeeder. Everything else is smart. The latter is particularly important – the autofeeder does not shift, but the apex does, so the pump-off window misses the feed window by an hour and the fish don’t eat – you have to be home during DST shift to fix.
    • Now been almost 2 weeks pending for the production access request in SES. Absolutely unacceptable on aws’ part. https://repost.aws/questions/QULBachhO7QkmUsrUEoeAcoA/ses-production-access-total-lack-of-response. Directly on the SES page: “The AWS Support team provides an initial response to your request within 24 hours.” Gave them some strong feedback on the support case.
    • Bubbletip spawned while I was gone!
    • Emptied trident waste. Garden and aquarium maintenance. For hydroponics, you can almost just multiply squirts by 5 to get mL (vs 20 v1 / 30 v2).
    • Paid ortho.
    • Supercontest. Started work on the cloudwatch ticket. Just adding awslogs to docker-compose for a container running in EC2; pipes the app output to cw where you can alarm/dashboard/etc on the data.