October 2019 – Page 4 – Mostly Software

Oct 10, 2019

Gitlab/sentry:

I don’t think you can default an assignee for tickets or MRs in gitlab (yet).
Moved my project from my user namespace to a group. This is like an epic across projects, combining them together to see all tickets, etc. You can also see cycle analytics.

Updated my remote urls to bmahlstedt-group/supercontest.git (updated namespace). Did this on my local machine and sbsc prod.

Resolved the old AnonymousUserMixin sentry issue.
Added the supercontest repo to sentry, and integrated the app in gitlab, so all my source commits are available to sentry.

Blandish – to flatter someone manipulatively.
MLB postseason is now in the semifinals. Nationals and Cardinals in the NLCS, Yankees and Astros? in ALCS.
The carnivore diet is absent of vitamin C, vitamin E, and fiber.
flask-user utilizes flask-login, not flask-security.
Tony Bennett is currently 93 years old. Frank Sinatra died at age 82 in 1998.
Supercontest.

Fixed a few small bugs. Some wording.
Added sentry releases.
Rejiggered the picks_open logic to natively include is_current_week. Restructured the IF logic in the your-picks/all-picks routes to be extremely clear about the scenarios, and which info bubbles to present to the user based on timing (no lines, no picks, post-picks, picks-closed, etc).
Integrated flask-admin.

The main benefit is that you get ModelViews out of the box, allowing a web interface to create/edit/delete/search the database.
The secondary benefit is just organization. My dashboard, scheduler, and all the other custom admin views for the supercontest can be added to the general /admin navbar.
Added the model views. Protected with CRSF (flask_admin.forms.SecureForm) and restricted to an authenticated user with the admin role.

I now have 8 tables in the app.
There was an apscheduler os write error that sentry reported: https://sentry.io/organizations/bmahlstedt-org/issues/1266918507/?project=1773879. Not sure.

Remembering 15 years ago to college applications, since I’m starting to do the same with job apps; I’m much more of a sniper than a machine gunner.
I’ve used ~100m of 2000 (5%) of my shared runner pipeline quota.

Oct 9, 2019

Supercontest.

Added a global @app.before_request to print the email of the user for all requests, to make the logs more clear.
Made the matchups js a lot smarter about iterating through picks. Instead of going over all matchups rows and comparing to picks, it gives the ids to the td cells and then targets the specific row, just iterating over the picks instead of all matchups.
It now adds a 2px border to cells for complete games, to distinguish between what is finished and what isn’t. This happens on the your-picks and all-picks views.
Fixed the emailer. The problem was the asyncronous job adding. APscheduler is a piece of shit, from all my experience with it so far. Jobs added before scheduler.start() will trigger properly, but having another view call add_job later has never successfully triggered that job. Now, it just adds the email reminder at app start (for prod app only). I left the scheduler admin view around, but put warnings that it doesn’t really work due to apscheduler.
I’ve been more conscious about pushing, now that I have CI set up. I batch a lot more local commits now. This will save the 2k min limit on the shared runners.
In setting up CI/CD on the droplet yesterday, I had accidentally removed the public key in /home/bmahlstedt/.ssh/authorized_keys (the one that corresponds to my private .ssh/digitalocean). I added it back (/root/.ssh still had it), and I could ssh again.
In moving source checkouts over from github to gitlab, I lost all the backups (they aren’t tracked in vcs). This is fine, I have a new one with all the current data.
Changed the matchups template to use picks_open as well.
Added cards to the messages above each view’s table. Styled better.
Added the roles table, create admin, added myself. Updated all views to use this properly. https://gitlab.com/bmahlstedt/supercontest/issues/115.
First MR, ticket closure, master kickoff, deploy; all that ci/cd went well in gitlab.
Wednesday deployment, changed banner.
Quick bugfix with picks: https://gitlab.com/bmahlstedt/supercontest/issues/117.

Sentry:

List of importable integrations: https://github.com/getsentry/sentry-python/tree/master/sentry_sdk/integrations.
Added sqlalchemy.
You can’t add the gitlab integration for sentry unless your project is in a group (not owned by single user) – https://github.com/getsentry/sentry-plugins/issues/458.
Changed sentry to only run in prod mode, so the local dev instance doesn’t eat up my sentry monthly event limit.
The uncaught “AnonymousUserMixin object has no attribute id” errors were from queries.is_user_in_league in the url_value_preprocessor. I protected that with if current_user.is_authenticated, so we shouldn’t see those anymore.

Protected all other current_user calls too (FMD’s get_current_user_email, etc).
Note for future apps: if you’re ever calling current_user outside of a view or other function where the user is guaranteed to be logged in (protected by @login_required or such), you should condition the current_user object on is_authenticated.

Left the email notifications for every exception ON. I don’t anticipate spam, now that the AnonymousUserMixin error is fixed.

Flask debug toolbar changed its default to disable the profiler. I re-enabled it.
Lol not the biggest jon jones fan but completely agree with this: https://www.reddit.com/r/MMA/comments/df49ql/jon_jones_to_izzy_im_not_out_here_searching_for/. Izzy is annoying.
Created an radnet account to access my MRI results.

https://myradiologyconnectportal.com/Exam.
Bad: “There are mild degenerative changes of the coronoid process.”
Good: “There are no areas of abnormal signal involving the trochlea or capitellum. There are no areas of abnormal signal involving the radial head. The radial collateral ligament and attachment of the common extensor tendon and ulnar collateral ligament and attachment of the common flexor tendons are normal. There is no triceps or biceps tendon tear. There are no soft tissue masses. No ulnar nerve lesion is identified.“

You can make waiver claims on tuesday night for ff. I had thought that it was wednesday all season so far.
Ortho appointment in Van Nuys. Basically just said it’s ok, go to PT. 10 appointments concluded with 0 treatment.
Picked up DL from Brews Brothers after.
Ate lunch at Dave’s Hot Chicken in NoHo. It was amazing. First time I’ve been impressed with outside food in a long time. 20min wait around noon.
Loving Terry Reid’s music. Always liked seed of memory, but had never heard faith to arise or any of the others. Unfortunately, that whole album isn’t on spotify. He sounds a lot like Robert Plant, and the music is chill.
pylint no-else-return has always been a dumb rule. Flow is so much more clear when it’s explicit about the else.
I’ve said it many times before, but unfamiliarity is the root of most discrimination. Racism would be nearly extinct if the whole world spoke the same language. Then you could tackle the minority remainder, the heart of the problem; anger.
In gitlab, you don’t need to put a pound sign in the branch name! This makes command-line-git a lot easier. You still put it in the commit message though.

Oct 8, 2019

My 15(?) year old wix website has finally been cancelled (which is probably for the better).
“And the day came when the risk to remain tight in a bud was more painful than the risk it took to blossom.” – Anais Nin
Finished Once Upon a Time in Hollywood.
Smoked the pastrami beef ribs that had cured for 6 days. Amazing flavor.

Next set of beef ribs, I want to try wrapping earlier, closer to the time you would for brisket. I’m curious how much it changes tenderness vs bark.

Paid off the remaining balance on the sequoia hospital bill from years ago; was like $220.
Cancelled the amex.

They tried to give two incentives, one for miles (20k) and one for cashback ($175), both requiring that you spend 3k in 3 months.
The companion pass is forfeited (I had 1), and the annual fee is not prorated (there were 4 months left in that this billed year).
Removed my amex from google play. Removed the amex browser bookmark. Mint automatically made it inactive, I didn’t have to do anything!

I would like the Vitamix A3500 – it’s so much better than my ninja, and I use it at least once daily. It’s $525. I’ll probably wait until after I move.
The Strassburg sock is definitely helpful for plantar faciitis. WFH makes it a lot easier to get hours in during the day, since I find it annoying to fall asleep wearing.
Put the 2020 reg on the Ducati. It’s yellow, so it looks better than the blue 2019 sticker.
Watch Rob Zombie’s new 3 From Hell, the third installment in the series after House of 1000 Corpses -> The Devil’s Rejects.
The carolina reaper single chip is sold out online, but I just discovered that the riteaid on aviation/artesia has these 3 flavors:

I like the haunted ghost pepper the best, then salsa verde, then chile limon.

Kubernetes/GCP.

Good article on kubernetes: https://www.digitalocean.com/community/tutorials/an-introduction-to-kubernetes. Very useful for orchestrating many containers across many machines together. Metrics, storage options, scalability, ML resources, much more through the full GCP suite.
Created a project (supercontest) on GCP and removed the default API Project and My First Project.
Enabled the Google Compute Engine API for this project.
n1-standard-1 machine, just 1 vCPU and 3.75G mem (the smallest standard machine).
Billing is by resource. It’s like 3 cents per cpu hour and 0.5 cents per gb mem hour, so the n1-standard-1 is about 5 cents per hour, or about a dollar a day, or about $30/mo. My digital ocean droplet is $5/mo.
With the $300 promotion on signup, looks like I can test this for free pretty easily.
Ended up staying with digital ocean for now. I don’t need any of the other GCP products (compute, storage, ml, etc). The one attractive feature is scalability, but I can upgrade with DO as well.

GPG key = GNU Privacy Guard.
Sentry.

Created account, added the flask integration to supercontest.
For the free plan, you can record 5,000 errors per month and it keeps history 1 month back. That’s 160 errors a day.
Linked my gitlab project to the sentry api, created auth token.

GitLab.

If your project is public, and you have gitlab ultimate, then you get a security dashboard and dependency list. Pretty cool, but I obviously can’t take advantage of this.
In general, I’ve been very pleased with the experience on this platform. Moreso than the atlassian stack, jenkins, etc.
Added the deploy stage to my yml. Added the variable SSH_PRIVATE_KEY (for the droplet) to gitlab. Added it as a known host directly through ssh-keyscan.

Gitlab runs the deployment as root, and the deployment has ansible pull new source on southbaysupercontest.com, so I had to make sure the machine had the proper ssh keys under the root account (not just mine). sudo cp -r ~/.ssh/ /root

Ansible doesn’t maintain docker images officially anymore, but some user groups keep them around (albeit not very current). Also…you have to specifically install ansible after you pull the image lol.
Did not integrate prometheus; there’s already a metrics agent that digitalocean provides to its droplets. I put a link to the external DO dashboard in gitlab tho.
Updated the readme to indicate where metrics were, as well as SCM, CI/CD, etc. The GitLab and DigitalOcean logos (clearbit) look good.
Modified my view to be fluid.
The minutes you’ve used in CI/CD are at https://gitlab.com/profile/pipeline_quota.
Integrated with sentry so that error tracking was in gitlab as well.
I don’t need to use the container registry associated with the project, although this feature is great. My workflow is not build -> test -> pushImage -> deploy, so a record of all the image artifacts in a registry is less important. I test the source and then build the image right at deploy time, which is more than satisfactory until I start scaling the test/review app portion of things.

Sometimes dpkg will prompt for config changes (like apt does for install y/n). Use -o DPkg::Options::=”–force-confnew”
Changed my mem usage alert on the droplet from 90 to 95%.

Oct 7, 2019

Gitlab.

Migrated all projects from github to gitlab.

Added the public ssh keys from both bmahlstedt-xps13 and southbaysupercontest.com.
Delete the checked out repos, then recloned from the gitlab url (you could also just set the remote url).
Privatized and archived all my projects on github, with a pointer to gitlab.

Cool features:

Activity charts and filetypes: https://gitlab.com/bmahlstedt/supercontest/-/graphs/master/charts.
All contributor information, and when: https://gitlab.com/bmahlstedt/supercontest/-/graphs/master.
Great CI/CD, security/compliance, operations, package support, more.
Commit heatmap: https://gitlab.com/bmahlstedt.

Got initial CI set up to run test-python on commits. Using shared runners for now, 2000 min/month should be enough. The wait times are zero. If needed, I’ll register my own machine as the runner (since I’m the only developer, and the machine is guaranteed up).

Using custom stages, rather than autodevops. That’s a cool feature though.

Added build and coverage badges to the readme.

Google Cloud Platform.

Kubernetes is obviously the google production container orchestration platform. Gitlab has lots of integrations with it, as well as Google Cloud Platform in general. I should try it out.
GKE = Google Kubernetes Engine.
Started a trial with GCP.

Removed pay-with-points from my amazon card. You’re losing the cashback if you pay with points. Just always use the card, and withdraw all the cashback from the chase account whenever you want.
Homemade protein bars for my mom yesterday. Pecans, oats, prunes, cinnamon, protein.
Won both fantasy games. Went from 0-4 in yahoo, last place, to highest scorer of the week. 2.5 in sbsc.
Placed fresh order.
Supercontest.

Put in the niners pick for jcriss and econstan.
Changed FMD to monitoring level 1 for the user and scheduler blueprints, while leaving my main and season blueprints at 3.

Finished the algorithms and data structures book, added all notes to google drive.
Ordered a small bedroom trash can, since amazon lost the other delivery (got refund).
I would like to do a hanging display for all the tough mudder headbands I have. The most basic is just wood blocks, maybe 1×8″ with 1/2″ thickness, then hook screws connecting them on both sides.
There’s a ufc event every Saturday for the next 6 Saturdays, and they’re all good.

Oct 6, 2019

Tough mudder yesterday was great. It was on the easier side (only 8 miles), but I saw a few new obstacles and got the pink headband. Awesome to do it with Eric. Plantar fasciitis did surprisingly ok.
Fogo de chao. Prime rib and filet were good, as always. Beef rib, sirloin, new york, pork rib, picanha, chicken – everything else just can’t compare to the smoker.
Cal lost to the ducks, warriors lost to the lakers (pregame).
Watched the comain and main for ufc 243. Dan hooker is awesome. Adesanya looked fantastic but is annoying as hell. Too cocky and had one of the dumbest walkouts I’ve ever seen. Tried to make it a choreogrpahed performance like WWE and boxing. Gtsoh.
Supercontest

Entering the python shell is not resource-cheap. It creates an entirely new instance of the app. Not with a server, but will all the other frills; apscheduler, etc.
Submitted late picks for Cam, Wes, Raj, Jesse Thompson, Ben Yezer, and Phil Twist.

Credit card research.

BoA preferred rewards offers 5.25/3.5/1.75, which is pretty damn good but you have to have 100k+ in your combined BoA/Merrill accounts. Otherwise it’s 3/2/1. If you have that much sitting in your account, then you can gain a lot more by moving it somewhere with higher growth rather than reaping higher cashback.
While looking at NEW cards, I saw that there’s a better version of my old card. This is a ridiculous mistake. I’ve had an Amazon Rewards card since July 2013, which earns 3% back on Amazon. If you have Prime, which I do, you can upgrade to the Amazon Prime Rewards card, which gets 5% back. It did not automatically upgrade me, and I never noticed. I exported an order report, and I have spent $30,000 since July 2013 on Amazon. I missed $600 free dollars by not clicking the button. Chase will ship a new card automatically, even if the upgrade was done on amazon.com (I called Chase to confirm).
Made my last payment on AmEx Platinum Delta Skymiles, will cancel in the next couple days. I no longer do business with Delta. The miles won’t expire as long as I maintain my Delta account; the amex card is not necessary. I can finish off the remaining ~135k miles whenever I want, as well as the buddy pass this year.
Looking for purely cash back. Gas is easy with the motorcycle. Not a big traveler, don’t need air/hotel. Already a bomb chef, don’t need restaurant rewards.
Chase ultimate rewards program is good for certain use cases. You can regard all “points” as basically 1% cashback, with some variation. In reality, a point can be anywhere from 0.8 to 1.5 cents, and can change based on what you’re redeeming them for as well. The most common use case is for travel, where you can get 1.25-1.5x the value of your points. You can also share points between cards, which is useful for people with business cards – they can use the points earned on work trips toward their personal rewards program. It’s only raw chase cards – cobranded cards (like my amazon) don’t share. Because of this, and the de-emphasis on travel, I don’t need a card in the chase ultimate program in particular.
Cashback-focal cards with no annual fee and no rotating schedule:

Citi double cash. 2%.
HSBC cash rewards mastercard. 1.5%. $150 bonus once if you spend 10k in first year.
Paypal cashback mastercard. 2%.
Capital one quicksilver cash rewards. 1.5%. $150 bonus once if you spend 0.5k in first 3 months.
Chase freedom unlimited. 1.5%. $300 bonus if you spend 20k in first year.
(Chase freedom doesn’t count because they rotate categories, which I don’t need).
Amex cash magnet. 1.5%. $150 bonus once if you spend 1k in the first 3 months.

Decided to go with citi double cash. Highest basic cashback. I already have visa through my amazon prime rewards card, and this one is mastercard. Applied and accepted.

Common, boring mistakes: https://medium.com/signal-v-noise/the-4-questions-you-should-stop-asking-during-your-one-on-one-meetings-ed7431da11aa.
Cool NFL data of score combo possibilities: https://nflscorigami.com/.

Oct 4, 2019

Redis can read a json 16x faster than postgres.
“Vice signaling” (as opposed to virtue signaling) is a cool trait. Admitting one’s faults, weaknesses, etc. I like that in people.
Petty finally sent me the list of people who had paid so far. Added them to the prod table, lb looks much better now.

enter-python-prod
paid_league_2019 = db.session.query(League).filter(League.season_id == Season.id, Season.season == 2019, League.name == ‘Paid’).one()
user = db.session.query(User).filter(User.email.like(‘%petty%’)).one()
paid_league_2019.users.append(user)
Repeat 3-4 for any other users
db.session.commit()

33 so far. More are planning to pay (Art, Wes, Harner, …).

Oct 3, 2019

Supercontest.

Created and finished a small handful of improvements: https://github.com/brianmahlstedt/supercontest/issues/110.
Changed that stupid logic where every new endpoint would have to be excepted in nav_active. Now it actively highlights profile if it sees edit_user_profile in the route. If there are others in the Profile bucket, I’ll add them as they come.
Fixed the FMD, adding grouping by user email. The actual problem was solved by the docker system prune and docker volume rm (for the fmd only).
Added an awesome admin navbar (custom).
Redesigned the whole schedule interface. No jobs are automatically started anymore, and none are started at application started. Wrote a nice interface in the admin panel that allows you to kick off each job individually, and view the currently running jobs.

Split it so that the lines job does not trigger the scores job. You must add both (and then they auto-trigger to actually start, of course).
This makes it nice because you can individually tell the scores job to look in the db at already-existing lines and add future jobs for them. Otherwise, if the app crashed after the lines were ingested on wednesday, you’d have to go back in and add the score jobs again manually.

Continually unimpressed by how fickle apscheduler is. Having fought more against it, I decided it was too unreliable to use in production. It was committing duplicate lines, not running at other times, and much more. It can’t even report its own status accurately. I’m going to leave it in place for the harmless, non-self-modifying jobs (email reminder). I’m not going to use it for the repeat-until-post westgate line fetch, or introspect-live-scoring.

MemoryJobStore is fine if you don’t need to persist jobs across something like an application crash. If so, the SQLAlchemyJobStore would be easy to change to. You can use sqlite or postgres.
The usual docker system prune -af, on both dev and prod envs. Wiped the volume for FMD, not the regular sbsc db. The usual git remote prune origin as well.
You have to pass the csrf token in form submission (if the action is going somewhere that has csrf protection, like my site). It’s pretty easy: https://flask-wtf.readthedocs.io/en/stable/csrf.html#html-forms.