Mostly Software

      • Supercontest:
        • Backed up the supercontest db, made sure everything worked (with old db models still today).
        • If a user honestly misses lockdown, but wants to pick before kickoff, you can manually add their picks (if you deem their case ok). Just `python manage.py shell` then import and run `commit_picks(…, verify=False)` for the user id and week id and teams, etc.
      • NFL Sunday. Remilins all day.
      • Smoked the 16lb ribeye roast.
      • Made gravy out of the trimmings:
        • Stock: Salt the trimmings in a pan, sear them with olive oil until very brown, deglaze with wine, add herbs/onions/whatever and water, simmer for hours (reduce) then filter.
        • Gravy: Whisk in flour (like 2tbsp per cup of liquid), boil, then turn off.
      • Moved bbq_log from docs to sheets. Lot cleaner now. Froze top row and left two columns during scrolling to make everything a lot easier to see. This was my 32nd smoke on the WSM today!
      • Stack is always lifo, way faster, every thread gets a stack, etc. Heap has much less restriction- it’s bigger and slower. It’s usually across multiple threads (usually one for whole app), data can be written/read in different orders, etc.
      • Quadruple-filtered the batch of oat milk today. Made an ok difference. Each time removes a bit of the suspension. Also added 6 pitted dates. Not the biggest change for half a gallon of milk. Gonna save the dates for the next round of protein bars: pecan butter, tahini, oats, dates, protein powder.
      • Really interesting story about Nick Diaz’ life: https://www.mmafighting.com/2015/9/14/9327767/nick-diaz-opens-old-wounds-on-a-dark-day-in-his-career. Didn’t graduate middle school, been fighting all his life. Biggest motivation was his girlfriend killing herself in high school. Now suspended for 5 years for having marijuana in his system. Frustrated that he can’t be there for his brother.
      • Bought round-trip flight to the bay for the tough mudder, oct 4-7. 15k miles and $11 in fees.
      • Significant rework on a lot of vim shortcuts, mostly.
        • Use on full tmux terminal for vim, and have vim handle the grid between panes for editing multiple files at once. This allows you to copy between files easily. Then keep other grids in tmux for different shell contexts, like python, bash, psql, git, logs, etc.
        • tmux:
          • ctrl-a then c to create tab
          • ctrl-a then n to move to next tab
          • ctrl-a then p to move to previous tab
          • ctrl-a then – to split window horizontally
          • ctrl-a then to split window vertically
          • ctrl-a then arrow to move windows
          • ctrl-a then x to close window, or tab if no more windows (confirm with y)
        • vim:
          • ctrl-w then c to create tab (with filetree explorer)
          • ctrl-w then n to move to next tab
          • ctrl-w then p to move to previous tab
          • ctrl-w then – to split window horizontally (with filetree explorer)
          • ctrl-w then to split window vertically (with filetree explorer)
          • ctrl-w then arrow to move windows
          • ctrl-w then x to close window, or tab if no more windows
        • ctags
          • ctrl-] to jump to def
          • ctrl-t to jump back
          • ctrl- to open def in new vim tab
        • There’s no great way to maximize the current window in vim, like ctrl-a then z does in tmux.
      • Good UFC card. Pereira is one of the craziest fighters I have ever seen. So much capoeira. He did a straight backflip on a downed opponent.
      • Supercontest.
        • Finished updating the core modules for the new db and model structure. Now just need to test with the live app.
        • Removed the read/commit from excel from source. This was a one-team use to pull from petty’s old data. If I need to reference it again, I can scrape the vcs archives. Nominally it won’t be needed ever again.
      • Pay transparency! https://onezero.medium.com/leak-of-microsoft-salaries-shows-fight-for-higher-compensation-3010c589b41e. Leaving your job and coming back into the same role to get a raise…should never be a thing.
      • Cool device that closes your windows when a coworker walks in lol: https://github.com/dekuNukem/daytripper.
      • Confirmed my plan covers 2 chiro visits per month at all facilities, not each facility. Called Pendergraft, going to buy a 3-visit pack for $90. First appt monday.
      • Round trip to the border of hermosa/manhattan, where the street numbers restart, along the strand, is 3.6 miles.
      • Ran a 5k (3.1miles). Felt pretty good. A little over 20min.
      • Met with Roubik, Stacey, and an engineer from 5m. I liked them, and understand the product uniqueness. I don’t know enough about the market to assess their viability and future.
      • Got quest bloodwork report back. Everything good except creatinine, slightly higher than the upper limit. Range is 0.60-1.35, I had 1.44. I think this was the same as last time? Electrolytes were good though, I think renal function is fine. False positive due to creatine supplementation, likely.
        • Lipid panel was fantastic. Cholesterol numbers solid, even with my weekly meat smoking.
      • Import from the distribution name at the toplevel (absolute), but then within each package (__init__, modules) feel free to import relatively. When referencing another package (within the same distribution), go back to absolute imports.
      • Supercontest.
        • Made all the get_*_picks() queries combine into one. Can query on season, week, or user, modularly. Returns full pick rows.
        • Finished going through views, cores, and all other modules to change the db interactions around. Major re-org.
      • Always a cool video about encryption and diffie-helman: https://www.youtube.com/watch?v=3QnD2c4Xovk. You just need a function that’s easy in one direction and hard in another. Think paint. Easy to determine what two color mix to make, hard to determine what two colors went in a mixture.
        • Start with one color. Server and client each have a private color that they mix with the public color. Then send it over. Then mix private color again. Each then has a 3 color mixture of their private, the other’s private, and the public color. A third party sniffer knows the combo of each private+public, but doesn’t know all 3 because it has never seen the privates without being mixed with the public.
        • Instead of paint, a discrete logarithm is usually used. Example: 3^Xmod17. Each party’s private is an X. Easy to calculate what that would yield, very hard to figure out what X is if only the answer is known (especially as the numbers get large).
      • The n+1 query problem in databases is for one-many relationships (or many-many, as well). You query once for the parent and then n times for the children, which can be very overloading if there are a lot of queries. This is called lazy loading. It’s the default for most ORMs. It’s faster if the relationships are small.
        • The opposite is called eager loading. It’s simply a join on the data. For large datasets, it’s much faster.
      • Flask extension research, again:
      • New manhattan softball league started last night.
      • France eliminated USA from the FIBA world cup in the quarterfinal lol.
      • Supercontest.
        • My migration was trying to move scores from lines with UPDATE, but scores is a new table. Needed INSERT INTO. Changed.
        • Query for the full leaderboard (without totals) is:
          • old models: select season, week, email, sum(points) from picks, users where picks.user_id = users.id group by season, week, email order by season asc, week asc, sum(points) desc;
          • new models: 
        • Lb totals:
          • select row_number() over (order by sum(points) desc) as row_number, email, sum(points) from picks, users where picks.user_id = users.id and season = 2019 group by email order by sum(points) desc;
        • Without rankings:
          • select sum(points) from picks, users where picks.user_id = users.id and season = 2019 group by email order by sum(points) desc;
          • Change year as necessary.
        • SUM in postgres returns null if the sum is zero, so this only shows people who scored a point over the whole season. You can use max(coalesce(col, 0)) to avoid that.
        • Cycled this to test db: restore backup, upgrade, query.
        • Quick interim ticket with UI clarifications: https://github.com/brianmahlstedt/supercontest/issues/79
          • Highlighting pick capability on the matchups tab.
          • Sorting second tier after datetime on matchups. Sorting team abbvs on picks tab.
          • LB shows percentage now.
          • Niners->Warriors in adspace.
        • Made the graph endpoint respect “don’t show this week’s results until complete” just like the leaderboard does for its colorization. Abstracted that logic out into an initializer for both views.
        • Added the INSERTs to the user-season association table in the migration. Everyone goes to 19, a list is crosschecked against 18.
        • Made a ticket to remove the src volume in the production app, and bundle all the static content with flask-assets: https://github.com/brianmahlstedt/supercontest/issues/81.
        • TNF game was delayed due to lightning. The nfl’s xml scoresheet reported back “Suspended” as the status. The app handled it just fine.
        • Changed manage.py’s shell context to inject the symbols for the model names explicitly, not just the models object. Now you can copy-paste exact queries from the app (db.session.query(User) instead of db.session.query(models.User)).
        • Abstracted all interactions with the db to a package called dbsession, with modules `queries` and `commits`. This will make the application side of migrations much easier to update.
        • You can incrementally build tables for further querying in sqlalchemy, like you would naturally do with nested SELECT and JOIN statements in SQL. This is super useful for layered relationships, like Pick -> Line -> Week -> Season. The trick is calling .subquery() at the end of your query. Then, you can query it (or join it) later. To access the labelled columns, you use c.
          • See supercontest.dbsession.queries for examples.
          • You can rename columns with .label(‘new_name’) if you want to change name, deduplicate, or do math and create a new col, etc.
        • I’m starting to agree that ORM usage can be way more complicated and abstract than SQL for complex queries.
        • All of this model and query redesign made the views module much slimmer. Almost all logic lives outside the routes, which is the way it should be.
        • To query on a relationship column, use .has().
      • Westgate Supercontest.
        • Entry fee is $1500.
        • Winner last year was at 70.1%, 59.5 total out of 85 (17*5). He won $1,422, 214.20.
        • There were 3,123 contestants for a total of $4,309,740.
      • Generated a personal access token for octotree (it uses the free auth for the first X requests).
      • In sqlalchemy, use .one() if you expect one and only one result. Returns the object, not a list like .all(). If there might be multiple, use .first(). This won’t raise exceptions like .one() will.
      • In python dir(var) will give private and dunder attributes, whereas var.__dict__ will just give the main ones (better).
      • Uber laid off about 8% of its workforce.
      • Tax-gain harvesting. Just balancing tax brackets for the year. If your income or investments are lower than usual at some point, then you can sell some of your better stocks (and usually buy them back immediately). Do this to maximize your current income tax bracket (10-12-22-24-32-35-37) and cap gains tax bracket (0-15-20). This allows you to pay a portion of the tax at a time that is cheapest for you.
        • The opposite is tax-loss harvesting. If you have stocks that have suffered, and your year had particularly high capital gains, then you can sell them at a loss to offset. Then you buy them back immediately, effectively shifting your basis in that holding to a lower amount.
      • Did a little fantasy research to set lineups for this week. Forgot how annoying it was lol. Watching the games is more fun without that stress.
      • Bought and smoked a whole octopus.
        • $13/lb, 4.7lbs. This was from redondo pier – I’ll try an asian market next time, probably a little cheaper.
        • Frozen is fine, unlike most poultry/game/cattle. They’ll usually have the beak/eyes/guts removed, but do so if the market didn’t.
        • Let it sit out at room temp for about an hour, with a fan, before cooking. This creates a pellicle – a dried outer layer which smoke can cling to much better than the natural underwater skin.
        • Pepper dry rub with maple syrup binder. Mesquite chunks.
        • Take to ~140. About 2hrs with the smoker @150.
        • Octopus is super low in fats and carbs. Super high in protein and cholesterol.
        • They’re in the cephalapod class, but above that they’re in the mollusk phylum!
      • Chopped and prepared all veggies for smoothies and juices.
      • SQL.
        • Today was almost all SQL.
        • You can query distinct on multiple columns, and it will return rows with all the permutations.
        • Remember kids, don’t hardcode static values into your migrations. If it’s based on the current data (moving rows/cols, checking max, etc), but that update infer from the existing data. Also: write and test your downgrades!!
        • Foreign keys need definition in the database, and therefore need to be in the alembic migrations. Relationships, however, do not! They’re only on the sqlalchemy side, the portion that decides how to write to the db. But the db structure itself, the sql side, does not have knowledge of a “relationship”. You still need to define the association tables, foreign keys, etc in the migrations.
          • Along the same lines – whether or not the many-many relationship here is unidirectional or bidirectional is purely in the python side as well. The raw db has an association table. That’s it. The ORM can choose to reference in both directions on both original tables, or just one.
        • Remember, SELECT statements (and subqueries) just return rows. That’s why you can pass them to an INSERT statement or likewise. That’s also why you can nest them.
        • Example join for common supercontest use case:
          • select email, team from users inner join picks on users.id = picks.user_id where season = 2019;
          • You can be specific about what you want back if the tables share column names:
            • select users.id, picks.points from users inner join picks on users.id = picks.user_id;
        • Outer join does an inner join first, then checks a condition on one of the tables. Left checks the condition for the first table in the join. Right checks the condition on the second A full outer join does both.
        • Cross join does permutations, returning a table with all the columns from both tables in the join.
        • Remember, can’t add a non-nullable col to existing rows. Needs a default, or add it regular and execute whatever inserts you want, then add the nullable=False constraint last.
        • Listing all fk constraints for a table is a long query: https://stackoverflow.com/a/1152321.
        • If you do `FROM table1, table2` postgres does an implicit CROSS JOIN, for all permutations. You usually don’t want this naked, but often the query comes in the form `FROM table1, table2 WHERE table1.col1 = table2.col2`. This implicitly makes it an INNER JOIN. It’s the same syntax as `FROM table1 INNER JOIN table2 on col1 = col2`.
        • You can join more than two, as well. Example syntax: `FROM table1, table2, table3`.
        • For postgresql and alembic, you have to manually create the sequence in the migration before creating the table:
          • from sqlalchemy.schema import Sequence, CreateSequence
          • op.execute(CreateSequence(Sequence(‘groups_field_seq’)))
        • INSERT INTO requires parentheses around the columns that follow. SELECT should not have parentheses around the multiple values. If you put parentheses, it tries to combine the contents into a compound value.
      • Supercontest.
        • Small bug in the prod app. sorted_user_ids wasn’t defined in the picks view when it’s in “only-me” mode wed-sat. It’s just your id, one line change.
        • Added the week 2 lines. Backed up before and after.
        • Verified that “last week” colorizes on the leaderboard now. Verified all the new picking UI functionalities work. The light blue is a great change. The /picks tab shows yours on wed-sat also (only yours).
        • Made my picks for this week.
        • Overall db change in hierarchy, following foreign keys and relationships:
          • Score -> Line -> Week -> Season
          • Pick -> Line -> Week -> Season
          • Pick -> User
          • User -> Season
        • Manually created the extensive migration to the new table structure. Some changes were easier: table creation, rename, not nullable, fk, etc. Some changes were harder: conditional updates, moving cols, etc. Practiced a lot of SQL.
        • The final migration (just upgrade) was over 100 lines. Not great.
        • While you should write downgrades whenever possible, some migrations are not reversible. This is one of them. Data was lost (Pick.points, Matchup.winner, etc). It’s not worth recalculating those and programmatically allowing a downgrade. You have backups from before then.
        • Got the db to upgrade. Now going to test with sql (manually) and update the app to use the new structure.
        • Rebased on the master prod changes.
      • Supercontest.
        • Made the leaderboard current_user row highlight with border like the picks current_user row.
        • Standardized all the flask_user templates. Made them extend my bases, so the headers are consistent throughout the whole site.
        • Changed all the container/row/cols to be more responsive, and optimized for different viewports.
        • Changed the matchups table to white-space:nowrap. This was specifically for the datetime column, but applies to all cols and all rows. Nothing should span two lines. As the viewport shrinks, it should scroll overflow.
        • Added make enter-dev/prod-app for ease.
        • A recent alembic migration had iteritems(), had to upgrade to items() now that the app is py3.
        • Deleted all remote branches and pruned refs.
        • Deployed the cosmetic and responsive changes from the first #68 branch.
        • Backed up before all prod changes to be sure.
        • Migration:
          • Pluralized table names for easy querying.
          • Dropped default values for user cols: email, is_active, password.
          • Made user cols nullable: first_name, last_name.
          • Set the val/nextval in the sequences for all 3 tables to match max(id).
        • Merged and deployed the changes to the existing db first. Adding new season table and all the others next.
        • Made logout tab dynamically change to login based on auth state.
        • Gigantic table redesign (full source changes to the model captured here https://github.com/brianmahlstedt/supercontest/commit/0684fdc731d975bbdb80a8a961cc2d83cda1aea2):
          • New Season table, with sequence on id. year col, int, not nullable.
          • Add seasons col to User table, many-many relationship with Season. Unidirectional.
          • New Week table, with seq on id. Two col, season_id (fk), and week. Both ints, both non nullable.
          • Rename Matchup to Line table. Include sequence. id, season_id (fk), week_id (fk), favored_team, underdog_team, datetime, line, and home_team moved. The favored_team_score, underdog_team_score, and status cols moved to the new Score table. Winner moved to a new Coverer table. home_team became nullable=False.
          • The Pick table lost its season and week cols. It now uses a foreign key to the line table for the id of the matchup that contained the team it picked. Also dropped the points col (did not move). The cols user and line became not nullable.
          • New Score table. Id pk with seq. Line_id, fk, int, not nullable. Moved favored_team_score and underdog_team_score and status from the old matchups table here, and made them not nullable.
        • I had initially created PickPoints, WeekPoints, and SeasonPoints tables, but I think it’s better to recalc than to aggregate and store. This is a design decision based on application usage. Scores can change 3 days out of the 7. If ever a score changes, the points for that pick changes, which requires a recalc of PickPoints, WeekPoints, and SeasonPoints. So storing the calc output doesn’t really save much. If you data is more static, then it’s better to store the calcs.
      • You can customize the graphiql template, but it’s not easy. Flask-graphql allows you to pass a template string, but it’s a full react app, so you can break it quite easily. I would have to copy over the jumbotron and settings_navs manually, since jinja can’t integrate them, so I’ll leave it as-is.
      • Watched the rob lowe and bruce willis roasts in the background while working. The new alec baldwin roast is coming up this sunday.
      • Upgraded the mint link to BoA.
      • col, col-12, and col-xs-12 are all the same thing. xs and 12 are the defaults. The breakpoint setting (the viewport size) is, remember, a minimum, so it applies to all screens that are larger. Therefore they all mean one column, across the full width, on all screens.
      • Flask-sqlalchemy will autoincrement the first primary key with integer type.
      • Coordinated with Roubik (Nel’s dad), going to meet Thursday in Glendale.
      • A tablespoon of my homemade tahini in a big cup of coffee = delicious. Adds some good fats and a nutty flavor.
      • Placed Fresh order.
      • Useful psql:
        • List all sequences:
          • select c.relname from pg_class c where c.relkind = ‘S’;
        • List all defaults:
          • select column_name, column_default from information_schema.columns where (table_schema, table_name) = (‘public’, ‘<mytable>’) order by ordinal position;
        • Drop a default:
          • alter table public.<table> alter column <column> drop default;
        • List cols as nullable or not:
          • select table_name, column_name, is_nullable from information_schema.columns where table_name = ‘<mytable>’;
        • List indexes:
          • select * from pg_index where tablename not like ‘pg%’;
      • Remember, don’t execute crucial sql in the database without including it in the alembic upgrades/downgrades. You can put arbitrary op.execute(<sql>) to make sure that all you manual actions are captured programmatically.
      • It is understandable, but still kinda sucks how manual db migrations still are. Alembic can’t auto-understand every change to your model, especially when abstracted away through an ORM like sqlalchemy, and flask-sqlalchemy.
      • Basic relationship patterns: many-one, one-many, one-one, many-many.
        • For many-one or one-many, you simply specify a foreign key to another table (usually pulling in the id). If you want actually attribute the whole row (like make all the children available in the parent object), then specify the FK as well as a relationship with back_populates. You can go both directions.
        • One-one is easy. It’s just many-one or one-many with uselist=False, where you simply attribute a scalar for the parent/child instead of many.
        • Many-many isn’t that bad. You require an extra table called an association table which connects the column in each of the left and right tables that you want to map. Then you simply reference the association table in each of the two tables you want to associate. You put in both if you want bidirectional (ie child object has parents col and parent obj has a children col), or you can only put it in one if you want.
      • I want to smoke a big rib roast next time. This is basically the same meat as the ribeye (the best steak), but in a much larger roast cut instead of a steak cut. Slices of a big rib roast are what prime rib is.
        • It is very expensive, obviously. Even a good cheap distributor is gonna charge $15-20/lb for a prime rib roast. Wagyu ribeye can be over $100/lb.
        • Costco sells a boneless 7lb for $130.
      • Art mentioned biologics and biosimilars: https://www.phrma.org/advocacy/research-development/biologics-biosimilars.
        • Did another domain scrape. curebench and pillemporium are both available. I love em.
      • When designing the models, you can just put the foreign key IDs without backpopulation/backref. This keeps your tables leaner, but it makes your queries heavier (because you have to join). This is obvious, but is a pretty important decision when designing a system.