-
- Private work.
- Some regex refreshers. Remember
\b = word boundary (basically all characters except other letters/numbers/underscore.
- More pandas.
iloc, str.match
- Sleeping poorly for about a week.
- I should probably set up a gitlab-runner on my macbook? The desktop is not always running.
- You can read variable definitions from a conf file and pass as env vars to a command.
- Example in bash:
(source <path/to/conf> && echo $<var_from_conf>)
- The parentheses are obviously for a subshell. Then the env vars are defined ONLY in the subsequent command, not the host shell you’re running everything from.
- For redis it’s more efficient to use
SCAN than KEYS to list all keys.
- Supercontest.
- More with elasticache today. Played with
flask-caching and the redis cluster.
- Tested
make sync-db-dev for the first time. Worked well.
- Added a gnumake target to drop into the redis-cli.
- Deployed all infra changes (deps, redis, flask-caching init, auth, app init, etc), as well as the queries/selects splitout.
- Now just ready to ADD caching wherever appropriate (using
flask-caching‘s decorators). And test.
- Connect to the sbsc EC2 with vscode’s remote explorer for ide benefits (vs vim). Can have both windows open (wsl2 and remote), with separate terminals, etc.
- Was getting gunicorn timeouts in prod after deployment.
- The flask-caching default timeout is 300s. Gunicorn is 30, hence dying first.
- No long running queries or anything. Happening on boot.
- Confirmed redis-py is threadsafe, since I use gthread workers with gunicorn: https://redis-py.readthedocs.io/en/stable/advanced_features.html
- So it’s a connection issue. Then simply times out.
- Wrote a little script and debugged till I found it. Unlike host/pw/etc (directly exposed by flask-caching), redis-py expects
ssl=True (which flask-exposes through the generic CACHE_OPTIONS).
- My elasticache redis cluster requires tls, so it was quietly rejecting connections. Added the kwarg. Fixed.
- Tested the PROB restore for the first time, from
pg_restore. Still haven’t restored from an RDS snapshot.
- Was failing at first. “relations/seq/PKs/etc already exists”.
- Confirmed that
--clean --create was being passed, as usual. So it should drop everything before restoring from a dump.
- Worked fine in dev.
- Maybe something in RDS doesn’t allow drops?
- Nope. It was just that you can’t drop a db if there are open connections to it, so
--clean was failing. In dev, I stop the app container before restoring. I had deleted that from prod. Added it back.
- The line fetch was broken because (for the first time in 10 years) westgate added spaces between the team name and the asterisk for home. Fixed.
