• Apr 30, 2019

  • MD. Articles on ML and React component testing.
  • Jasmine.
  • Ahhhh, played with jasmine a bit more. “$ is not defined” and such is because I was using jasmine-node. jQuery requires the browser’s DOM api, which node does not have. You have to use the html runner for jasmine (so you can add it as a usual script, from a cdn or otherwise).
  • Here is a library that allows you to load html/css/other fixtures into the jasmine context: https://www.npmjs.com/package/jasmine-jquery.
  • You could still just run tests with a headless browser to exercise the frontend, which I’m already doing. It’s just a bit easier to unittest js functions directly with jasmine.
  • Opting to not pursue this further for now, coverage is already sufficient.
  • Installed pkg-config zlib1g-dev openjdk-8-jdk bazel
  • ipython was busted because of an incompatible version of prompt-toolkit (probably from the jupyter install yesterday), so I reinstalled that package.
  • Remember at the ipython prompt you can type ! before a command to execute something in the system shell (useful for pip installations and such).
  • Lightning in a bottle schedule was released. Didn’t really look at it much, just a quick skim.
  • Webscraping.
  1. (preferable) Use an exposed API, with structure that is designed for programmatic requests of data.
  2. HTTP with form assistance. Use a library like mechanize or robobrowser to handle all the browser defaults for the form fields you don’t care about (everything except un/pw). This handles session.
  3. Pure HTTP. Use devtools to watch the network activity of your manual request. Copy all the form data. Use a requests session to persist returned cookies and other things across subsequent requests.
  4. Selenium. Simulate the frontend and perform your usual data entry / clicking.
  • Pharma.
  • Logged into new.andanet.com and searched for some drug prices, then tried to fetch the data programmatically and serve it in an aggregate marketplace across a few vendors.
  • Mechanize doesn’t support py3 and robobrowser is newer, so I used that. Both robobrowser and pure requests would not return the login form. “Your current web browser session has been closed due to inactivity. Please login to create a new session.” This is likely because neither uses an actual browser, which is an interesting deterrent to programmatic interaction. Let me try with Selenium next.
  • Selenium was able to successfully log in. The next problem is the 2FA. It emails a verification code. I could automate this with Selenium as well, fetching the code from email and clicking the second submit, but then we’d need creds to the email account as well. Before trying that, I want to see if there’s an easy way to emulate a trusted device.
  • There are many parameters a server could use to determine a valid session. It’s more than just cookies. Could be MAC, could be timing-related, could be much more. It’s a hidden decision by the service that you couldn’t figure out without brute forcing. The client can’t know.
  • Instead, I’ll try changing the email to mine and actually following the the 2FA programmatically.
  • That worked!
  • webdriverElement.click() can be covered by other elements, display off, etc. To force the submission, use webdriver.execute_script(‘arguments[0].click();’, element).
  • It looks like I only needed to confirm 2FA once. Even if I close python and the session, selenium is saving something somewhere that is deemed by the anda service as the same trusted device. Could be IP, who knows. This is great news! Could be different for other vendors though.
  • Successfully webscraped through 2FA.
  • Moved this logic from a script into an app. flask-table. Just a search bar and an updating table.
  • Demo marketplace, with only 1 vendor, is complete.
  • Used query params (?q=<>) instead of variable args in the route. Just request.args.get(). Very simple and clean.
  • Updated link action based on the value of another (input) element, with just HTML. Easy.
  • Github shows file details now: i.e. what percent python, what percent css, etc.
  • ctrl-alt-shift-r will start/stop screen recording on ubuntu. Amazing. It saves to .webm files, which chrome can open and play.
• Apr 29, 2019

  • The ribs yesterday ended up VERY spicy. The ghost pepper sauce (bhut jolokia) is 150k scoville heat units. I used a lot. Also made sweet potato pancakes properly today.
  • Game of thrones was good, but the resolution was too simple. No warging, no myths, no allegiance swapping, no surprises, bad battle tactics, no legendary explanations, no major deaths, etc. The night king isn’t apparently the primary antagonist. We’re gonna end politically with a human vs human war.
  • Python at Netflix: https://medium.com/netflix-techblog/python-at-netflix-bba45dae649e. Their CDN for all media is called Open Connect, but a lot of their service infrastructure is Python (load balancing, ML, automation, monitoring, alerting, infosec, marketing, metrics).
  • AWS for hosting, spinnaker for deployment, atlas for monitoring, genie for job execution, winston to pull it all together.
  • BLESS = their SSH certificate authority.
  • Jupyter
  • I’ve never really found a need for Jupyter notebooks, but they’re becoming more essential for data scientists.
  • Features
  • You can run code, visualize output, execute and tinker all the browser, and more.
  • You can basically created parametrized notebooks as templates to allow inputs and output configuration, across many languages (not just python!).
  • You can add notes, diagrams, and more surrounding the code.
  • There’s an API for data in and out of notebooks, for integration with actual services.
  • I installed it and played around (via pip into my sys python).
  • You can edit local files in the browser, run them, open a generic terminal. Useful stuff.
  • You can create a “notebook” which allows you to enter code and markdown together, as before (for reports, for datasharing, for notes, for more!). This saves as extension `.ipynb`.
  • Jasmine
  • Comparison to python unittesting
  • describe = class (like `unittest.TestCase`)
  • it = unittest (like `def test_x()` – these are also called “specs”)
  • expect = assert (there are a ton)
  • beforeEach beforeAll afterEach afterAll = setup setupClass teardown teardownClass
  • self = this (persists across tests)
  • spies = mocks
  • `jasmine server` allows you to iterate your tests in the browser. `jasmine ci` uses selenium to run for builds.
  • While jasmine focuses on being a test framework (for describing the tests, like unittest), karma is a very popular js testrunner (for execution of the tests, like nose).
  • Supercontest
  • Separated all python and js tests into distinct make recipes. The python suite still enters with tox, while eslint and jasmine run for js.
  • Globally installed nodejs and npm (although not using for anything right now).
  • Added the jasmine infra and a basic test. My js isn’t very complicated so I was intentionally light here. I could organize some of the event callbacks into separate functions to be tested, but it’s fine in scope for now.
  • Closed #20 after lots of testing! Pylint, pytests, eslint, jasmine.
  • Modified my app dockerfile to use joyzoursky/python-chromedriver directly instead of pulling the python base image and installing chrome/chromedriver/selenium in the dockerfile.
• Apr 28, 2019

  • A king cobra bite can kill an elephant!
  • Bought and prepped 2 spare rib racks yesterday. Smoked them today for game of thrones. Getting much better at trimming, especially with focus to both the tips and the st louis cut.
  • Shopped for motorcycles at a few shops with jcriss yesterday, he brought a grommmm.
  • WAF = web application firewall.
  • BitMitigate is a cool service that provides ddos protection, waf, cdn, and more.
  • To get the thumbnail from a youtube video, just enter the ID into: https://img.youtube.com/vi/<>/maxresdefault.jpg
• Apr 26, 2019

  • Supercontest.
  • Added pytest-cov. 45% overall. Not bad! Most of the core functionality is covered.
  • JS lint/tests.
  • Activate your python venv, then pip install nodeenv, then run nodeenv -p. This creates a virtualenv for node modules integrated with your current virtualenv for python modules. Gives you the `npm <>` command.
  • ./node_modules instead of ./venv. Has a bin and all the usual. This should be gitignored. The package-lock.json should not.
  • `npx` can be used instead of `npm` at the command line. This will fetch packages as needed into tmpdirs and run them.
  • Instead of coupling it with python/tox, you can use `nave` to create a pure node venv, and then add make targets to execute whatever test commands (instead of tox).
  • You can also run through npm itself, by defining a lint task in package.json (setup.py) and `npm run lint` (pip run lint). I’ve never like coupling the package manage with the admin commands. An external infra like make is better for this.
  • Ran npm init to create a skeleton package (required for style guide).
  • Ran eslint –init to create an eslintrc.json.
  • Ran a full eslint evaluation! 113 errors in my 4 simple files. Extending Google style guide. Common fixes I performed:
  • Single instead of double quotes.
  • Const instead of let for vars that are never reassigned.
  • No double space before inline comment, like Python.
  • Regular line indents are all 2 spaces. Indents for line continuation are 4 spaces.
  • No space between function and ().
  • Trailing common after the last item in sequences.
  • jasmine init, then added the spec yml to vcs.
  • Creed from The Office is performing at Saint Rocke on June 29: https://www.ticketweb.com/event/creed-bratton-saint-rocke-tickets/9166055?pl=saintrocke. It’s not comedy, it’s music: https://www.youtube.com/watch?v=Vt6kcF-PIsE.
  • HN. Some mild interest this week.
  • Removed the js blocks from vimrc. Didn’t like the folding.
  • JS speed is important, obviously, because of clientside implications. Hence the preference for 2 spaces instead of 4. JS typically isn’t compiled or compressed, so this small percentage makes a little difference. Also, with tons of callbacks, 4 spaces can visually get way too indented.
  • PharmaDB
  • Talked a little with Art about aggregating prices into a central marketplace, like Kayak for drugs.
  • Currently, technicians spend a lot of time shopping around for the lowest prices. Pharmacies (or their parents) have contracts with vendors which change the prices. These are reflected in the account, and work automatically with the creds to log in to the vendor site.
  • Some common vendors: AmerisourceBergen, Cardinal, HD Smith, Anda, Parmed.
  • Medicaid hosts a database with (I assume) average prices: https://data.medicaid.gov/Drug-Pricing-and-Payment/National-Pharmacy-Pricing-Database-xls/uima-szn8.
  • Gonna try to get a demo account (can only see prices, not order) to write an app that fetches all this information and aggregates it in a simple marketplace platform.
  • Sugar caramelizes at about 340F. If you’re doing low and slow ~200, it’s fine. I prefer a bit of brown sugar in the texas crutch. And don’t worry about bbq sauce on ribs if you’re cooking low. Do not use sugar/bbq sauce for hot and fast.
  • Prepared the rib recipe for sunday. Going to focus on the rib tips this time instead of the st louis cut (still doing full spare racks). That’s cartilage in the tips, not bone!
  • Lots of info came for lighting in a bottle. Skimmed most of it.
  • Pivotal, the company Rachel came from, makes RabbitMQ and Jasmine. They’re owned by Dell. I knew neither of these facts.
  • Csslint is also a thing.
  • Finished the suspiria remake (2018). I watch a lot of horror movies, and I loved how different this one is. That ENDING whattttt.
  • If you have chrome devtools open on a tab, you have a LOT of data. If you login to a site, the network tab shows you the form data of the request, which literally contains your plaintext password. This can be recorded (for that tab only). After that, it’s obviously stored in a cookie so that subsequent requests don’t contain the password. Still, given all that information, another client could copy and imitate that request.
  • Sharks and Warriors playoff games.
  • Remember, TDD = Test Driven Development and BDD = Behavior Driven Development.
• Apr 25, 2019

  • Avengers Endgame. I’ve never been a fan of comic book movies. This one had a ton of hype, almost Avatar level. It was average. There is a lot of time travelling and then a big fight and they knock some stones off his glove and win. Who could have predicted such a shocking, unlikely ending.
  • PW: some cool app ideas to practice skills: https://medium.freecodecamp.org/here-are-some-app-ideas-you-can-build-to-level-up-your-coding-skills-39618291f672.
  • Supercontest.
  • Unittests.
  • Had to `sudo systemctl restart postgresql` on my host before running the unittests. Remember, it requires an existing supercontest_unittests db. And remember, I do dev on this laptop as well, so postgresql doesn’t start on boot on my host (it would conflict with docker since I use default ports). Laptop must have died at some point.
  • Used spoof_lines and spoof_scores to populate the test db.
  • Had to install chromedriver on my host (unfortunate dep of the test now, I can fix later but it’s not urgent).
  • Added selenium capability to the tests so that I can actually check the data in the dom after js runs. I don’t do any click functionality to test registration or anything, just verification of the route APIs.
  • Changed to LiveServerTestCase from flask-testing so that I could use selenium. self.get_server_url is offered now. This slows down the tests a bit.
  • Loving selenium so far for unittests. Load a page, submit for data, click buttons, check results.
  • The webdriver API: https://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webdriver. find_element[s]_by_[id,xpath,css_selector,name,class_name,link_text].
  • Changed commit_picks to allow bypass of all checks (on the python app side).
  • Did some color mapping between names and rgba values (which is what selenium returns for the background-color css property).
  • Finished all the python tests that I wanted: user/line/score/pick commit, frontend login, matchup/pick/leaderboard view. Now on to eslint and jasmine for the js.
  • Docker
  • There’s an image on dockerhub with python/chromedriver/chrome/selenium: joyzourski/python-chromedriver. Switched to that instead of basing off python and installing the webdriver functionality manually in the dockerfile.
• Apr 24, 2019

  • FW/MD
  • Function declaration (definition) vs expression (anonymous). Obvious differences. Former allows a variable of global scope for callbacks, hoisting, etc. Latter less mem.
  • Change detection in a project usually means VCS changes for CI. Change detection in an app usually means frontend or backend. How does the view/controller respond when the model changes data?
  • In docker, `expose` in the dockerfile is just for documentation. `publish` (-p at the cli) is necessary to actually make the ports available. In docker-compose, `expose` in the yml is sufficient to do both.
  • Facebook internals. https://www.wired.com/story/facebook-mark-zuckerberg-15-months-of-fresh-hell/. Didn’t read the article because it’s way too long, but looked interesting.
  • The sharks/vgk game 7 last night was the best hockey game I’ve ever seen. The sharks came back from a 3-0 deficit by scoring 4 goals on a power play due to a 5m major. The knights tied it and pushed it to overtime, where the sharks scored to win. This sealed game 7 after coming back from a 3-1 game deficit in the series.
  • The Damian Lillard buzzer beater last night was also one of the best shots I’ve even seen. 37′ out, at 0 seconds, against Paul George, when tied, to end the series and knock OKC out of the playoffs.
  • Ordered new shoelaces. 54″ just for ref (medium high boots).
  • Jury duty summons can pull from registered voter pools or driver’s license pools (or both).
  • The “utm” you always see in ref links (like HN) is “Urchin Tracking Module”. Urchin is the predecessor to Google Analytics, and is simply a marketing tool. These 5 query params (utm_*) can be added to a URL to collect data: source, medium, content, campaign, term.
  • Did a couple typing tests. I’m about 90wpm with 95% accuracy.
• Apr 23, 2019

  • Reapplied thermal tape to the compressor on the ducati yesterday to protect it from the exhaust pipes.
  • Dodgeball championships last night, was great playing with the team. I liked this group (cowbell).
  • Went to Slab (same as Trudy’s) BBQ today. Brisket was better than Bludso’s, chicken was about equal, and pulled pork sucked. They also didn’t serve chicken quarters, only halves, which is dumb. Total was $33, which is absolutely ridiculous – that’s the same price as an entire 10lb packer of USDA prime. Very thankful I learned to smoke at home. It’s better quality at about 1/10th the price, for the sacrifice of a little effort!
  • Soundcloud requires you to pay for premium mode to download tracks for offline.