• Apr 30, 2019

  • MD. Articles on ML and React component testing.
  • Jasmine.
  • Ahhhh, played with jasmine a bit more. “$ is not defined” and such is because I was using jasmine-node. jQuery requires the browser’s DOM api, which node does not have. You have to use the html runner for jasmine (so you can add it as a usual script, from a cdn or otherwise).
  • Here is a library that allows you to load html/css/other fixtures into the jasmine context: https://www.npmjs.com/package/jasmine-jquery.
  • You could still just run tests with a headless browser to exercise the frontend, which I’m already doing. It’s just a bit easier to unittest js functions directly with jasmine.
  • Opting to not pursue this further for now, coverage is already sufficient.
  • Installed pkg-config zlib1g-dev openjdk-8-jdk bazel
  • ipython was busted because of an incompatible version of prompt-toolkit (probably from the jupyter install yesterday), so I reinstalled that package.
  • Remember at the ipython prompt you can type ! before a command to execute something in the system shell (useful for pip installations and such).
  • Lightning in a bottle schedule was released. Didn’t really look at it much, just a quick skim.
  • Webscraping.
  1. (preferable) Use an exposed API, with structure that is designed for programmatic requests of data.
  2. HTTP with form assistance. Use a library like mechanize or robobrowser to handle all the browser defaults for the form fields you don’t care about (everything except un/pw). This handles session.
  3. Pure HTTP. Use devtools to watch the network activity of your manual request. Copy all the form data. Use a requests session to persist returned cookies and other things across subsequent requests.
  4. Selenium. Simulate the frontend and perform your usual data entry / clicking.
  • Pharma.
  • Logged into new.andanet.com and searched for some drug prices, then tried to fetch the data programmatically and serve it in an aggregate marketplace across a few vendors.
  • Mechanize doesn’t support py3 and robobrowser is newer, so I used that. Both robobrowser and pure requests would not return the login form. “Your current web browser session has been closed due to inactivity. Please login to create a new session.” This is likely because neither uses an actual browser, which is an interesting deterrent to programmatic interaction. Let me try with Selenium next.
  • Selenium was able to successfully log in. The next problem is the 2FA. It emails a verification code. I could automate this with Selenium as well, fetching the code from email and clicking the second submit, but then we’d need creds to the email account as well. Before trying that, I want to see if there’s an easy way to emulate a trusted device.
  • There are many parameters a server could use to determine a valid session. It’s more than just cookies. Could be MAC, could be timing-related, could be much more. It’s a hidden decision by the service that you couldn’t figure out without brute forcing. The client can’t know.
  • Instead, I’ll try changing the email to mine and actually following the the 2FA programmatically.
  • That worked!
  • webdriverElement.click() can be covered by other elements, display off, etc. To force the submission, use webdriver.execute_script(‘arguments[0].click();’, element).
  • It looks like I only needed to confirm 2FA once. Even if I close python and the session, selenium is saving something somewhere that is deemed by the anda service as the same trusted device. Could be IP, who knows. This is great news! Could be different for other vendors though.
  • Successfully webscraped through 2FA.
  • Moved this logic from a script into an app. flask-table. Just a search bar and an updating table.
  • Demo marketplace, with only 1 vendor, is complete.
  • Used query params (?q=<>) instead of variable args in the route. Just request.args.get(). Very simple and clean.
  • Updated link action based on the value of another (input) element, with just HTML. Easy.
  • Github shows file details now: i.e. what percent python, what percent css, etc.
  • ctrl-alt-shift-r will start/stop screen recording on ubuntu. Amazing. It saves to .webm files, which chrome can open and play.