# MVM The Mini VM. The beginnings Picking up where i left off - I'm now writing the mark and sweep algorithm. Starting with the "mark" phase - I need to add another attribute into each object to mark it as "reachable". # KA Lite Dev Some stuff in KA Lite that I devved. # [CLOSED] ka-lite: 2408 -- add midyear exams https://github.com/learningequality/ka-lite/issues/2408 Hehey! This should be easy. Changes are in this commit: https://github.com/aronasorman/mlcsv/commit/8359ca3e80451b49f9215006a68bfcb632d75040 But hey! It's not working. It's not writing out the exam jsons. Rather than doing the normal `insert-debugger-line-here` method I normally do in python, I'm gonna try using `strace` to see what's wrong. I `strace`d the executable and it seems that it doesn't write the expected files at all. Which was totally what I expected :P looks I'm gonna have to use the normal debugging techniques again. Turns out the problem is two-fold: the Exam and playlist IDs in the new CSVs weren't in their right places, and we weren't processing lines with "Mid Year Exam" as an exam. Fixed those in [3c87d1c](https://github.com/aronasorman/mlcsv/commit/8359ca3e80451b49f9215006a68bfcb632d75040) # [CLOSED] ka-lite: 2385: -- Integrate PDF.js into KA Lite https://github.com/learningequality/ka-lite/issues/2385 OK! Goal for now is to jump off Richard's audio content html page, remove his stuff, add in PDF.js, then make it work. I've downloaded the "build" version of PDF.js which includes a viewer.html and some static assets. I'm looking into integrating that wholesale into KA Lite, then just ask Dylan/Jessica to integrate it. First step: just added `pdf.js` and `pdf.worker.js` into the static-libraries. I then added a "PDF" category into this switch statement: https://github.com/learningequality/ka-lite/pull/2399/files?diff=split#diff-df80504875a8ec892881c8b2b3d3d536R407 I don't have any code yet, but i'm stuck at trying to make the topic tree load a PDF. Now how would I do that? I'm looking at the JS stack trace to see how the whole thing loads. I just faked it by making a PDF viewer load when `kind` == "Video". Heh. Then just load a static pdf rather than what the actual node is. On to making the PDF viewer work! I'm considering just embedding the builtin pdf.js `viewer.html` inside an iframe for the prototype. Let's see how this goes. So simply embedding it in an iframe seems to be working well, except for the fact that it's too small right now. Next step is to find a way to resize it dynamically based on the window size. Bootstrap can hopefully help in that aspect. And oh reminder to myself: stick `PDFJS.disableFontFace = true` into the PDF.js initialization code so that it will work on mobile browsers. OK, tried it out with the test tablet from China. Loading is a bit slow with the PDF (about 5-7 seconds) but after that there seems to be no bugs in rendering. Yahoo! Ok, moving to PDF progress tracking now. I'm skimming through the `viewer.js` and other files bundled by Mozilla (which I stuck inside the iframe described above). And I saw this: https://github.com/mozilla/pdf.js/blob/master/web/viewer.js#L277 Looks helpful for tracking the user's progress on a PDF. I tried accessing `PDFVIew` at which `watchScroll` is defined, but it's not defined as it's named. It's probably defined somewhere as a variable. Will continue reading. For now it seems to much work to log every page they visited, so we're going with checking if they opened the PDF for X seconds. So to do any kind of logging at all I'm gonna have to find a way to access the javascript variables inside the iFrame. Researching that now. So to access the `window` object of an iframe, you use `this.contentWindow`. Nifty. Ha! So PDFJS has an event called `pagechange`: https://github.com/aronasorman/ka-lite/blob/pdfjs/static-libraries/pdfjs/web/viewer.js#L5798-L5799 There it is :D There's also the `DOMContentLoaded` event. Will investigate how this is used. So I made the skeleton frontend code that listens to the `pagechange` event and the `pageshow` event to track progress. Doesn't do any actual API calls yet: https://github.com/aronasorman/ka-lite/commit/0ab9f7b6548dbd312ce8a440258bf3542240bae3 I'm gonna work on tracking progress right now through the `ContentLog`. So to fetch a ContentLog we have to ensure that the user is logged in. I just copy pasted richard's code to mine: https://github.com/aronasorman/ka-lite/blob/pdfjs/kalite/distributed/static/js/distributed/audio/views.js#L15-L30 Now, this common code can be shared. I prefer a mixin, as recommended here: http://ricostacruz.com/backbone-patterns/#mixins I'll hold off on doing that though. I'll confer with Richard and Jamie on what they prefer. For now I just copy pasted that snippet. So some of the time an event's target is the iframe's document itself. One of the difficulties I was having was how to access the `window` object from the document. To do that, I just had to call `document.defaultView` and that would return the appropriate `window` object. Woohoo [content logging is done](https://github.com/aronasorman/ka-lite/blob/pdfjs/kalite/distributed/static/js/distributed/pdf/views.js#L71-L105)! Now on to setting points. Point tracking done: https://github.com/aronasorman/ka-lite/commit/1f647ffd8b7b37c6caf8345d67019a1141490926 Some gotchas: - have to update the `statusModel.points` attribute for points in the UI to update in realtime. - [have to edit the compute_total_points function to take the ContentLog into account](https://github.com/aronasorman/ka-lite/commit/d552ee48bc9ca1f52223cc6d00af0494cae56a2c) # [CLOSED] ka-lite: 2548 -- Add download link for content https://github.com/learningequality/ka-lite/issues/2548 Ok, the whole things has been refactored heavily by Richard. Have to read the contents of `ContentBaseView` and `BaseView`. T'was a simple affair. Just added a link to `ContentWrapperView`'s handlebars template: https://github.com/aronasorman/ka-lite/commit/74be3d333b0b03d56a0762e3e26c7acdc3b429b1 # ka-lite: 2338 https://github.com/learningequality/ka-lite/issues/2338 2014/06/09 - [ ] Write browser test - [ ] Make JS function that calls the proper backbone view - [ ] Make button in test itself that "clears" a test The task is to have a "clear test" UI for teachers so they can retake it. Backend wise, seems like the easiest way is to just delete the test log, then make an endpoint for it, and finally a button. Lots of refactoring work for the StudentTestTest. Need to update it to use mixins. As well as update it to use the new Test.total_questions attribute. Now apparently some old call of `self.create_facility` is failing. Due to changed forms again? Specifically, this line: `https://github.com/aronasorman/ka-lite/blob/nalanda-rct3/kalite/distributed/tests/browser_tests/base.py` With this error: ``` Traceback (most recent call last): File "/home/aron/src/ka-lite/kalite/distributed/tests/browser_tests/student_testing.py", line 44, in setUp super(StudentTestTest, self).setUp() File "/home/aron/src/ka-lite/kalite/distributed/tests/browser_tests/base.py", line 207, in setUp self.facility = self.create_facility(facility_name=self.facility_name) File "/home/aron/src/ka-lite/kalite/testing/mixins/facility_mixins.py", line 17, in create_facility obj, created = Facility.objects.get_or_create(**fields) File "/home/aron/src/ka-lite/python-packages/django/db/models/manager.py", line 146, in get_or_create return self.get_query_set().get_or_create(**kwargs) File "/home/aron/src/ka-lite/python-packages/django/db/models/query.py", line 470, in get_or_create return self.get(**lookup), False File "/home/aron/src/ka-lite/python-packages/django/db/models/query.py", line 379, in get clone = self.filter(*args, **kwargs) File "/home/aron/src/ka-lite/python-packages/django/db/models/query.py", line 655, in filter return self._filter_or_exclude(False, *args, **kwargs) File "/home/aron/src/ka-lite/python-packages/django/db/models/query.py", line 673, in _filter_or_exclude clone.query.add_q(Q(*args, **kwargs)) File "/home/aron/src/ka-lite/python-packages/django/db/models/sql/query.py", line 1266, in add_q can_reuse=used_aliases, force_having=force_having) File "/home/aron/src/ka-lite/python-packages/django/db/models/sql/query.py", line 1134, in add_filter process_extras=process_extras) File "/home/aron/src/ka-lite/python-packages/django/db/models/sql/query.py", line 1332, in setup_joins "Choices are: %s" % (name, ", ".join(names))) FieldError: Cannot resolve keyword 'facility_name' into field. Choices are: address, address_normalized, contact_email, contact_name, contact_phone, counter, deleted, description, facilitygroup, facilityuser, id, latitude, longitude, name, signature, signed_by, signed_version, user_count, zone_fallback, zoom ``` Just removed that line. Heh. Ok, I seem to be getting an unrelated Selenium error, with Firefox crashing with this message: ``` Traceback (most recent call last): File "/home/aron/src/ka-lite/kalite/testing/browser.py", line 174, in setUp (self.browser, self.admin_user, self.admin_pass) = setup_test_env(browser_type=browser_type) File "/home/aron/src/ka-lite/kalite/testing/browser.py", line 38, in setup_test_env local_browser = getattr(webdriver, browser_type)() # Get local session of browser File "/home/aron/src/ka-lite/python-packages/selenium/webdriver/firefox/webdriver.py", line 59, in __init__ self.binary, timeout), File "/home/aron/src/ka-lite/python-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__ self.binary.launch_browser(self.profile) File "/home/aron/src/ka-lite/python-packages/selenium/webdriver/firefox/firefox_binary.py", line 61, in launch_browser self._wait_until_connectable() File "/home/aron/src/ka-lite/python-packages/selenium/webdriver/firefox/firefox_binary.py", line 105, in _wait_until_connectable self.profile.path, self._get_firefox_output())) WebDriverException: Message: 'Can\'t load the profile. Profile Dir: /tmp/tmpbXa7eJ Firefox output: \n(process:27038): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\n1410068812152\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/addons/XPIProvider.jsm: ["XPIProvider"]\n1410068812154\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/LightweightThemeManager.jsm: ["LightweightThemeManager"]\n1410068812156\taddons.xpi\tDEBUG\tstartup\n1410068812157\taddons.xpi\tINFO\tMapping online-accounts@lists.launchpad.net to /usr/lib/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/online-accounts@lists.launchpad.net\n1410068812158\taddons.xpi\tINFO\tMapping {2e1445b0-2682-11e1-bfc2-0800200c9a66} to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/{2e1445b0-2682-11e1-bfc2-0800200c9a66}\n1410068812158\taddons.xpi\tINFO\tMapping webapps-team@lists.launchpad.net to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/webapps-team@lists.launchpad.net\n1410068812158\taddons.xpi\tINFO\tMapping ubufox@ubuntu.com to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/ubufox@ubuntu.com\n1410068812158\taddons.xpi\tINFO\tMapping {972ce4c6-7e08-4474-a285-3208198ce6fd} to /usr/lib/firefox/browser/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}\n1410068812158\taddons.xpi\tINFO\tMapping langpack-en-ZA@firefox.mozilla.org to /usr/lib/firefox/browser/extensions/langpack-en-ZA@firefox.mozilla.org.xpi\n1410068812159\taddons.xpi\tINFO\tMapping langpack-en-GB@firefox.mozilla.org to /usr/lib/firefox/browser/extensions/langpack-en-GB@firefox.mozilla.org.xpi\n1410068812159\taddons.xpi\tINFO\tMapping fxdriver@googlecode.com to /tmp/tmpbXa7eJ/extensions/fxdriver@googlecode.com\n1410068812160\taddons.xpi\tDEBUG\tcheckForChanges\n1410068812171\taddons.xpi\tDEBUG\tNo changes found\n1410068812176\taddons.xpi\tDEBUG\tRegistering manifest for /usr/lib/firefox/browser/extensions/langpack-en-ZA@firefox.mozilla.org.xpi\n1410068812177\taddons.xpi\tDEBUG\tRegistering manifest for /usr/lib/firefox/browser/extensions/langpack-en-GB@firefox.mozilla.org.xpi\n*** Blocklist::_preloadBlocklistFile: blocklist is disabled\n' ``` Ugh. What will I do to make this test run? Debugging that firefox error later. Dropped back to phantomjs. Getting this error now: ``` ERROR: test_exercise_mastery (kalite.distributed.tests.browser_tests.student_testing.StudentTestTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/aron/src/ka-lite/kalite/distributed/tests/browser_tests/student_testing.py", line 46, in setUp self.fac = self.create_facility(self.facility_name) TypeError: create_facility() takes exactly 1 argument (2 given) ``` Seems like we're still getting the old "create_facility" defined in FacilityTestCase. How to skip that? Turns out `create_facility` only accepts KEYWORD arguments, and we're passing the facility as a POSITIONAL argument. The correct line should be: ```python self.fac = self.create_facility(name=self.facility_name) ``` Now we're getting a timeout: ``` ====================================================================== ERROR: test_exercise_mastery (kalite.distributed.tests.browser_tests.student_testing.StudentTestTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/aron/src/ka-lite/kalite/distributed/tests/browser_tests/student_testing.py", line 85, in test_exercise_mastery self.browser_submit_answers(answer) File "/home/aron/src/ka-lite/kalite/distributed/tests/browser_tests/student_testing.py", line 67, in browser_submit_answers (By.CSS_SELECTOR, "#solutionarea > input[type=text]"))) File "/home/aron/src/ka-lite/python-packages/selenium/webdriver/support/wait.py", line 71, in until raise TimeoutException(message) TimeoutException: Message: '' ``` It's failing with some UI exception, but I can't see the error because I'm running phantomjs. Time to make Firefox work. as per this stack overflow answer: http://stackoverflow.com/a/25645344 I had to downgrade my version of Firefox to version 30 to make everything work again. Ran the `test_exercise_mastery` test again. Success. Ran all tests. Failure, with half of them coming from the deletion of the default facility for KALiteDistributedBrowserTestCase. Fixing. # [CLOSED] ka-lite: 2455 -- Extend search functionality to include other content https://github.com/learningequality/ka-lite/issues/2455 Ok, first off develop doesn't have a search bar anymore. So I added it back with this code: ```html {# Search box #}
    • ``` Second, I'm gonna have to make the JS code in `search_autocomplete.js` at least find that search box so I can see what endpoint is being called. NVM. Turns out it's already detecting it once I put a breakpoint on `fetchTopicTree`. Now I gotta extend that API endpoint to return other content types as well. Querying the api directly on the browser (with `http://127.0.0.1:8008/api/flat_topic_tree/en`) resulted in this JSON: ``` { Topic: { demo: { available: true, path: "demo/", kind: "Topic", title: "demo" }, classrooms: { available: true, path: "demo/classrooms/", kind: "Topic", title: "classrooms" }, physics: { available: true, path: "demo/science/physics/", kind: "Topic", title: "physics" }, math: { available: true, path: "demo/math/", kind: "Topic", title: "math" }, science: { available: true, path: "demo/science/", kind: "Topic", title: "science" } }, Video: { 3db321ccaec3154139dd42ded78f19b5: { available: true, path: "demo/classrooms/healing-classrooms-1", kind: "Video", title: "Healing Classrooms 1" } }, Exercise: { } } ``` So it's mostly working already, I just need to extend it to add the other content types. Whoo! Thought I need to do more hackzor. Alright so I actually had to edit `generate_node_cache` to add in other types, like so: https://github.com/aronasorman/ka-lite/blob/search/kalite/topic_tools/__init__.py#L259-L260 I also had to edit `get_content_cache` to be able to filter the content: https://github.com/aronasorman/ka-lite/blob/search/kalite/topic_tools/__init__.py#L113-L126 So now the API endpoint returns the content kinds we explicitly support! Nice! So I really didn't do much, and the search autocomplete is now working already. Problems I saw: - No icons for the new content types - The separate search page isn't styled correctly - Pressing enter for text like "math" leads to a 404. Passed off to Richard. # ka-lite: 2238 -- update crowdin management command https://github.com/learningequality/ka-lite/issues/2238 So first step is to make a skeleton management command. I used `NoArgsCommand` since I don't expect any argument to this. Proceeding to make a test for this. The basic aim of the test is to see if we're making a request to the right url. So it will involve the `mock` library. Mocking requests might be easier with [httreplay](https://github.com/davepeck/httreplay). Looking into it to see if it's easy to work with. I saw this cool library called [vcr.py](https://github.com/kevin1024/vcrpy). Kinda like `vcr` from the Rails community. I'm thinking of switching to this from manually constructing request mock objects as it will be easier to keep up to date with the remote server. I'm thinking of making a wrapper for `vcr.py` so that testers don't have to fudge with the cassette names. They just call a `with` statement (or a decorator) and boom! The wrapper will be the new KALiteTestCase. Lelz. I had KALiteTestCase inherit from `CreateDeviceMixin` and `TestCase`. That caused a couple of MRO resolution errors, which I promptly fixed. From `KALiteTestCase` I made a `KALiteClientTestCase` which is for Django client tests, and a `KALiteBrowserTestCase` which is for browser tests. I'm encountering some test failures now with `student_testing.CoreTests`, which I modified to inherit from `KALiteClientTestCase`. The errors look something like: ``` Traceback (most recent call last): File "/home/aron/src/ka-lite/kalite/student_testing/tests.py", line 81, in setUp super(CoreTests, self).setUp() File "/home/aron/src/ka-lite/kalite/student_testing/tests.py", line 38, in setUp self.assertTrue(self.client.facility) AssertionError: None is not True ``` Fixed. Just had `CoreTests` and `CurrentUnitTests` inherit from `KALiteClientTestCase`. Now fixing more test breakage caused by MRO errors (a test case inheriting from both `CreateDeviceMixin` and `KALiteTestCase`, which itself inherits `CreateDeviceMixin`). Now getting this error when running all the tests: ``` Traceback (most recent call last): File "/home/aron/src/ka-lite/python-packages/django/core/management/base.py", line 222, in run_from_argv self.execute(*args, **options.__dict__) File "/home/aron/src/ka-lite/python-packages/django/core/management/commands/test.py", line 80, in execute super(Command, self).execute(*args, **options) File "/home/aron/src/ka-lite/python-packages/django/core/management/base.py", line 255, in execute output = self.handle(*args, **options) File "/home/aron/src/ka-lite/python-packages/south/management/commands/test.py", line 8, in handle super(Command, self).handle(*args, **kwargs) File "/home/aron/src/ka-lite/python-packages/django/core/management/commands/test.py", line 106, in handle failures = test_runner.run_tests(test_labels) File "/home/aron/src/ka-lite/kalite/testing/testrunner.py", line 68, in run_tests return run_tests_wrapper_fn() File "/home/aron/src/ka-lite/python-packages/django/test/utils.py", line 220, in inner return test_func(*args, **kwargs) File "/home/aron/src/ka-lite/kalite/testing/testrunner.py", line 67, in run_tests_wrapper_fn return super(KALiteTestRunner,self).run_tests(test_labels, extra_tests, **kwargs) File "/home/aron/src/ka-lite/python-packages/django/test/simple.py", line 366, in run_tests suite = self.build_suite(test_labels, extra_tests) File "/home/aron/src/ka-lite/kalite/testing/testrunner.py", line 79, in build_suite testfun = getattr(test, test._testMethodName) AttributeError: 'TestSuite' object has no attribute '_testMethodName' ``` Investigating. Only happens on the code called when `--failfast` is on. Something related to these lines, found in: ```python if self.failfast: for test in test_suite._tests: testfun = getattr(test, test._testMethodName) setattr(test, test._testMethodName, auto_pdb()(testfun)) return test_suite ``` .... I got into this refactoring blackhole. Coming back out with refactored tests! While rewriting the tests, I decided it would be nice to have a multiple cursor feature like sublime text. So I just downloaded the `multiple-cursors` library and integrated it into Emacs. Back! Turns out [someone already wrote the function](https://github.com/fle-internal/ka-lite-central/blob/7afcc6f27977b906251d63c2bbb3e48c937af4cd/centralserver/i18n/management/commands/update_pot.py#L121). Just gotta clean it up and extend it to namespace the po files by version. So the main problem I'm stuck with right now is calling `add-file` vs. `update-file` crowdin api call. They literally do only what they say. Guess I can't avoid some duplication of code. Along the way [I refactored the `version.py`](https://github.com/learningequality/ka-lite/commit/db4975a8ea00f5e2d292e7defbc093370e2be722) file where the compontents of the version are broken down into their own variables, and VERSION just reconstructing itself based on that. Urgh I somehow can't form the right API call with the `requests` library. Here's the function: ```python def upload_to_crowdin(files, project_key, project_id="ka-lite"): api_call = "add-file" url = "https://api.crowdin.com/api/project/{project_id}/{api_call}" url = url.format(project_id=project_id, api_call=api_call) version_namespace = "%s.%s" % (version.MAJOR_VERSION, version.MINOR_VERSION) pot_path = pathlib.Path(POT_DIRPATH) files_to_upload = {"files[/KA Lite UI/%s-django.po]" % version_namespace: open((pot_path / "django.pot").resolve().__str__()), "files[/KA Lite UI/%s-djangojs.po]" % version_namespace: open((pot_path / "djangojs.pot").resolve().__str__())} get_params = {"key": project_key} r = requests.post(url, params=get_params, data=files_to_upload) ``` And CrowdIn is throwing me this error (as the body of `r`): ``` '\n\n 4\n No files specified in request\n\n' ``` Finally got it working! You have to add in the file array syntax **and** put it in the files parameter for `requests.post`: ```python files_to_upload = {"files[/versioned/%s-django.po]" % version_namespace: open((pot_path / "django.pot").resolve().__str__()), "files[/versioned/%s-djangojs.po]" % version_namespace: open((pot_path / "djangojs.pot").resolve().__str__())} get_params = {"key": project_key} r = requests.post(url, params=get_params, files=files_to_upload) ``` # ka-lite 2566 -- Track downloaded content https://github.com/learningequality/ka-lite/issues/2566 Ok so the problem is that when a user clicks "Download" for a content we have no idea how long they've viewed it. We want to track that. Jamie suggested looking into the focus/blur JS events.