The Mini VM.
The beginnings
Picking up where i left off - I’m now writing the mark and sweep algorithm.
Starting with the “mark” phase - I need to add another attribute into each object to mark it as “reachable”.
Some stuff in KA Lite that I devved.
Hehey! This should be easy. Changes are in this commit: https://github.com/aronasorman/mlcsv/commit/8359ca3e80451b49f9215006a68bfcb632d75040
But hey! It’s not working. It’s not writing out the exam jsons. Rather than doing the normal insert-debugger-line-here method I normally do in python, I’m gonna try using strace to see what’s wrong.
I straced the executable and it seems that it doesn’t write the expected files at all. Which was totally what I expected :P looks I’m gonna have to use the normal debugging techniques again.
Turns out the problem is two-fold: the Exam and playlist IDs in the new CSVs weren’t in their right places, and we weren’t processing lines with “Mid Year Exam” as an exam. Fixed those in 3c87d1c
OK! Goal for now is to jump off Richard’s audio content html page, remove his stuff, add in PDF.js, then make it work.
I’ve downloaded the “build” version of PDF.js which includes a viewer.html and some static assets. I’m looking into integrating that wholesale into KA Lite, then just ask Dylan/Jessica to integrate it.
First step: just added pdf.js and pdf.worker.js into the static-libraries.
I then added a “PDF” category into this switch statement: https://github.com/learningequality/ka-lite/pull/2399/files?diff=split#diff-df80504875a8ec892881c8b2b3d3d536R407
I don’t have any code yet, but i’m stuck at trying to make the topic tree load a PDF. Now how would I do that? I’m looking at the JS stack trace to see how the whole thing loads.
I just faked it by making a PDF viewer load when kind == “Video”. Heh. Then just load a static pdf rather than what the actual node is. On to making the PDF viewer work!
I’m considering just embedding the builtin pdf.js viewer.html inside an iframe for the prototype. Let’s see how this goes.
So simply embedding it in an iframe seems to be working well, except for the fact that it’s too small right now. Next step is to find a way to resize it dynamically based on the window size. Bootstrap can hopefully help in that aspect.
And oh reminder to myself: stick PDFJS.disableFontFace = true into the PDF.js initialization code so that it will work on mobile browsers.
OK, tried it out with the test tablet from China. Loading is a bit slow with the PDF (about 5-7 seconds) but after that there seems to be no bugs in rendering. Yahoo!
Ok, moving to PDF progress tracking now. I’m skimming through the viewer.js and other files bundled by Mozilla (which I stuck inside the iframe described above). And I saw this: https://github.com/mozilla/pdf.js/blob/master/web/viewer.js#L277
Looks helpful for tracking the user’s progress on a PDF.
I tried accessing PDFVIew at which watchScroll is defined, but it’s not defined as it’s named. It’s probably defined somewhere as a variable. Will continue reading.
For now it seems to much work to log every page they visited, so we’re going with checking if they opened the PDF for X seconds.
So to do any kind of logging at all I’m gonna have to find a way to access the javascript variables inside the iFrame. Researching that now.
So to access the window object of an iframe, you use this.contentWindow. Nifty.
Ha! So PDFJS has an event called pagechange: https://github.com/aronasorman/ka-lite/blob/pdfjs/static-libraries/pdfjs/web/viewer.js#L5798-L5799
There it is :D
There’s also the DOMContentLoaded event. Will investigate how this is used.
So I made the skeleton frontend code that listens to the pagechange event and the pageshow event to track progress. Doesn’t do any actual API calls yet: https://github.com/aronasorman/ka-lite/commit/0ab9f7b6548dbd312ce8a440258bf3542240bae3
I’m gonna work on tracking progress right now through the ContentLog.
So to fetch a ContentLog we have to ensure that the user is logged in. I just copy pasted richard’s code to mine: https://github.com/aronasorman/ka-lite/blob/pdfjs/kalite/distributed/static/js/distributed/audio/views.js#L15-L30
Now, this common code can be shared. I prefer a mixin, as recommended here: http://ricostacruz.com/backbone-patterns/#mixins
I’ll hold off on doing that though. I’ll confer with Richard and Jamie on what they prefer. For now I just copy pasted that snippet.
So some of the time an event’s target is the iframe’s document itself. One of the difficulties I was having was how to access the window object from the document. To do that, I just had to call document.defaultView and that would return the appropriate window object.
Woohoo content logging is done! Now on to setting points.
Point tracking done: https://github.com/aronasorman/ka-lite/commit/1f647ffd8b7b37c6caf8345d67019a1141490926
Some gotchas:
statusModel.points attribute for points in the UI to update in realtime.Ok, the whole things has been refactored heavily by Richard. Have to read the contents of ContentBaseView and BaseView.
T’was a simple affair. Just added a link to ContentWrapperView‘s handlebars template: https://github.com/aronasorman/ka-lite/commit/74be3d333b0b03d56a0762e3e26c7acdc3b429b1
The task is to have a “clear test” UI for teachers so they can retake it.
Backend wise, seems like the easiest way is to just delete the test log, then make an endpoint for it, and finally a button.
Lots of refactoring work for the StudentTestTest. Need to update it to use mixins.
As well as update it to use the new Test.total_questions attribute.
Now apparently some old call of self.create_facility is failing. Due to changed forms again?
Specifically, this line: https://github.com/aronasorman/ka-lite/blob/nalanda-rct3/kalite/distributed/tests/browser_tests/base.py
With this error:
Traceback (most recent call last):
File "/home/aron/src/ka-lite/kalite/distributed/tests/browser_tests/student_testing.py", line 44, in setUp
super(StudentTestTest, self).setUp()
File "/home/aron/src/ka-lite/kalite/distributed/tests/browser_tests/base.py", line 207, in setUp
self.facility = self.create_facility(facility_name=self.facility_name)
File "/home/aron/src/ka-lite/kalite/testing/mixins/facility_mixins.py", line 17, in create_facility
obj, created = Facility.objects.get_or_create(**fields)
File "/home/aron/src/ka-lite/python-packages/django/db/models/manager.py", line 146, in get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "/home/aron/src/ka-lite/python-packages/django/db/models/query.py", line 470, in get_or_create
return self.get(**lookup), False
File "/home/aron/src/ka-lite/python-packages/django/db/models/query.py", line 379, in get
clone = self.filter(*args, **kwargs)
File "/home/aron/src/ka-lite/python-packages/django/db/models/query.py", line 655, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/home/aron/src/ka-lite/python-packages/django/db/models/query.py", line 673, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/home/aron/src/ka-lite/python-packages/django/db/models/sql/query.py", line 1266, in add_q
can_reuse=used_aliases, force_having=force_having)
File "/home/aron/src/ka-lite/python-packages/django/db/models/sql/query.py", line 1134, in add_filter
process_extras=process_extras)
File "/home/aron/src/ka-lite/python-packages/django/db/models/sql/query.py", line 1332, in setup_joins
"Choices are: %s" % (name, ", ".join(names)))
FieldError: Cannot resolve keyword 'facility_name' into field. Choices are: address, address_normalized, contact_email, contact_name, contact_phone, counter, deleted, description, facilitygroup, facilityuser, id, latitude, longitude, name, signature, signed_by, signed_version, user_count, zone_fallback, zoomJust removed that line. Heh.
Ok, I seem to be getting an unrelated Selenium error, with Firefox crashing with this message:
Traceback (most recent call last):
File "/home/aron/src/ka-lite/kalite/testing/browser.py", line 174, in setUp
(self.browser, self.admin_user, self.admin_pass) = setup_test_env(browser_type=browser_type)
File "/home/aron/src/ka-lite/kalite/testing/browser.py", line 38, in setup_test_env
local_browser = getattr(webdriver, browser_type)() # Get local session of browser
File "/home/aron/src/ka-lite/python-packages/selenium/webdriver/firefox/webdriver.py", line 59, in __init__
self.binary, timeout),
File "/home/aron/src/ka-lite/python-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
self.binary.launch_browser(self.profile)
File "/home/aron/src/ka-lite/python-packages/selenium/webdriver/firefox/firefox_binary.py", line 61, in launch_browser
self._wait_until_connectable()
File "/home/aron/src/ka-lite/python-packages/selenium/webdriver/firefox/firefox_binary.py", line 105, in _wait_until_connectable
self.profile.path, self._get_firefox_output()))
WebDriverException: Message: 'Can\'t load the profile. Profile Dir: /tmp/tmpbXa7eJ Firefox output: \n(process:27038): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\n1410068812152\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/addons/XPIProvider.jsm: ["XPIProvider"]\n1410068812154\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/LightweightThemeManager.jsm: ["LightweightThemeManager"]\n1410068812156\taddons.xpi\tDEBUG\tstartup\n1410068812157\taddons.xpi\tINFO\tMapping online-accounts@lists.launchpad.net to /usr/lib/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/online-accounts@lists.launchpad.net\n1410068812158\taddons.xpi\tINFO\tMapping {2e1445b0-2682-11e1-bfc2-0800200c9a66} to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/{2e1445b0-2682-11e1-bfc2-0800200c9a66}\n1410068812158\taddons.xpi\tINFO\tMapping webapps-team@lists.launchpad.net to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/webapps-team@lists.launchpad.net\n1410068812158\taddons.xpi\tINFO\tMapping ubufox@ubuntu.com to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/ubufox@ubuntu.com\n1410068812158\taddons.xpi\tINFO\tMapping {972ce4c6-7e08-4474-a285-3208198ce6fd} to /usr/lib/firefox/browser/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}\n1410068812158\taddons.xpi\tINFO\tMapping langpack-en-ZA@firefox.mozilla.org to /usr/lib/firefox/browser/extensions/langpack-en-ZA@firefox.mozilla.org.xpi\n1410068812159\taddons.xpi\tINFO\tMapping langpack-en-GB@firefox.mozilla.org to /usr/lib/firefox/browser/extensions/langpack-en-GB@firefox.mozilla.org.xpi\n1410068812159\taddons.xpi\tINFO\tMapping fxdriver@googlecode.com to /tmp/tmpbXa7eJ/extensions/fxdriver@googlecode.com\n1410068812160\taddons.xpi\tDEBUG\tcheckForChanges\n1410068812171\taddons.xpi\tDEBUG\tNo changes found\n1410068812176\taddons.xpi\tDEBUG\tRegistering manifest for /usr/lib/firefox/browser/extensions/langpack-en-ZA@firefox.mozilla.org.xpi\n1410068812177\taddons.xpi\tDEBUG\tRegistering manifest for /usr/lib/firefox/browser/extensions/langpack-en-GB@firefox.mozilla.org.xpi\n*** Blocklist::_preloadBlocklistFile: blocklist is disabled\n'Ugh. What will I do to make this test run?
Debugging that firefox error later. Dropped back to phantomjs. Getting this error now:
ERROR: test_exercise_mastery (kalite.distributed.tests.browser_tests.student_testing.StudentTestTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/aron/src/ka-lite/kalite/distributed/tests/browser_tests/student_testing.py", line 46, in setUp
self.fac = self.create_facility(self.facility_name)
TypeError: create_facility() takes exactly 1 argument (2 given)Seems like we’re still getting the old “create_facility” defined in FacilityTestCase. How to skip that?
Turns out create_facility only accepts KEYWORD arguments, and we’re passing the facility as a POSITIONAL argument. The correct line should be:
self.fac = self.create_facility(name=self.facility_name)Now we’re getting a timeout:
======================================================================
ERROR: test_exercise_mastery (kalite.distributed.tests.browser_tests.student_testing.StudentTestTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/aron/src/ka-lite/kalite/distributed/tests/browser_tests/student_testing.py", line 85, in test_exercise_mastery
self.browser_submit_answers(answer)
File "/home/aron/src/ka-lite/kalite/distributed/tests/browser_tests/student_testing.py", line 67, in browser_submit_answers
(By.CSS_SELECTOR, "#solutionarea > input[type=text]")))
File "/home/aron/src/ka-lite/python-packages/selenium/webdriver/support/wait.py", line 71, in until
raise TimeoutException(message)
TimeoutException: Message: ''It’s failing with some UI exception, but I can’t see the error because I’m running phantomjs. Time to make Firefox work.
as per this stack overflow answer: http://stackoverflow.com/a/25645344
I had to downgrade my version of Firefox to version 30 to make everything work again.
Ran the test_exercise_mastery test again. Success.
Ran all tests. Failure, with half of them coming from the deletion of the default facility for KALiteDistributedBrowserTestCase. Fixing.
Ok, first off develop doesn’t have a search bar anymore. So I added it back with this code:
{# Search box #}
<li>
<form class="navbar-form" action="{% url 'search' %}" method="get" id="search-box" role="search">
{% comment %}translators: this will appear in the navigation bar. please try to keep it as short as possible.{% endcomment %}
<div class="input-group">
<input type="text" name="query" id="search" class="form-control" placeholder="{% trans 'video, topic, or exercise...' %}" />
<button class="btn btn-default" type="submit"><i class="glyphicon glyphicon-search"></i></button>
</div>
</form>
</li>Second, I’m gonna have to make the JS code in search_autocomplete.js at least find that search box so I can see what endpoint is being called.
NVM. Turns out it’s already detecting it once I put a breakpoint on fetchTopicTree. Now I gotta extend that API endpoint to return other content types as well.
Querying the api directly on the browser (with http://127.0.0.1:8008/api/flat_topic_tree/en) resulted in this JSON:
{
Topic: {
demo: {
available: true,
path: "demo/",
kind: "Topic",
title: "demo"
},
classrooms: {
available: true,
path: "demo/classrooms/",
kind: "Topic",
title: "classrooms"
},
physics: {
available: true,
path: "demo/science/physics/",
kind: "Topic",
title: "physics"
},
math: {
available: true,
path: "demo/math/",
kind: "Topic",
title: "math"
},
science: {
available: true,
path: "demo/science/",
kind: "Topic",
title: "science"
}
},
Video: {
3db321ccaec3154139dd42ded78f19b5: {
available: true,
path: "demo/classrooms/healing-classrooms-1",
kind: "Video",
title: "Healing Classrooms 1"
}
},
Exercise: { }
}So it’s mostly working already, I just need to extend it to add the other content types. Whoo! Thought I need to do more hackzor.
Alright so I actually had to edit generate_node_cache to add in other types, like so: https://github.com/aronasorman/ka-lite/blob/search/kalite/topic_tools/__init__.py#L259-L260
I also had to edit get_content_cache to be able to filter the content: https://github.com/aronasorman/ka-lite/blob/search/kalite/topic_tools/__init__.py#L113-L126
So now the API endpoint returns the content kinds we explicitly support! Nice!
So I really didn’t do much, and the search autocomplete is now working already. Problems I saw:
Passed off to Richard.
So first step is to make a skeleton management command. I used NoArgsCommand since I don’t expect any argument to this.
Proceeding to make a test for this. The basic aim of the test is to see if we’re making a request to the right url. So it will involve the mock library.
Mocking requests might be easier with httreplay. Looking into it to see if it’s easy to work with.
I saw this cool library called vcr.py. Kinda like vcr from the Rails community. I’m thinking of switching to this from manually constructing request mock objects as it will be easier to keep up to date with the remote server.
I’m thinking of making a wrapper for vcr.py so that testers don’t have to fudge with the cassette names. They just call a with statement (or a decorator) and boom!
The wrapper will be the new KALiteTestCase. Lelz.
I had KALiteTestCase inherit from CreateDeviceMixin and TestCase. That caused a couple of MRO resolution errors, which I promptly fixed. From KALiteTestCase I made a KALiteClientTestCase which is for Django client tests, and a KALiteBrowserTestCase which is for browser tests.
I’m encountering some test failures now with student_testing.CoreTests, which I modified to inherit from KALiteClientTestCase. The errors look something like:
Traceback (most recent call last):
File "/home/aron/src/ka-lite/kalite/student_testing/tests.py", line 81, in setUp
super(CoreTests, self).setUp()
File "/home/aron/src/ka-lite/kalite/student_testing/tests.py", line 38, in setUp
self.assertTrue(self.client.facility)
AssertionError: None is not TrueFixed. Just had CoreTests and CurrentUnitTests inherit from KALiteClientTestCase.
Now fixing more test breakage caused by MRO errors (a test case inheriting from both CreateDeviceMixin and KALiteTestCase, which itself inherits CreateDeviceMixin).
Now getting this error when running all the tests:
Traceback (most recent call last):
File "/home/aron/src/ka-lite/python-packages/django/core/management/base.py", line 222, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/aron/src/ka-lite/python-packages/django/core/management/commands/test.py", line 80, in execute
super(Command, self).execute(*args, **options)
File "/home/aron/src/ka-lite/python-packages/django/core/management/base.py", line 255, in execute
output = self.handle(*args, **options)
File "/home/aron/src/ka-lite/python-packages/south/management/commands/test.py", line 8, in handle
super(Command, self).handle(*args, **kwargs)
File "/home/aron/src/ka-lite/python-packages/django/core/management/commands/test.py", line 106, in handle
failures = test_runner.run_tests(test_labels)
File "/home/aron/src/ka-lite/kalite/testing/testrunner.py", line 68, in run_tests
return run_tests_wrapper_fn()
File "/home/aron/src/ka-lite/python-packages/django/test/utils.py", line 220, in inner
return test_func(*args, **kwargs)
File "/home/aron/src/ka-lite/kalite/testing/testrunner.py", line 67, in run_tests_wrapper_fn
return super(KALiteTestRunner,self).run_tests(test_labels, extra_tests, **kwargs)
File "/home/aron/src/ka-lite/python-packages/django/test/simple.py", line 366, in run_tests
suite = self.build_suite(test_labels, extra_tests)
File "/home/aron/src/ka-lite/kalite/testing/testrunner.py", line 79, in build_suite
testfun = getattr(test, test._testMethodName)
AttributeError: 'TestSuite' object has no attribute '_testMethodName'Investigating.
Only happens on the code called when --failfast is on. Something related to these lines, found in:
if self.failfast:
for test in test_suite._tests:
testfun = getattr(test, test._testMethodName)
setattr(test, test._testMethodName, auto_pdb()(testfun))
return test_suite….
I got into this refactoring blackhole. Coming back out with refactored tests!
While rewriting the tests, I decided it would be nice to have a multiple cursor feature like sublime text. So I just downloaded the multiple-cursors library and integrated it into Emacs.
Back! Turns out someone already wrote the function. Just gotta clean it up and extend it to namespace the po files by version.
So the main problem I’m stuck with right now is calling add-file vs. update-file crowdin api call. They literally do only what they say.
Guess I can’t avoid some duplication of code.
Along the way I refactored the version.py file where the compontents of the version are broken down into their own variables, and VERSION just reconstructing itself based on that.
Urgh I somehow can’t form the right API call with the requests library. Here’s the function:
def upload_to_crowdin(files, project_key, project_id="ka-lite"):
api_call = "add-file"
url = "https://api.crowdin.com/api/project/{project_id}/{api_call}"
url = url.format(project_id=project_id,
api_call=api_call)
version_namespace = "%s.%s" % (version.MAJOR_VERSION, version.MINOR_VERSION)
pot_path = pathlib.Path(POT_DIRPATH)
files_to_upload = {"files[/KA Lite UI/%s-django.po]" % version_namespace: open((pot_path / "django.pot").resolve().__str__()),
"files[/KA Lite UI/%s-djangojs.po]" % version_namespace: open((pot_path / "djangojs.pot").resolve().__str__())}
get_params = {"key": project_key}
r = requests.post(url, params=get_params, data=files_to_upload)And CrowdIn is throwing me this error (as the body of r):
'<?xml version="1.0" encoding="ISO-8859-1"?>\n<error>\n <code>4</code>\n <message>No files specified in request</message>\n</error>\n'Finally got it working! You have to add in the file array syntax and put it in the files parameter for requests.post:
files_to_upload = {"files[/versioned/%s-django.po]" % version_namespace: open((pot_path / "django.pot").resolve().__str__()),
"files[/versioned/%s-djangojs.po]" % version_namespace: open((pot_path / "djangojs.pot").resolve().__str__())}
get_params = {"key": project_key}
r = requests.post(url, params=get_params, files=files_to_upload)Ok so the problem is that when a user clicks “Download” for a content we have no idea how long they’ve viewed it. We want to track that.
Jamie suggested looking into the focus/blur JS events.