Can anyone draw me a plot of √[6(1 + 1/2^2 + 1/3^2 + 1/4^2...)] →π? Excel is not only ok, it’s recommended. Should bounce around and close in.

— Matt Parker (@standupmaths) March 14, 2012

Lots of people have done cool graphs using spreadsheets. Here’s a google spreadsheets version:

Here’s a nice spreadsheet simulating √[6(1 + 1/2^2 + 1/3^2 + 1/4^2...)] →π #piday RT @si_kelly: @standupmaths docs.google.com/spreadsheet/cc…

— Matt Parker (@standupmaths) March 14, 2012

As Matt says, Excel is recommended here because it’s very easy to set up the formulas and graph them. It makes it very intuitive. But, I’m going to try this out using Python just because I feel like playing with some math today and spreadsheets are pretty awkward on my little netbook.

Here’s the formula, made a little easier to read thanks to LaTeX and MathJax:

Here’s my first implementation:

>>> sum_fractions = 0.0
>>> values = []
>>> for i in range(1,20):
...     sum_fractions += 1.0/(i**2)
...     value = numpy.sqrt((6.0 * sum_fractions))
...     values.append(value)
...     print "After adding 1/%s^2 value is %0.10f" % (i, value)
...
After adding 1/1^2 value is 2.4494897428
After adding 1/2^2 value is 2.7386127875
After adding 1/3^2 value is 2.8577380332
After adding 1/4^2 value is 2.9226129861
After adding 1/5^2 value is 2.9633877010
After adding 1/6^2 value is 2.9913764947
After adding 1/7^2 value is 3.0117739478
After adding 1/8^2 value is 3.0272978567
After adding 1/9^2 value is 3.0395075896
After adding 1/10^2 value is 3.0493616360
After adding 1/11^2 value is 3.0574815067
After adding 1/12^2 value is 3.0642878178
After adding 1/13^2 value is 3.0700753719
After adding 1/14^2 value is 3.0750569156
After adding 1/15^2 value is 3.0793898260
After adding 1/16^2 value is 3.0831930203
After adding 1/17^2 value is 3.0865580258
After adding 1/18^2 value is 3.0895564350
After adding 1/19^2 value is 3.0922450523

We’ve stored the first 20 values in an array, so let’s graph them:

>>> pyplot.axhline(numpy.pi, linewidth=2, color='r')
<matplotlib.lines.Line2D object at 0xa2bb8ac>
>>> pyplot.plot(range(1,20), values, color='b')
[<matplotlib.lines.Line2D object at 0xa21938c>]

Not bad, but we’ll want to go a little farther out. Also, a fun addition to this was also posted:

@standupmaths A guess: did you want the alternating odd inverses summing to 4/π*? In case you did: twitter.com/outofthenorm2/…

— out of the norm (@outofthenorm2) March 14, 2012

*Oops, @outofthenorm2 meant to say π/4

So, let’s try this again defining some functions:

>>> def sqrt_six_times_sum_inv_squares(n):
...     return numpy.sqrt(6.0*sum(1.0/k**2 for k in range(1,n)))
... 
>>> def four_times_sum_inv_odds(n):
...     return 4.0*sum(((-1.0)**(k+1))/(2*k-1) for k in range(1,n))
... 
>>> for i in range(1, 10):
...     value = sqrt_six_times_sum_inv_squares(i)
...     print "At step %s value is %0.10f" % (i, value)
...
At step 1 value is 0.0000000000
At step 2 value is 2.4494897428
At step 3 value is 2.7386127875
At step 4 value is 2.8577380332
At step 5 value is 2.9226129861
At step 6 value is 2.9633877010
At step 7 value is 2.9913764947
At step 8 value is 3.0117739478
At step 9 value is 3.0272978567
>>> for i in range(1, 10):
...     value = four_times_sum_inv_odds(i)
...     print "At step %s value is %0.10f" % (i, value)
...
At step 1 value is 0.0000000000
At step 2 value is 4.0000000000
At step 3 value is 2.6666666667
At step 4 value is 3.4666666667
At step 5 value is 2.8952380952
At step 6 value is 3.3396825397
At step 7 value is 2.9760461760
At step 8 value is 3.2837384837
At step 9 value is 3.0170718171

Now we can graph it:

>>> pyplot.clf()
>>> N = 30
>>> xs = range(1,N)
>>> pyplot.axhline(numpy.pi, linewidth=2, color='r')
<matplotlib.lines.Line2D object at 0xa38060c>
>>> pyplot.plot(xs, [sqrt_six_times_sum_inv_squares(i) for i in xs], color='b')
[<matplotlib.lines.Line2D object at 0xa1a92cc>]
>>> pyplot.plot(xs, [four_times_sum_inv_odds(i) for i in xs], color='g')
[<matplotlib.lines.Line2D object at 0xa38096c>]
>>> pyplot.axis([0, N, 2, 4])
[0, 30, 2, 4]

There’s more information about approximating pi on this wikipedia page. You can get the source code for this blog post and play around with it on github.

In this tutorial we’ll walk through creating and then modifying a WordPress blog post. All the features mentioned in this blog post are available in Dexy 0.5.7. In order to post content to WordPress we will use its XMLRPC API, so before you try this yourself make sure you have enabled this option; it is disabled by default.

We’ll start in an empty directory. We need to configure our directory to know where our WordPress installation is, and to provide our username and password. Fortunately Dexy’s new filter commands make this really easy. The create_keyfile command from the WordPress filter will create the configuration file we need to store our credentials:

$ dexy help -on fcmd
====================
Help for 'dexy fcmd'
====================
   Run a command defined in a dexy filter.
   Arguments:
      alias - The alias of the filter which defines the custom command,
       [required] e.g. 'dexy fcmd --alias <value>'
      cmd - The name of the command to run,
       [required] e.g. 'dexy fcmd --cmd <value>'
      help - If true, just print docstring rather than running command
       [optional, defaults to 'False'] e.g. 'dexy fcmd --help False'
   Keyword Arguments:
      Additional arguments to be passed to the command
$ dexy fcmd -alias wp -cmd create_keyfile -help
   Creates a key file for WordPress in the local directory.

We could also call the create_keyfile filter command on the ApiFilter classs (the parent class of WordPressFilter), and this would create a keyfile in the user’s home directory that we could use to store all our API credentials:

$ dexy fcmd -alias apis -cmd create_keyfile -help
   Creates a key file in location specified by MASTER_API_KEY_FILE.

The user-wide approach is usually more convenient, since you can run your dexy scripts from anywhere and only need to have your credentials in one place. On the other hand, per-project config is useful if you need to override the defaults to post to a different blog or if you prefer to keep the credentials closer to where you will be using them.

Now we’ll call the filter command:

$ dexy fcmd -alias wp -cmd create_keyfile
$ ls -l .dexyapis
-rw-r--r-- 1 ana ana 108 2012-03-14 01:06 .dexyapis

This has created a JSON file named .dexyapis which looks like this:

{
    "wordpress": {
        "password": "TODO", 
        "url": "TODO", 
        "username": "TODO"
    }
}

You should replace the TODO items with the values appropriate for your website. The url required is the base url for your blog. Here is the config file that I used to create this example:

{
    "wordpress" : {
        "url" : "http://dexydemo.wordpress.com",
        "username" : "$WP_USERNAME",
        "password" : "$WP_PASSWORD"
    }
}

You’ll notice that I didn’t actually type my username and password into the .dexyapis file. You can type your actual username and password, or if you have environment variables set up to store your useranme and password then you can put references to environment variables by starting with a $, and the ApiFilter will fetch these for you. The advantage of using environment variables is that you can commit your .dexyapis file to your project repository without committing sensitive information.

Let’s test this out to make sure it’s working before we actually start writing our blog post. Let’s get a list of all the filter commands that are provided by the WordPressFilter:

$ dexy fcmds -alias wp
Filter commands defined in WordPressFilter...
create_keyfile
list_categories
list_methods

The list_categories filter command will print out the categories that are defined in your blog. It has to authenticate and visit the correct URL to do this, so it’s a good test (especially if you have custom categories set up):

$ dexy fcmd -alias wp -cmd list_categories -help
   List available blog post categories.
$ dexy fcmd -alias wp -cmd list_categories
categoryName
Dexy
Uncategorized

Ok, now that we have that working let’s move on to an actual blog post. To start with, let’s just try to post a simple static HTML file like this:

My new blog post! yay!

We also need to create a configuration file named wordpress.json to hold some simple metadata about our blog post.

{
    "title" : "My First Automated Blog Post"
}

Here is the .dexy configuration file we need:

{
    "index.html|wp": {
        "inputs": [
            "wordpress.json"
        ]
    }
}

The WordPress filter alias is just wp (you can see the filter reference here). We specify wordpress.json as an input so that if there are any changes to that file, Dexy will upload our blog post to wordpress again. You could omit this and just touch your blog post file to trigger an upload.

And that’s all we need to do to create our first blog post! Because this is a new dexy project, we need to run dexy setup:

$ dexy setup
Ok, we've created directories called logs and artifacts.
You are now ready to run dexy! If you have problems,
please check the log file at logs/dexy.log for clues.
Online help is available from dexy.it/help

And then we can run dexy:

$ dexy -loglevel DEBUG
batch id is 1
sorting 2 documents into run order, there are 1 total dependencies
ratio of dependencies to documents is 0.5
running Output
running LongOutput
running Run
running Source
source files saved in logs/batch-source-00001

If this has completed successfully, then our wordpress.json has changed. It now looks like this:

{
    "postid": "163", 
    "publish": false, 
    "title": "My First Automated Blog Post"
}

The postid has been filled in. This is very important since next time we run this filter, we don’t want to create another blog post, we want to update this same blog post. You can fill in the postid yourself if you already have created a draft post via your WordPress admin interface. Also notice that there is a key called publish with a value of false. By default, we create draft blog posts, so they aren’t visible to the public yet. But, you can get a preview if you are logged in. Here’s our preview:

Let’s go ahead and publish this before we move on to creating a more interesting example. To make this visible, just change the value of publish from false to true, and run dexy again.

Now this is published and anyone can read it.

Where things really get interesting is when we start making use of Dexy to create blog post content that includes code examples and results, so let’s do an example of that next.

One thing I’m not going to cover in this post, but will cover in detail very soon, is how to include dexy-generated images in your blog post. The WordPress filter will actually already upload images for you to WordPress, but there is a snag. The overwrite option in the WordPress API is broken, so each time Dexy thinks your image has changed, another copy will be uploaded to WordPress. This clutters up your Media Library very quickly. (See this bug report for more information.) My approach is to use Amazon S3 for image hosting, and there is a boto filter in Dexy which will upload images to S3 and return the URL, which I can then use to reference the image. That’s how I am displaying the dexy-generated screenshots in this blog post. I’m also working on adding filters for other image and hosting services. There will be a comprehensive blog post on this topic soon, so watch this space (and if you are in a position to do anything to help that WordPress bug get fixed, please do).

When we include code snippets and examples, we probably want to include syntax highlighting. The approach I find easiest is to use pygments within Dexy to apply syntax highlighting, and then to have a pygments stylesheet as part of my blog CSS. If you can’t modify the CSS for your blog, then you can still use pygments with the "noclasses" : true option, and this will embed the styles directly.

If you would like to use a different syntax highlighting system, and are having trouble getting it to work, then please open a support ticket.

For this example, we will tell pygments to use inline styles rather than classes, so no modification to the CSS is required (which is good since I’m doing this demo using a free wordpress.com blog, so I can’t modify the CSS).

We will write a short blog post that shows parallel code written in Python and JavaScript, and executed using the python REPL and the rhino interpreter respectively. Pygments has support for python console transcripts, so we will have syntax highlighting on the Python output. However, there is not (yet) a comparable highlighter for JavaScript, so we will just show the raw output and surround it with <pre> tags.

We’ll use markdown rather than HTML, so we don’t have to type as much markup. Here is the source for our blog post:

Let's look at defining and calling functions in Python and JavaScript.

Here is an example of defining a function in Python:

{{ d['example.py|idio|pycon|pyg']['define-function'] }}

Here is how this function might be defined in JavaScript:

<pre>
{{ d['example.js|idio|rhinoint']['define-function'] }}
</pre>

Now we'll call this function in each language:

{{ d['example.py|idio|pycon|pyg']['call-function-1'] }}
<pre>
{{ d['example.js|idio|rhinoint']['call-function-1'] }}
</pre>

And again, with different arguments:

{{ d['example.py|idio|pycon|pyg']['call-function-2'] }}
<pre>
{{ d['example.js|idio|rhinoint']['call-function-2'] }}
</pre>

We’ll apply the jinja and markdown filters first, then post to wordpress. We also have several inputs, including a python script and a JavaScript example. Here is the .dexy config:

{
    "example.js|idio|rhinoint": {}, 
    "example.py|idio|pycon|pyg": {
        "pygments": {
            "noclasses": true
        }
    }, 
    "index.md|jinja|markdown|wp": {
        "allinputs": true
    }, 
    "wordpress.json": {}
}

We will also tweak our wordpress.json to change the title of our blog post, and assign it to a category:

{
    "categories": [
        "Dexy"
    ], 
    "postid": "163", 
    "publish": true, 
    "title": "My Fancier Automated Blog Post"
}

And, here’s the screenshot of our updated blog post:

If you are new to Dexy, then reviewing tutorials 0 and 1 will help you understand how the HTML for that blog post was created.

That’s all for this tutorial! You can check out the source code of this blog post at github. If you have any questions please leave a comment, or if you need help getting this to work please open a support ticket. You can also download an example project directory like the one described here to help you get started (although for learning it’s better to do it yourself).

• Pygments stylesheets for HTML and LaTeX are now available within the Jinja filter. This means you don’t have to make a separate stylesheet, you can insert the styles directly into your LaTeX document or HTML header (LaTeX example). Or, you can use Dexy to generate a stylesheet so you always have the most up-to-date style definitions and can easily change them.
• Virtual files are now more ‘virtual’. Previously if you wanted to create a .tgz or .zip archive from files dexy has generated, you needed to create a dummy empty file. Now you can just say “@archive.tgz|tgz” and a file with that name will be created for you (example).
• Filters have been added for PythonMagick and imlib2. I’ve run into some problems with PythonMagick, perhaps because I’ve had to install an older version, but the basic filters do work and they have been written so it’s easy to subclass the filter and just specify actions on the ‘image’ object. The imlib2 filters follow a similar pattern. I’ll be using these to add a template gallery to the Dexy website soon, so expect a blog post with more detail.
• Dexy now has basic support for accessing files in remote git repositories. If you have the GitPython package installed, then you can specify a repository URL and a path and Dexy will fetch this file for you. Fancy stuff like specifying a particular commit will be coming soon.
• Now if you have an ‘allinputs’ document in a subdirectory, this document will be available to any ‘allinputs’ documents in parent directories. So, say, your website index page can have access to PDF pages in a subdirectory, without needing to list them explicitly as inputs.

Something else to note is that Dexy development has moved over to Github, and the Bitbucket repositories are going to be decommissioned soon. More updates on this, new documentation and all other Dexy news coming soon!

The Computable Document Format (CDF) is a document format with a published, open specification (apparently, although I haven’t been able to actually find this specification as yet). Right now the only way to create CDF documents is by using Mathematica, and the only way to view CDF documents is by using Wolfram’s CDF viewer.

Here is a summary of some of the differences between Mathematica/CDF and Dexy:

Free vs. Commercial

The most obvious difference is that Dexy is free software. For now, in order to create CDF documents, you must have Mathematica. Even after paying your Mathematica license fee, you may need to pay more in order to distribute the CDF documents you create. It’s free to give away your CDF documents (your readers will have to put up with a Wolfram splash screen), but you can’t sell or even prevent others from republishing your blog posts without paying. Also “Enhanced capabilities, including import/export of external data, arbitrary input fields, dialog windows, and file saving” cost extra. Yep, this solution for “reproducibility” won’t let you export the data (unless Wolfram provides the data) unless you pay up.

Choice of Output Formats

Wolfram’s Computable Document Format is a document format. Dexy is a tool which lets you output in basically any document format which can be scripted or compiled. In the sense that you can publish CDF documents for viewing on the web (except, for now, on Linux), on the Desktop, and for iPad/mobile devices (these are “coming soon”), then CDF does give you some choice in output formats. However, all of these options require your user to download and install new software. Currently, this software is 231 MB in size. Big enough that I cannot download it at present as I am travelling.

Right now the only way to author CDF documents is to use Mathematica, and the only way to view CDF documents is to use Wolfram’s viewer. If there were an open source authoring tool to create CDF documents that did not depend on Mathematica code, well, then CDF would simply be another output option for Dexy.

Reusability

One of the things that differentiates Dexy from most other literate programming frameworks is the decoupling of code and documents. With Dexy, your code lives in code files which can be located anywhere, even in-situ in your project. You simply pull code into your document when and where you need it, via filters which transform it in various ways. This means you can write an example script and re-use this script in several places, say a blog post, a tutorial, a research paper and a book. You can just show code, actually run it, or both.

Mathematica-created CDFs use the traditional 1:1 paradigm of having you write your code within your document. So your code is embedded and can really only be used in that one place. This may be a little simpler and more intuitive, especially for non-programmers, but it seriously limits what you can do. (I’m sure there are workarounds but this defeats the purpose.)

Interactivity

The CDF format is highly interactive. You can tweak parameters and watch how this changes a graph. That is cool, and may be really useful for getting people interested in your data, but interactivity is not typically compatible with reproducibility.

A nice exception to this is an approach like http://www.commentspace.net/ where you can explore interactive data, snapshot it so others can see how you were viewing it, and comment.

Interactivity without a “save as source” option means that readers are simply consumers. A student interacting with your model using CDF can’t write a report about it, a customer can’t change some specifications and send you back a document with “I think this is more realistic”. They can take screenshots perhaps, but these are not reproducible.

Documents created using Dexy are static, not interactive (unless you use Dexy to feed data and source code to, say, a flash app where the results can be manipulated). However, in many circumstances this is preferable. With Dexy you can manipulate source code and see the results quickly, rather than instantly, and if you share your sources anyone can interact with your data and source at a deeper, more meaningful level than just playing around with rotating a 3D model, as fun as that might be. And if you use open source software, then people don’t have to pay someone else a license fee to engage with your content.

Choice

Languages supported by Mathematica: Mathematica

Languages/Systems supported by Dexy (so far)*: Python, R, Ruby, Java, PHP, Erlang, JavaScript, JRuby, Jython, Dot, C, C++, Clojure, Ragel

Formats supported by CDF: CDF

Formats supported by Dexy (so far)*: plain text, markdown, textile, HTML, LaTeX, Asciidoc

*If what you want isn’t listed, get in touch.

Transparency

When you document your scripts with Dexy, you can choose to document the source code underlying the implementations of the functions you call, assuming these are open source. Basing scientific computations on a closed-source interpreter means having to take Mathematica’s word for the correctness of its implementation, and having to reply on Mathematica’s documentation for the specifics of that implementation.

It’s Not All Bad

I have been very harsh here on Mathematica/CDF. That is not to say that it isn’t a highly streamlined and convenient package which I’m sure will provide a lot of value to business users. Open source projects can always use more inspiration on how to make data management and reporting a more seamless part of the process. However, for the sake of science and learning, a closed source pay-to-play engine with prohibitively large dependencies is going to make reproducibility and meaningful interactivity worse, not better.

I will absolutely agree that for a non-programmer, and probably even for most programmers, creating a document for the first time in Mathematica is probably a much faster and smoother process than doing the same in Dexy. Dexy does have a learning curve, although I am always working to make this as gentle as possible. For some people, this convenience will trump all other considerations. However for reproducibility, flexibility and control over your data and source code, these other considerations are very weighty indeed.

When we eventually get to look at the open specification for CDF, then it may be that this turns out to be a useful standalone document format which, once decoupled from Mathematica, could provide the benefits of interactivity to open source projects, and hopefully this would enable the development of features to promote iterative feedback and learning. This would depend on the standard itself and also the attitude of Wolfram towards CDF being a genuinely open standard. If this did happen, unfortunately Mathematica users would probably be left out as their licensing terms would preclude using these features.

Please note that due to the huge download size I was unable to actually run a CDF viewer, and while I have done my best to be accurate, it it possible that I have some details wrong (since I also have no way of running Mathematica). Please, please let me know in the comments if I have made any inaccurate statements about how Mathematica or CDF works.

I’ve also been chatting with several new and prospective Dexy users, and trying to get a feel for what issues people encounter when they try to install and run Dexy and start working through the tutorials. It’s great when I can provide just a few small tips to help people get started, but of course I have no idea how many other people run into those small little roadblocks and decide “oh, I don’t have time to troubleshoot this now, I’ll come back to this later” or even “that error looks scary, I’m never running this program again”. I do use feedback to refine tutorials and improve error messages, but I know I’m not seeing all the potential problems.

On the other side of this, I had an incredibly frustrating Sunday with a not-quite-polished web application, and a particularly frustrating afternoon today trying to figure out how to set up my EC2 environment for command line use. In both cases I really wanted to provide constructive feedback on my experiences so the app and documentation (respectively) could be improved, but by the time I had solved the issues in question I realized that (a) I really didn’t want to spend any more time on them and (b) I probably couldn’t remember and reconstruct all the issues I ran into, so it wouldn’t have been a terribly useful exercise for me to provide feedback, without a lot of work (GOTO a).

However, out of frustration came an idea. While last week I was toying with the idea of suggesting that prospective Dexy users arrange a time and Skype me when they are first attempting to use Dexy so I can provide real-time assistance, now I have refined this to the following idea which I am going to put into practice:

Whenever you are about to install or upgrade software, use new or recently upgraded software, or try to do something unfamiliar, or just any time your spidey senses are tingling, start a screen recorder. Preferably one that catches sound and, ideally, webcam video. Use the audio to explain what you are doing as you do it. When you are finished, if all has gone well, then you can just delete the recording. If all has not gone well, then you can watch the recording, note down the times at which important things happen, and email the developers with your documented tale of woe.

Will this be practical? I have no idea. Will this backfire with users not bothering to do as much troubleshooting as they otherwise might? Not sure about that either.

I know that I, as a developer, want to know about people having bad experiences with my software. Many of these experiences will have nothing to do with my software, but I still want to know about them.

I know that I, as a user, want to be able to express my frustration when things go wrong to the creators of software. I want to do this in a constructive way, but part of me also wants to convey the very real cost that poor design or poor documentation imposes. I also want to be able to easily illustrate ways in which software could be made more beneficial, and to be able to provide accurate and useful feedback when I encounter an issue.

So, I personally will be doing this screen recording experiment. And, if you are a Dexy user or are about to try Dexy, I encourage you to do it too. There’s no need to send me links to videos where everything went well (unless you’ve done something very cool with Dexy and want to tell me about it – then please do!) but I encourage you to send me links to videos that illustrate a problem or roadblock. Or, if you have suggestions for improving the tutorials or other documentation. Preferably with an explanatory email telling me what you were attempting to do (if it’s not clear from the voiceover), and I’d really appreciate timestamps where important things happen. I can’t promise to watch everything, but I will do what I can. And, please use this as a supplement to rather than replacement for the usual information you submit with a support request.

I have started a list of screen recording software with an emphasis on free, multi-platform software. Please feel free to make additional suggestions in the comments. I suggest people do a trial run to ensure that the software doesn’t place an undue load on your machine, and that your console screen is actually readable (a text-based, searchable console transcript is probably a good idea anyway – easy if you’re using GNU screen). A console transcript is also an option if you don’t want to record a video, and you can delete any sensitive content this way too.

If you upload your screencast to a hosted service, take note of their Ts and Cs in relation to the ownership of your upload. Take care that your screencasts don’t include any sensitive information like passwords or API keys, or any dodgy browser history that you don’t want the world to see. I certainly won’t rebroadcast anybody’s screencast without their explicit consent, or mention anybody by name [assuming recordings are in good faith, that is], but I will take note of the OS and software you are using in order to better tailor training materials, and I may discuss these in an anonymized way like “I notice a number of people who submitted screencasts to me weren’t using GNU Screen, let me tell you what a great utility this is.” If that bothers you, then, well, probably best not to send me anything since it will be hard for me to unsee. If you are happy for me to publish your screencast on the Dexy blog or elsewhere, then let me know and this will make things quicker for me in the event that I want to share some examples in future. If you give me suggestions for documentation improvements, I will assume that you want me to go ahead and make use of these.