I. Quick demo

Computer science is all about speeding things up. This system doesn't escape the rule. Using it for posting a new entry on a DotClear2 blog is as quick as typing:

bash:Goldberg $ make new
Enter directory name: foobar
New directory created: [999_foobar]

bash:Goldberg $ cd 999_foobar

bash:Goldberg/999_foobar $ make edit

bash:Goldberg/999_foobar $ make view

bash:Goldberg/999_foobar $ make CONFIG=ovh publish

  1. make new clone a template directory to create our new message folder 999_foobar, where foobar is my message codename.

  2. make edit, launch ${EDITOR} to edit a markdown file. Markdown is a text-to-HTML conversion tool designed to help writing HTML content. You can see what it looks like by getting the source text of this current blog entry.

  3. make view displays a local preview of my message in my preferred web browser.

  4. make CONFIG=ovh publish uploads my message formatted into HTML, including the media files referenced inside with <object data="" />, <a href=""> or <img src="" />, to a remote DotClear2 blog with XML-RPC enabled. CONFIG=XXX indicates which XML-RPC configuration file I am using.

While invoking make publish, I can set different values for CONFIG=XXX. This way, I can test a message on a dummy test server before publishing it to my real blog. I just have to type the command make CONFIG=dummyServer publish.

If no value for CONFIG is given, publish.pl try to use the last one present in publish.log. This log file keeps track of the dates of the first publication and of the last edition.

    1 # ConfigFile MsgId first_date last_date
    2 sheeva  4   2010-07-12  2010-07-12
    3 mirror  11  2010-07-12  2010-07-12
    4 ovh 11  2010-07-13  2010-07-13

II. Rube Goldberg explained

II.a. Root directory

The system is organised in a hierarchy of folders. At the top, we've got something like this:

bash:~ $ cd Goldberg
bash:Goldberg $ ls
10_deepBlue/ 11_mails/ 12_SVGs/
999_jekyll/ 999_mysql/
Makefile Scripts/ Template/ Tests/

There is one folder for each blog entry (deepBlue, mails, SVGs, jekyll, ...). Artificial prefixes with numbers are used to sort directories by chronological orders. Those beginning with 999_ are, by convention, new blog entries which have not yet been published.

The Scripts/ directory contains various shell scripts used for processing text, communicating with XML-RPC or managing the message folders. The Template/ directory which is cloned when invoking make new. It holds the prototype folder of every new message.

Tests/ contains a bunch of unit test scripts.

Finally, the Makefile file can recurse over child folders and can be used to control the whole system:

bash:Goldberg $ make help
Make targets:
    [help]: display this help
    clean : remove all generated files
    all   : format all documents
    new   : create a new directory based on Template/
    view  : open all documents within a web browser
    test  : try unit tests

II.b. Message directory

Each message directory contains:

  • its own Makefile to manage its content

  • a Media/ directory where to-be-uploaded-with-our-HTML-message pictures and videos files are stored.

  • a publish.log file keeping track of where and when the blog entry was published

  • last but not the least, text.mkd, the markdown file containing the message

The provided Makefile gives access to a bunch of handful commands:

bash:Goldberg/999_foobar $ make help
make [all]:
    format the text document

make help:
    display this help

make edit:
    launch $EDITOR to edit "text.mkd"

make view:
    open the document with a web browser

make [CONFIG=foobar] publish:
    publish *.xhtml to a DotClear blog, including *ALL* the files
    in the directory "Media/"

    Configuration files are searched for in ../Scripts/XML-RPC/.
    If no configuration name is given, use, the last one logged
    in "publish.log"

make clean:
    remove all generated files

make debug:
    process markdown syntax and wrap the result with XHTML headers
    output the data to a local "debug.xhtml" file

The make targets view, clean and help are pretty self-explanatory. What is left for explanation is the text formatting process. As mentioned previously, it is based on an original text.mkd markdown file. This file is parsed by John Gruber perl script Markdown.pl and by others filters in Scripts/. All of them are standalone scripts piped together, as you can see in this excerpt of the Makefile:

    1 all: $(XHTML)
    2 
    3 $(XHTML): $(DOC) Makefile
    4     $(SCRIPTS)/Markdown_1.0.1/Markdown.pl $(DOC) |\
    5         $(SCRIPTS)/XML-RPC/dotclear.py -w |\
    6         $(SCRIPTS)/Externalize.pl |\
    7         $(SCRIPTS)/NumberHeaders.pl |\
    8         $(SCRIPTS)/LowerHeaders.pl |\
    9         $(SCRIPTS)/DOCTYPE_XHTML.pl >$@

If you want to extend the text formatting pipeline, simply add your script in the Script/Makefile.generic XHTML target. Rule of thumb : if formatting with simple regexes, prefer parsing the text before sending it to Markdown.pl; if formatting with XML filters (SAX, DOM), add your script after Markdown.pl and dotclear.py -w.

II.c. Under the hood

This whole system of Makefiles and text formatting scripts are here to put into action my previous proof of concept Script/XML-RPC/dotclear.py python script. If you just want to exchange data remotely with DotClear without all the current mess, look at it and try to see if it can help you.

bash:Goldberg $ ./Scripts/XML-RPC/dotclear.py -h
Usage: dotclear.py [options]

A simple script to manage entries on a remote DotClear2 host thanks to the
XML-RPC protocol

Options:
  -h, --help            show this help message and exit
  -c FILE, --conf=FILE  Read configuration from filename [default:
                        default.cfg]
  -l N, --list=N        List the last N posts
  -n, --new             Create a new entry
  -s ID, --show=ID      Show post given its ID
  -x ID, --extract=ID   Print to stdout the content of post ID
  -e ID, --edit=ID      Edit a post given its ID using your favorite $EDITOR
  -d ID, --delete=ID    Delete an entry
  -w, --wrap            Read stdin and wrap it inside internal
                        XHTML template
  -r FILE, --raw=FILE   Do not wrap HTML code with headers
  -u FILE, --upload=FILE
                        Upload a file to the server

III. Pro and cons

III.a. Pros

You can easily customized your text processing pipeline. I choose to write my messages using the markdown syntax and the highlight utility to colorize source code fragments. But if you wish, you can replace them by anything you want, markdown-extra, textile, pygments, ...

I've chosen markdown because of 1) the syntax is really keen on eyes; 2) you can insert inline HTML soup when you need more control over how things should be displayed.

Highlight is good because I've included its highlight.css CSS stylesheet in my own blog theme (!) (one different highlight CSS theme include in each of my DotClear theme) and ... it was the only solution I knew of when I started using VIM/XML-RPC.

If you need more help, I've tried to put some documentation in each of my scripts. So try the command line argument -h or --help, PERL and Python online documentation -- perldoc script.pl and pydoc ./script.py -- or simply read the source code !

Last but not the least, it doesn't run on Windows® :-)

III.b. Cons

If you cannot bear the command line or the text interface, this program is obviously not for you.

The programming languages are not homogeneous. It's a mix of Makefiles, Perl and Python (2.x). If you are brave enough, go ahead and just re-implement all the scripts (and Makefiles ?) in whatever has your preference. If I've got enough time, I may try to rewrite them using Haskell, one day ...

Some problems remain concerning text encoding. Although I've created unit tests about this, UTf-8 may not always work. If you find some error cases, I would be glad to add them to my test suite and to fix them.

Yes ! There are still some bugs lurking around. So if you find any, contact me !

/DS

Tip: To get both markdown and HTML syntax highlighting in VIM when editing a markdown file, you can set the filetype variable to markdown.html. See this vim add-on to enable VIM snippets and syntax for markdown files.