Chapter 33 Make a GitHub repo browsable
The unreasonable effectiveness of GitHub browsability. One of my favorite aspects of GitHub is the ability to inspect a repository’s files in a browser. Certain practices make browsing more rewarding and can postpone the day when you must create a proper website for a project. Perhaps indefinitely.
33.1 Be savvy about your files
Keep files in the plainest, web-friendliest form that is compatible with your main goals. Plain text is the very best. GitHub offers special handling for certain types of files:
- Markdown files, which may be destined for conversion into, e.g., HTML
- Markdown files named
- HTML files, often the result of compiling Markdown files
- Source code, such as
- Delimited files, such as CSVs and TSVs
- PNG files
33.2 Get over your hang ups re: committing derived products
Let’s acknowledge the discomfort some people feel about putting derived products under version control. Specifically, if you’ve got an R Markdown document
foo.Rmd, it can be
knit() to produce the intermediate product
foo.md, which can be converted to the ultimate output
foo.html. Which of those files are you “allowed” to put under version control? Source-is-real hardliners will say only
foo.Rmd but pragmatists know this can be a serious bummer in real life. Just because I can rebuild everything from scratch, it doesn’t mean I want to.
The taboo of keeping derived products under version control originates from compilation of binary executables from source. Software built on a Mac would not work on Windows and so it made sense to keep these binaries out of the holy source code repository. Also, you could assume the people with access to the repository have the full development stack and relish opportunities to use it. None of these arguments really apply to the
foo.Rmd --> foo.md --> foo.html workflow. We don’t have to blindly follow traditions from the compilation domain!
In fact, looking at the diffs for
foo-figure-01.png can be extremely informative. This is also true in larger data analytic projects after a
make clean; make all operation. By looking at the diffs in the downstream products, you often catch unexpected changes. This can tip you off to changes in the underlying data and/or the behavior of packages you depend on.
This chapter explores cool things GitHub can do with various file types, if they happen to end up in your repo. I won’t ask you how they got there.
You will quickly discover that GitHub renders Markdown files very nicely. By clicking on
foo.md, you’ll get a decent preview of
foo.html. Yay! You should read GitHub’s own guide on how to leverage automatic Markdown rendering.
Exploit this aggressively. Make Markdown your default format for narrative text files and use them liberally to embed notes to yourself and others in a repository hosted on Github. It’s an easy way to get pseudo-webpages inside a project “for free”. You may never even compile these files to HTML explicitly; in many cases, the HTML preview offered by GitHub is all you ever need.
33.4 R Markdown
What does this mean for R Markdown files? Keep intermediate Markdown. Or only render to Markdown. Commit both
foo.md, even if you choose to
.gitignore the final product, e.g.
foo.docx. From September 2014, GitHub renders R Markdown files nicely, like Markdown, and with proper syntax highlighting, which is great. But, of course, the code blocks just sit there un-executed, so my advice about keeping Markdown still holds.
If your target output format is not Markdown, you want YAML frontmatter that looks something like this for
or like this for
keep_md: TRUE part says to keep the intermediate Markdown. In RStudio, when editing
.Rmd, click on the gear next to “Knit HTML” for YAML authoring help.
rmarkdown offers a custom output format for GitHub-flavored markdown,
github_document. Read about R Markdown workflows for explicit examples of how to use this. If Markdown is your target output format, your YAML can be even simpler and look like this for
or like this for
For a quick, stand-alone document that doesn’t fit neatly into a repository or project (yet), make it a Gist. Example: Hadley Wickham’s advice on what you need to do to become a data scientist. Gists can contain multiple files, so you can still provide the R script or R Markdown source and the resulting Markdown, as I’ve done in this write-up of Twitter-sourced tips for cross-tabulation. I’ve collected YAML examples for all the above scenarios in a gist.
You probably already know that GitHub renders
README.md at the top-level of your repo as the de facto landing page. This is analogous to what happens when you point a web browser at a directory instead of a specific web page: if there is a file named
index.html, that’s what the server will show you by default. On GitHub, files named
README.md play exactly this role for directories in your repo.
Implication: for any logical group of files or mini project-within-your-project, create a sub-directory in your repository. And then create a
README.md file to annotate these files, collect relevant links, etc. Now when you navigate to the sub-directory on GitHub the nicely rendered
README.md will simply appear. The GitHub repo that backs the gapminder data package has a README in the
data-raw subdirectory that explains exactly how the package data is created. In fact, it is generated programmatically from
Some repositories consist solely of
README.md. Examples: Jeff Leek’s write-ups on How to share data with a statistician or Developing R packages. I am becoming a bigger fan of
README-only repos than gists because repo issues trigger notifications, whereas comments on gists do not.
If you’ve got a directory full of web-friendly figures, such as PNGs, you can use code like this to generate a
README.md for a quick DIY gallery, as Karl Broman has done with his FruitSnacks. I did same for all the fantastic O RLY book covers made by The Practical Dev.
33.6 Finding stuff
OK these are pure GitHub tips but if you’ve made it this far, you’re obviously a keener.
tto activate the file finder whenever you’re in a repo’s file and directory view. AWESOME, especially when there are files tucked into lots of subdirectories.
yto get a permanent link when you’re viewing a specific file. Watch what changes in the URL. This is important if you are about to link to a file or to specific lines. Otherwise your links will break easily in the future. If the file is deleted or renamed or if lines get inserted or deleted, your links will no longer point to what you intended. Use
yto get links that include a specific commit in the URL.
If you have an HTML file in a GitHub repository, simply visiting the file shows the raw HTML. Here’s a nice ugly example:
No one wants to look at that.
You can provide this URL to rawgit.com to serve this HTML more properly and get a decent preview. You can form two different types of URLs with rawgit.com: For sharing low-traffic, temporary examples or demos with small numbers of people, do this: https://rawgit.com/STAT545-UBC/STAT545-UBC.github.io/master/bit003_api-key-env-var.html Basically: replace
For use on production websites with any amount of traffic, do this: https://cdn.rawgit.com/STAT545-UBC/STAT545-UBC.github.io/master/bit003_api-key-env-var.html Basically: replace
This sort of enhanced link might be one of the useful things to put in a
README.md or other Markdown file in the repo.
You may also want to check out this Chrome extension or GitHub & BitBucket HTML Preview, though recently I’ve more success with rawgit.com. (Neither work with private GitHub repos, which is all the more reason to keep intermediate markdown files for HTML, as described above.)
Sometimes including HTML files will cause GitHub to think that your R repository is HTML. Besides being slightly annoying, this can make it difficult for people to find your work if they are searching specifically for R repos. You can exclude these files or directories from GitHub’s language statistics by adding a .gitattributes file that marks them as ‘documentation’ rather than code. See an example here.
33.8 Source code
You will notice that GitHub does automatic syntax highlighting for source code. For example, notice the coloring of this R script. The file’s extension is the primary determinant for if/how syntax highlighting will be applied. You can see information on recognized languages, the default extensions and more at github/linguist. You should be doing it anyway, but let this be another reason to follow convention in your use of file extensions.
Note you can click on “Raw” in this context as well, to get just the plain text and nothing but the plain text.
33.9 Delimited files
GitHub will nicely render tabular data in the form of
.csv (comma-separated) and
.tsv (tab-separated) files. You can read more in the blog post announcing this feature in August 2013 or in this GitHub help page.
Advice: take advantage of this! If something in your repo can be naturally stored as delimited data, by all means, do so. Make the comma or tab your default delimiter and use the file suffixes GitHub is expecting. I have noticed that GitHub is more easily confused than R about things like quoting, so always inspect the GitHub-rendered
.tsv file in the browser. You may need to do light cleaning to get the automagic rendering to work properly. Think of it as yet another way to learn about imperfections in your data.
Here’s an example of a tab delimited file on GitHub: lotr_clean.tsv, originally found
here (nope, IBM shut down manyeyes July 2015).
Note you can click on “Raw” in this context as well, to get just the plain text and nothing but the plain text.
PNG is the “no brainer” format in which to store figures for the web. But many of us like a vector-based format, such as PDF, for general purpose figures. Bottom line: PNGs will drive you less crazy than PDFs on GitHub. To reduce the aggravation around viewing figures in the browser, make sure to have a PNG version in the repo.
- This PNG figure just shows up in the browser
- A different figure stored as PDF
produces the dreaded, annoying “View Raw” speed bump. You’ll have to click through and, on my OS + browser, wait for the PDF to appear in an external PDF viewer.2015-06-19 update: since I first wrote this GitHub has elevated its treament of PDFs so YAY. It’s slow but it works.
Hopefully we are moving towards a world where you can have “web friendly” and “vector” at the same time, without undue headaches. As of October 2014, GitHub provides enhanced viewing and diffing of SVGs. So don’t read this advice as discouraging SVGs. Make them! But consider keeping a PNG around as emergency back up for now.
33.11 Other document formats
You may also have a document you want others to be able to browse and interact with, but it is not in the markdown format. Fortunately, the open-source Pandoc program, written by John MacFarlane, allows you to convert a range of formats into markdown, including the widely used
When you click the Knit button in RStudio it is actually Pandoc which performs the final conversion to HTML or Microsoft Word (
.docx) formats. If you are willing to use the command-line, you can perform the opposite conversion (eg
.md), commonly retaining features such as headings, tables, equations and even figures.
As some boilerplate, running in Windows PowerShell
pandoc --extract-media .\media -f docx .\example.docx -t markdown_github -o example_image.md converts a word document called
example.docx to markdown, and extracts the images into a directory which corresponds to a filepath in the newly created
example.md document. A full list of supported formats and example code for conversions are available at https://pandoc.org/.
You can also perform simple conversions to GitHub-flavored markdown from different markdown flavours (Pandoc supports
markdown_strict) from within RStudio. To do so you need to rename the file by changing the extension (eg from
foo.Rmd), then open the renamed file in RStudio and add the following text to the top of the document.
You can then click on “Knit” then “Knit to github document” to perform the conversion. See Output format for more details of controlling output formats with the YAML frontmatter.
33.14 Let people correct you on the internet
They love that!
You can create a link that takes people directly to an editing interface in the browser. Behind the scenes, assuming the click-er is signed into GitHub but is not you, this will create a fork in their account and send you a pull request. When I click the link below, I am able to actually commit directly to
master for this repo.
Here’s what that link looks like in the Markdown source:
[CLICK HERE to suggest an edit to this page!](https://github.com/jennybc/happy-git-with-r/edit/master/workflows-make-github-repo-browsable.Rmd)
and here it is with placeholders:
[INVITATION TO EDIT](<URL to your repo>/edit/master/<path to target source file>)
AFAIK, to do that in a slick automatic way across an entire repo/site, you need to be using Jekyll or some other automated system. But you could easily handcode such links on a small scale.