Fossil

Timeline
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

About branch markdown-footnotes

The primary purpose of a footnote is to provide a reader with supplementary information in an unobtrusive way, so that the reader's attention is not distracted from the main line of a narrative. Thus a natural length of a footnote is usually a couple of sentences, maybe a paragraph. If a footnote exceeds a paragraph then it may be beneficial to rearrange a composition and incorporate that footnote as a subsection of the current (or some other) document and point there using a regular hyperlink.

There may be (at least) two ways to think of footnotes:

  1. From the viewpoint of a document's author (i.e. Markdown syntax):

    Within this point of view a footnote is defined either in the place where the corresponding numeric marker is inserted or at some other location in the document (e.g. after a paragraph or at the end of a document's source) and referenced by the corresponding label.
    The former will be called as inline while the later as referenced (or labeled).

  2. From the viewpoint of a person who reads a rendered document:

    Within this point of view either a footnote is explicitly associated with a specific text or it is the task of the reader to deduce an exact phrase to which that footnote applies. The former will be called span-bounded (or fragment-bounded) and the later as free-standing.

Hence there are four types of footnotes (suggestions for better naming are welcomed), and this branch implements all of them. Footnotes rendering is supported in all places where Markdown is supported (but see issues and limitations).

The syntax

Proposed syntax is documented along with the other markdown rules.1 It was desired to have a syntax that naturally extends conventional Markdown and which is consistent for all four types of footnotes.2

Free-standing labeled footnotes

It seems that by "footnotes in Markdown" Internet typically means referenced free-standing case. The syntax for that is more or less widespread and settled as:

  Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed doeiusmod
  tempor incididunt[^label] ut labore et dolore magna aliqua.

  [^label]: Ut enimad minim veniam, quis nostrud exercitation ullamco
     laboris nisi utaliquip ex ea commodo consequat.

The advantage of referenced footnotes is that a single footnote may be referenced multiple times and the place(s) of use may appear before the footnote's definition.

Free-standing inline footnotes

These may be more convenient for simple cases and as noted in the forum might be easier in maintenance.

For the time being, free markdown processors that support inline footnotes are unknown. The following variants were considered:

  1.  Lorem ipsum dolor sit amet(^Ut enimad minim veniam)
   
  2.  Lorem ipsum dolor sit amet[^Ut enimad minim veniam]
   
  3.  Lorem ipsum dolor sit amet^[Ut enimad minim veniam]
   
  4.  Lorem ipsum dolor sit amet^(Ut enimad minim veniam)

The first variant was chosen since it looks a bit more natural in the source form.

Span-bounded footnotes

These seems to be a pretty rare thing.
For referenced footnotes the following syntax variants were considered:

  1.  Lorem ipsum [dolor sit amet][^label]
   
  2.  Lorem ipsum [dolor sit amet](^label)
  
  3.  Lorem ipsum [dolor sit amet]^[label]
  
  4.  Lorem ipsum [dolor sit amet]^(label)

and for inline footnotes the following:

  1.  Lorem ipsum [dolor sit amet](^Ut enimad minim veniam)
  
  2.  Lorem ipsum [dolor sit amet][^Ut enimad minim veniam]
  
  3.  Lorem ipsum [dolor sit amet]^[Ut enimad minim veniam]
  
  4.  Lorem ipsum [dolor sit amet]^(Ut enimad minim veniam)

In the examples above both the scope of the footnote and its content are pretty short. That might become the typical case but the parser is capable of a longer snippets that span several lines (albeit not paragraphs).

In all cases the scope is specified inside square brackets because it is consistent with the syntax for regular links. The first variant in each category was selected as it seems consistent with the syntax for regular hyperlinks and simplifies implementation.

The corresponding text fragment of a span-bounded footnote is highlighted when a user follows footnote's back-reference or when that span is hovered over.

Styling with user-provided classes

If a footnote's text starts with a token of the special form then this token is used to derive a set of CSS classes that are added to that footnote and its references. This enables users to style elements of a particular footnote provided that the administrator provisioned and documented some special CSS classes in a custom skin. Default skin does not provide any of such special classes.

The token must start with a dot and must end with a colon; in between of these it must be a dot-separated list of words; each word may contain only ASCII alphanumeric characters and hyphens.

Linting

The numbers for misreferences, unreferenced footnotes and joined footnotes (that have several definitions with the same label) are counted.

If any of these counter is non-zero then TH1 variable $footnotes_issues_counters is set to the space separated list of corresponding integers. This simplifies reporting about issues with footnotes from within a header of a page (if such warning is provisioned in a custom skin).

Also --lint-footnotes option is added to the test-markdown-render command3. If this flag is given and footnotes in the input have issues, then the above-mentioned counters are printed to stderr and non-zero exit code is set.

Known issues and limitations

  1. There is an issue for webpages where <base href="..."> is inconsistent with the actual REQUEST_URI of a page. As of this writing preview tabs of the /wikiedit and /fileedit pages are affected. There was an attempt to fix base href but as of this writing it's not merged onto mainline. An attempt to fix it by including RQUEST_URI into the generated hyperlinks does not seem to help for these AJAXified pages.
    There is a hope to solve this issue before version 2.19 of Fossil.

  2. A footnote's text is parsed and rendered in "inline mode". This is what is needed for the usual case, but it means that block-level markup might not work inside of a footnote. If the "block-level" markup is desired then it can be done via HTML tags in the text of a footnote.

  3. Source text of an inline footnote may not contain blank lines. This limitation comes from the current architecture of the parser where paragraphs are identified before the links (and footnotes alike).
    That does not seem like a big problem in the light of the second point and the provisioned purpose of footnotes (described in the beginning).

Notes about implementation

A static integer counter is incremented upon each rendering of a Markdown document within a single process execution. This enables to generate unique IDs for footnotes and back-references within a single webpage. The counter is incremented for each invocation of markdown_to_html() in the hope for better durability of hyperlinks that point to a footnote (or a footnote's marker) within a particular post of the forum.

Counters for issues are global variables (g.ftntsIssues). The other options were considered as over-complication.

Security considerations

This implementation adds three places (above the ordinary Markdown processing) where the user input is used to construct HTML.

  1. REQUEST_URI is escaped via newly added escape_quotes() function before getting into the values of href attributes.

  2. Labels and source code of unreferenced footnotes are processed in the same way as other code blocks in Markdown source (via html_escape() function).

  3. User provided classes are constrained to ASCII alphanumeric characters and hyphens.

Thus it is believed that this feature does not introduce vulnerability for HTML injection.

Performance considerations

Footnotes processing always terminates. The amount of CPU required for the processing of typical footnotes should be on par with the rest of the Markdown processor. The worst theoretical case constitutes of a long chain of recursively nested inline footnotes. For this case the theoretical complexity is O(n2). Thus to keep CPU consumption in bounds the maximal depth of nesting that is considered is limited to 5.

See also:

Wikipedia
GitHub
StackExchange
https://daringfireball.net/2005/07/footnotes
https://www.markdownguide.org/extended-syntax/
https://support.typora.io/Markdown-Reference/#footnotes
https://michelf.ca/projects/php-markdown/extra/#footnotes
https://rephrase.net/box/word/footnotes/syntax/

Footnotes


  1. ^ If this version of Fossil supports4 footnotes then the actual syntax should be documented at /md_rules.
  2. ^ Some examples (albeit unusual) may be seen at /doc/markdown-footnotes/test/markdown-test3.md
  3. ^ A link to the corresponding check-in should also test extraction of backlinks from within footnotes.
  4. ^ These four footnotes should test the functionality of footnotes on Wiki pages.
50 most recent check-ins related to "markdown-footnotes"
2022-04-23
21:32
Extend Markdown with footnotes support. See known limitations and the corresponding forum thread. ... (check-in: 3990518b user: george tags: trunk)
17:23
Change signature of add_inline_footnote() in order to move away from returning of unreliable pointer. Amend a few comments. Fix a couple of minor issues that fuzzer complains about. ... (Closed-Leaf check-in: 0850862e user: george tags: markdown-footnotes)
15:56
Fix a possible heap-buffer-overflow in parse_htmlblock() introduced by [1e919d601f774fdb]. This is not related to footnotes but was revealed by fuzzing (case 80cbb6b185807e98a953426af7b1f802c9d13957). ... (check-in: bc4c5b63 user: george tags: markdown-footnotes)
2022-04-22
12:49
Remove redundant assert() that fails for the case when the content of a span-bounded inline footnote is rendered into a void. This is a corner case that was revealed via fuzzing. ... (check-in: cab8a586 user: george tags: markdown-footnotes)
2022-04-21
22:31
Add a test case for fragment-bounded footnote that contains markup within the corresponding text fragment. ... (check-in: d38ec43d user: george tags: markdown-footnotes)
21:13
Fix another use-after-realloc bug in handling of inline footnotes which was discovered during fuzzing. Also fix a few other issues revealed via fuzzer. ... (check-in: c5456211 user: george tags: markdown-footnotes)
13:36
Revert Makefile.in to normal builds (leave a few comments about fuzzing). ... (check-in: 94077966 user: george tags: markdown-footnotes)
13:16
Fix a use-after-free bug in handling of nested inline footnotes. The bug was discovered by fuzzing with '-fsanitize=fuzzer,undefined,address -DFOSSIL_FUZZ' appended to TCCFLAGS in Makefile.in. It's noteworthy that the ',undefined,address' part was essential to find the bug (otherwise just 'double-free' was reported). Many thanks to Stephan for documenting the fuzzing procedures and support. ... (check-in: 31e5df5f user: george tags: markdown-footnotes)
2022-04-20
14:07
Added a missing blob initializer. ... (check-in: 72095938 user: stephan tags: markdown-footnotes)
11:48
Merged in trunk for fuzz.c changes. ... (check-in: c9f40135 user: stephan tags: markdown-footnotes)
11:46
Correct fuzz.c to honor --fuzztype markdown and add --fuzztype wiki2 which works like its previous --fuzztype wiki behavior, sending all inputs through both the fossil-wiki and markdown translators. Added a fatal error for --fuzztype artifact, as that tester is not implemented. ... (check-in: 8d4c4792 user: stephan tags: trunk)
2022-04-19
15:25
Remove unnecessary field from the auxiliary union 'bitfield64_t' and amend the corresponding comments. Also add comment about FOOTNOTES_WITHOUT_URI macro. ... (check-in: cf1e9691 user: george tags: markdown-footnotes)
12:35
Code style tweaks, typos, and resolved a couple footnotes-related cosmetic TODOs. No functional changes. ... (check-in: 3a5b3d5e user: stephan tags: markdown-footnotes)
11:41
Merged in latest trunk to simplify code review and ease potential upcoming merge to trunk. ... (check-in: 8a4b099f user: stephan tags: markdown-footnotes)
2022-04-16
16:29
Minor spelling corrections. No change in functionality. ... (check-in: 53754fff user: andybradford tags: trunk)
2022-02-23
12:33
Count overnesting as the fourth type of the footnote-related issues and report accordingly. ... (check-in: ae297bb6 user: george tags: markdown-footnotes)
09:45
Minor refactoring. Move the definition of BLOB_APPEND_LITERAL() macro from markdown_html.c to blob.c so that it could be used outside of markdown_html.c. Also rename it to lowercase for consistency with other API. Within markdown.c use that newly available macro instead of blob_append_string(). Within markdown_html.c use it for footnotes-relevant code. Other invocations of BLOB_APPEND_LITERAL() within markdown_html.c are left intact (they use an alias) in order to simplify the potential merge with the trunk. ... (check-in: c8a8d0c9 user: george tags: markdown-footnotes)
08:21
Minor refactoring. Move the definition of BLOB_APPEND_BLOB() macro from markdown_html.c to blob.c so that it could be used outside of markdown_html.c. Also rename it to blob_appendb() for consistency with blob_appendf() and other API. Within markdown.c use that newly available macro where appropriate. Within markdown_html.c use it for footnotes-relevant code. Other invocations of BLOB_APPEND_BLOB() within markdown_html.c are left intact (they use an alias) in order to simplify the potential merge with the trunk. ... (check-in: 33a681eb user: george tags: markdown-footnotes)
07:36
Fix handling of user-provided classes for unreferenced, joined and overnested footnotes. In all these cases the tokens of user-provided classes are rendered as plain-text and no special classes are added anywhere. ... (check-in: 875472a8 user: george tags: markdown-footnotes)
2022-02-21
05:14
Add a comment for append_footnote_upc(). Also substitute a variable of zero value with just "0" constant. No functional changes. ... (check-in: ae8a3dd5 user: george tags: markdown-footnotes)
04:29
Impose a limit on the depth of nesting of inline footnotes. Also add a few test cases: for depth limiting and HTML hijacking. ... (check-in: f4ff013a user: george tags: markdown-footnotes)
2022-02-20
23:00
If there are issues with footnotes then set TH1 variable $footnotes_issues_counters to a space separated list of integers that count for "misref", "unref" and "joins". This eliminates the need for JavaScript for the case when a custom skin wants to warn about issues with footnotes in the header of a page.
Also fix counting of "joins": count the number of unique labels that have multiple definitions (and not the number of such definitions).
... (check-in: 773cef5c user: george tags: markdown-footnotes)
2022-02-19
01:16
Parse inline footnotes even if a renderer does not define a callback for rendering of footnote markers. This seems more correct even though the current implementation of backlink processor does define such callback as an empty function. ... (check-in: e06c12d1 user: george tags: markdown-footnotes)
01:00
Handle some corner cases more thoroughly: dismiss empty footnotes, passthrough (more carefully) user-provided classlist if the token is not followed by a blank character or if a footnote's text consists just of such token and blank characters. Also simplify a little bit a few places inside of is_footnote() function. ... (check-in: fe315780 user: george tags: markdown-footnotes)
2022-02-18
01:33
Add --lint-footnotes option to the test-markdown-render command. If this flag is given and footnotes in the input have issues, then print to stderr the counters of "misrefs", "strays" and "split-defs" and exit with error. This should partially address a concern raised at the forum. ... (check-in: 1f525713 user: george tags: markdown-footnotes)
2022-02-17
22:09
If a footnote's text starts with a token of the special form then use this token to derive a set of CSS classes that are added to that footnote and its references. This enables users to style elements of a particular footnote provided that the administrator provisioned and documented some special CSS classes in a custum skin. Default skin does not provide any of such special classes which makes this feature an "opt-in". ... (check-in: 92516ced user: george tags: markdown-footnotes)
00:17
Clean-up and rephrase some comments. ... (check-in: a62c8768 user: george tags: markdown-footnotes)
2022-02-16
23:08
Make parsing slightly faster and fix a comment. No changes in functionality. ... (check-in: a36dd09d user: george tags: markdown-footnotes)
22:11
Include REQUEST_URI into footnotes' hyperlinks. This should make links work even if base href (in a page's header) is not consistent with the REQUEST_URI. If FOOTNOTES_WITHOUT_URI macro is defined while compiling src/markdown_html.c then bare "#fragment" hyperlinks (without REQUEST_URI) are generated. ... (check-in: 2c1f8f35 user: george tags: markdown-footnotes)
2022-02-14
23:32
Minor code refactoring: rename a temporary variable and utilize matching_bracket_offset() one more time. No changes in functionality. ... (check-in: 5b845a07 user: george tags: markdown-footnotes)
2022-02-13
19:29
Fix parsing of "free-standing" footnotes that was (slightly) broken by the previous check-in. ... (check-in: 23c3e0b2 user: george tags: markdown-footnotes)
2022-02-12
20:52
If markup is ambigous between a "span-bounded" footnote and a "free-standing" footnote followed by another footnote then interpret as the later case. ... (check-in: b363a4db user: george tags: markdown-footnotes)
2022-02-11
01:26
Fix parsing of a multiline definition of labeled footnote for the case when lines end with CR+LF. ... (check-in: ea66d15c user: george tags: markdown-footnotes)
2022-02-10
23:00
Clean-up and polish relevant CSS and HTML's class names. Insure visual spacing between footnotes' markers so that numbers are distinguishable when multiple footnotes in a row are used. Factor out auxiliary decorations from HTML into the default CSS, to enable customization via skins. ... (check-in: 2b1375ab user: george tags: markdown-footnotes)
2022-02-09
22:59
Handle unreferenced footnotes. If a labeled footnote is defined but there are no references to it, then add a special item at the end of footnotes. This item includes a label and the text of the strayed footnote - both rendered verbatim via html_escape(). Default skin makes such items visible and easily distinguishable. The order of such items match the order in the underlying source code. ... (check-in: ada55cd4 user: george tags: markdown-footnotes)
20:23
Cherrypicked [92221aaa192e82] and [7283ae6e120c10] on behalf of George. ... (check-in: f902814d user: stephan tags: trunk)
20:09
Join duplicated footnotes slightly faster. Fix a comment about auxiliary cmp_footnote_id() function. ... (check-in: 7f6a6418 user: george tags: markdown-footnotes)
19:38
Fix a misuse of an unsigned integer in the blobReallocMalloc() which can lead to redundant memory reallocations. ... (check-in: 92221aaa user: george tags: markdown-footnotes)
19:29
Fix a bug in the blob_reserve() function that was introduced by [1243bf39996b8a]. The current mainline is not affected because this function is not used anywhere. However it was causing memory corruption on the 'markdown-footnotes' branch since it was employed in [544df852b2d9a1]. ... (check-in: 7283ae6e user: george tags: markdown-footnotes)
2022-02-08
14:09
An attempt to fix a "double free crash" from the previous check-in. ... (check-in: 18c9d103 user: george tags: markdown-footnotes)
14:04
If several footnotes are defined with the same label then join them into a single footnote. Text from each definition becomes an item in the list. This solution makes such situations noticable for the usual case (when this is an oversight) but also not obtrusive for the rare cases (when this is intentional). The list is provided with a special class to enable styling via skin customization.
This check-in is known to cause crash, see the forthcoming check-in.
... (check-in: 544df852 user: george tags: markdown-footnotes)
13:39
Add const qualifier to the arguments of the blob_compare() function. ... (check-in: 2822b63b user: george tags: markdown-footnotes)
2022-02-06
22:53
Handle misreferences more thoroughly. Implement support of footnotes-within-footnotes with (hopefully) proper crosslinking (that's where it's getting tricky). ... (check-in: 1787f6df user: george tags: markdown-footnotes)
2022-02-04
23:07
Handle misreferences: a reference to undefined footnote. ... (check-in: 28e6a9cd user: george tags: markdown-footnotes)
19:47
Minor code refactoring. ... (check-in: 2636e224 user: george tags: markdown-footnotes)
19:24
Automatically render a horizontal rule before the list of footnotes. If desired a particular skin can hide it using CSS selector "hr.footnotes-separator". ... (check-in: 6807b434 user: george tags: markdown-footnotes)
19:08
Add file test/markdown-test3.md that is suggested as an accumulator of footnotes-specific test cases. ... (check-in: fe9e6ff9 user: george tags: markdown-footnotes)
17:28
For rendering a numeric footnote mark enclose HTML tag "a" inside of tag "sup" (instead of the opposite) and format anchor's id using "noteref%s-%i-%s" template (instead of "noteref-%s%i-%s"). Add highlighting when hovering over a span-bounded footnotes. ... (check-in: fb999972 user: george tags: markdown-footnotes)
16:54
Fix documentation so that an example of a referenced footnote definition inside of the fenced code block is not recognized as a real footnote defenition. This demonstrates a subtle gotcha and a possible work-arround of it. ... (check-in: 7229d0f5 user: george tags: markdown-footnotes)
00:37
Switch to (^...) for inline footnotes. Implement span-specific footnotes. Add documentation. ... (check-in: cae7a5d1 user: george tags: markdown-footnotes)