3 [![Build Status][travis-badge]][travis]
4 [![Coverage Status][codecov-badge]][codecov]
5 [![Chat][chat-badge]][chat]
7 **unified** is an interface for processing text using syntax trees. It’s what
8 powers [**remark**][remark], [**retext**][retext], and [**rehype**][rehype],
9 but it also allows for processing between multiple syntaxes.
11 The website for **unified**, [`unifiedjs.github.io`][site], provides a less
12 technical and more practical introduction to unified. Make sure to visit it
13 and try its introductory [Guides][].
26 var unified = require('unified')
27 var markdown = require('remark-parse')
28 var remark2rehype = require('remark-rehype')
29 var doc = require('rehype-document')
30 var format = require('rehype-format')
31 var html = require('rehype-stringify')
32 var report = require('vfile-reporter')
40 .process('# Hello world!', function(err, file) {
41 console.error(report(err || file))
42 console.log(String(file))
53 <meta charset="utf-8">
54 <meta name="viewport" content="width=device-width, initial-scale=1">
64 * [Description](#description)
66 * [processor()](#processor)
67 * [processor.use(plugin\[, options\])](#processoruseplugin-options)
68 * [processor.parse(file|value)](#processorparsefilevalue)
69 * [processor.stringify(node\[, file\])](#processorstringifynode-file)
70 * [processor.run(node\[, file\]\[, done\])](#processorrunnode-file-done)
71 * [processor.runSync(node\[, file\])](#processorrunsyncnode-file)
72 * [processor.process(file|value\[, done\])](#processorprocessfilevalue-done)
73 * [processor.processSync(file|value)](#processorprocesssyncfilevalue)
74 * [processor.data(key\[, value\])](#processordatakey-value)
75 * [processor.freeze()](#processorfreeze)
77 * [function attacher(\[options\])](#function-attacheroptions)
78 * [function transformer(node, file\[, next\])](#function-transformernode-file-next)
80 * [Contribute](#contribute)
81 * [Acknowledgments](#acknowledgments)
86 **unified** is an interface for processing text using syntax trees. Syntax
87 trees are a representation understandable to programs. Those programs, called
88 [**plugin**][plugin]s, take these trees and modify them, amongst other things.
89 To get to the syntax tree from input text there’s a [**parser**][parser]. To
90 get from that back to text there’s a [**compiler**][compiler]. This is the
91 [**process**][process] of a **processor**.
94 | ....................... process() ......................... |
95 | ......... parse() ..... | run() | ..... stringify() ....... |
97 +--------+ +----------+
98 Input ->- | Parser | ->- Syntax Tree ->- | Compiler | ->- Output
99 +--------+ | +----------+
109 Every processor implements another processor. To create a new processor invoke
110 another processor. This creates a processor that is configured to function the
111 same as its ancestor. But when the descendant processor is configured in the
112 future it does not affect the ancestral processor.
114 When processors are exposed from a module (for example, unified itself) they
115 should not be configured directly, as that would change their behaviour for all
116 module users. Those processors are [**frozen**][freeze] and they should be
117 invoked to create a new processor before they are used.
121 The syntax trees used in **unified** are [**Unist**][unist] nodes: plain
122 JavaScript objects with a `type` property. The semantics of those `type`s are
123 defined by other projects.
125 There are several [utilities][unist-utilities] for working with these nodes.
127 ###### List of Processors
129 The following projects process different syntax trees. They parse text to
130 their respective syntax tree and they compile their syntax trees back to text.
131 These processors can be used as-is, or their parsers and compilers can be mixed
132 and matched with **unified** and other plugins to process between different
135 * [**rehype**][rehype] ([**HAST**][hast]) — HTML
136 * [**remark**][remark] ([**MDAST**][mdast]) — Markdown
137 * [**retext**][retext] ([**NLCST**][nlcst]) — Natural language
139 ###### List of Plugins
141 The below plugins work with **unified**, unrelated to what flavour the syntax
144 * [`unified-diff`](https://github.com/unifiedjs/unified-diff)
145 — Ignore messages for unchanged lines in Travis
147 See [**remark**][remark-plugins], [**rehype**][rehype-plugins], and
148 [**retext**][retext-plugins] for lists of their plugins.
152 When processing documents metadata is often gathered about that document.
153 [**VFile**][vfile] is a virtual file format which stores data and handles
154 metadata and messages for **unified** and its plugins.
156 There are several [utilities][vfile-utilities] for working with these files.
160 To configure a processor invoke its [`use`][use] method, supply it a
161 [**plugin**][plugin], and optionally settings.
165 **unified** can integrate with the file-system through
166 [`unified-engine`][engine]. On top of that, CLI apps can be created with
167 [`unified-args`][args], Gulp plugins with [`unified-engine-gulp`][gulp], and
168 Atom Linters with [`unified-engine-atom`][atom].
170 A streaming interface is provided through [`unified-stream`][stream].
172 ###### Programming interface
174 The API gives access to processing metadata (such as lint messages) and
175 supports multiple passed through files:
178 var unified = require('unified')
179 var markdown = require('remark-parse')
180 var styleGuide = require('remark-preset-lint-markdown-style-guide')
181 var remark2retext = require('remark-retext')
182 var english = require('retext-english')
183 var equality = require('retext-equality')
184 var remark2rehype = require('remark-rehype')
185 var html = require('rehype-stringify')
186 var report = require('vfile-reporter')
199 .process('*Emphasis* and _importance_, you guys!', function(err, file) {
200 console.error(report(err || file))
201 console.log(String(file))
208 1:16-1:28 warning Emphasis should use `*` as a marker emphasis-marker remark-lint
209 1:34-1:38 warning `guys` may be insensitive, use `people`, `persons`, `folks` instead gals-men retext-equality
212 <p><em>Emphasis</em> and <em>importance</em>, you guys!</p>
215 ###### Processing between syntaxes
217 The processors can be combined in two modes.
219 **Bridge** mode transforms the syntax tree from one flavour (the origin) to
220 another (the destination). Then, transformations are applied on that tree.
221 Finally, the origin processor continues transforming the original syntax tree.
223 **Mutate** mode also transforms the syntax tree from one flavour to another.
224 But then the origin processor continues transforming the destination syntax
227 In the previous example (“Programming interface”), `remark-retext` is used in
228 bridge mode: the origin syntax tree is kept after retext is done; whereas
229 `remark-rehype` is used in mutate mode: it sets a new syntax tree and discards
232 * [`remark-retext`][remark-retext]
233 * [`remark-rehype`][remark-rehype]
234 * [`rehype-retext`][rehype-retext]
235 * [`rehype-remark`][rehype-remark]
241 Object describing how to process text.
245 `Function` — New [**unfrozen**][freeze] processor which is configured to
246 function the same as its ancestor. But when the descendant processor is
247 configured in the future it does not affect the ancestral processor.
251 The following example shows how a new processor can be created (from the remark
252 processor) and linked to **stdin**(4) and **stdout**(4).
255 var remark = require('remark')
256 var concat = require('concat-stream')
258 process.stdin.pipe(concat(onconcat))
260 function onconcat(buf) {
265 process.stdout.write(doc)
269 ### `processor.use(plugin[, options])`
271 Configure the processor to use a [**plugin**][plugin] and optionally configure
272 that plugin with options.
276 * `processor.use(plugin[, options])`
277 * `processor.use(preset)`
278 * `processor.use(list)`
282 * `plugin` ([`Plugin`][plugin])
283 * `options` (`*`, optional) — Configuration for `plugin`
284 * `preset` (`Object`) — Object with an optional `plugins` (set to `list`),
285 and/or an optional `settings` object
286 * `list` (`Array`) — List of plugins, presets, and pairs (`plugin` and
287 `options` in an array)
291 `processor` — The processor on which `use` is invoked.
295 `use` cannot be called on [frozen][freeze] processors. Invoke the processor
296 first to create a new unfrozen processor.
300 There are many ways to pass plugins to `.use()`. The below example gives an
304 var unified = require('unified')
307 // Plugin with options:
310 .use([plugin, pluginB])
311 // Two plugins, the second with options:
312 .use([plugin, [pluginB, {}]])
313 // Preset with plugins and settings:
314 .use({plugins: [plugin, [pluginB, {}]], settings: {position: false}})
316 .use({settings: {position: false}})
319 function pluginB() {}
322 ### `processor.parse(file|value)`
324 Parse text to a syntax tree.
328 * `file` ([`VFile`][file])
329 — Or anything which can be given to `vfile()`
333 [`Node`][node] — Syntax tree representation of input.
337 `parse` [freezes][freeze] the processor if not already frozen.
339 #### `processor.Parser`
341 Function handling the parsing of text to a syntax tree. Used in the
342 [**parse**][parse] phase in the process and invoked with a `string` and
343 [`VFile`][file] representation of the document to parse.
345 `Parser` can be a normal function in which case it must return a
346 [`Node`][node]: the syntax tree representation of the given file.
348 `Parser` can also be a constructor function (a function with keys in its
349 `prototype`) in which case it’s invoked with `new`. Instances must have a
350 `parse` method which is invoked without arguments and must return a
353 ### `processor.stringify(node[, file])`
355 Compile a syntax tree to text.
359 * `node` ([`Node`][node])
360 * `file` ([`VFile`][file], optional);
361 — Or anything which can be given to `vfile()`
365 `string` — String representation of the syntax tree file.
369 `stringify` [freezes][freeze] the processor if not already frozen.
371 #### `processor.Compiler`
373 Function handling the compilation of syntax tree to a text. Used in the
374 [**stringify**][stringify] phase in the process and invoked with a
375 [`Node`][node] and [`VFile`][file] representation of the document to stringify.
377 `Compiler` can be a normal function in which case it must return a `string`:
378 the text representation of the given syntax tree.
380 `Compiler` can also be a constructor function (a function with keys in its
381 `prototype`) in which case it’s invoked with `new`. Instances must have a
382 `compile` method which is invoked without arguments and must return a `string`.
384 ### `processor.run(node[, file][, done])`
386 Transform a syntax tree by applying [**plugin**][plugin]s to it.
390 * `node` ([`Node`][node])
391 * `file` ([`VFile`][file], optional)
392 — Or anything which can be given to `vfile()`
393 * `done` ([`Function`][run-done], optional)
397 [`Promise`][promise] if `done` is not given. Rejected with an error, or
398 resolved with the resulting syntax tree.
402 `run` [freezes][freeze] the processor if not already frozen.
404 ##### `function done(err[, node, file])`
406 Invoked when transformation is complete. Either invoked with an error or a
407 syntax tree and a file.
411 * `err` (`Error`) — Fatal error
412 * `node` ([`Node`][node])
413 * `file` ([`VFile`][file])
415 ### `processor.runSync(node[, file])`
417 Transform a syntax tree by applying [**plugin**][plugin]s to it.
419 If asynchronous [**plugin**][plugin]s are configured an error is thrown.
423 * `node` ([`Node`][node])
424 * `file` ([`VFile`][file], optional)
425 — Or anything which can be given to `vfile()`
429 [`Node`][node] — The given syntax tree.
433 `runSync` [freezes][freeze] the processor if not already frozen.
435 ### `processor.process(file|value[, done])`
437 Process the given representation of a file as configured on the processor. The
438 process invokes `parse`, `run`, and `stringify` internally.
442 * `file` ([`VFile`][file])
443 * `value` (`string`) — String representation of a file
444 * `done` ([`Function`][process-done], optional)
448 [`Promise`][promise] if `done` is not given. Rejected with an error or
449 resolved with the resulting file.
453 `process` [freezes][freeze] the processor if not already frozen.
455 #### `function done(err, file)`
457 Invoked when the process is complete. Invoked with a fatal error, if any, and
462 * `err` (`Error`, optional) — Fatal error
463 * `file` ([`VFile`][file])
468 var unified = require('unified')
469 var markdown = require('remark-parse')
470 var remark2rehype = require('remark-rehype')
471 var doc = require('rehype-document')
472 var format = require('rehype-format')
473 var html = require('rehype-stringify')
481 .process('# Hello world!')
484 console.log(String(file))
487 console.error(String(err))
498 <meta charset="utf-8">
499 <meta name="viewport" content="width=device-width, initial-scale=1">
502 <h1>Hello world!</h1>
507 ### `processor.processSync(file|value)`
509 Process the given representation of a file as configured on the processor. The
510 process invokes `parse`, `run`, and `stringify` internally.
512 If asynchronous [**plugin**][plugin]s are configured an error is thrown.
516 * `file` ([`VFile`][file])
517 * `value` (`string`) — String representation of a file
521 [`VFile`][file] — Virtual file with modified [`contents`][vfile-contents].
525 `processSync` [freezes][freeze] the processor if not already frozen.
530 var unified = require('unified')
531 var markdown = require('remark-parse')
532 var remark2rehype = require('remark-rehype')
533 var doc = require('rehype-document')
534 var format = require('rehype-format')
535 var html = require('rehype-stringify')
537 var processor = unified()
544 console.log(processor.processSync('# Hello world!').toString())
553 <meta charset="utf-8">
554 <meta name="viewport" content="width=device-width, initial-scale=1">
557 <h1>Hello world!</h1>
562 ### `processor.data(key[, value])`
564 Get or set information in an in-memory key-value store accessible to all phases
565 of the process. An example is a list of HTML elements which are self-closing,
566 which is needed when parsing, transforming, and compiling HTML.
570 * `key` (`string`) — Identifier
571 * `value` (`*`, optional) — Value to set. Omit if getting `key`
575 * `processor` — If setting, the processor on which `data` is invoked
576 * `*` — If getting, the value at `key`
580 Setting information with `data` cannot occur on [frozen][freeze] processors.
581 Invoke the processor first to create a new unfrozen processor.
585 The following example show how to get and set information:
588 var unified = require('unified')
592 .data('alpha', 'bravo')
603 ### `processor.freeze()`
605 Freeze a processor. Frozen processors are meant to be extended and not to be
606 configured or processed directly.
608 Once a processor is frozen it cannot be unfrozen. New processors functioning
609 just like it can be created by invoking the processor.
611 It’s possible to freeze processors explicitly, by calling `.freeze()`, but
612 [`.parse()`][parse], [`.run()`][run], [`.stringify()`][stringify], and
613 [`.process()`][process] call `.freeze()` to freeze a processor too.
617 `Processor` — The processor on which `freeze` is invoked.
621 The following example, `index.js`, shows how [**rehype**][rehype] prevents
622 extensions to itself:
625 var unified = require('unified')
626 var parse = require('rehype-parse')
627 var stringify = require('rehype-stringify')
629 module.exports = unified()
635 The below example, `a.js`, shows how that processor can be used and configured.
638 var rehype = require('rehype')
639 var format = require('rehype-format')
647 The below example, `b.js`, shows a similar looking example which operates on
648 the frozen [**rehype**][rehype] interface. If this behaviour was allowed it
649 would result in unexpected behaviour so an error is thrown. **This is
653 var rehype = require('rehype')
654 var format = require('rehype-format')
665 ~/node_modules/unified/index.js:440
669 Error: Cannot invoke `use` on a frozen processor.
670 Create a new processor first, by invoking it: use `processor()` instead of `processor`.
671 at assertUnfrozen (~/node_modules/unified/index.js:440:11)
672 at Function.use (~/node_modules/unified/index.js:172:5)
673 at Object.<anonymous> (~/b.js:6:4)
678 **unified** plugins change the way the applied-on processor works in the
681 * They modify the [**processor**][processor]: such as changing the parser,
682 the compiler, or linking it to other processors
683 * They transform [**syntax tree**][node] representation of files
684 * They modify metadata of files
686 Plugins are a concept. They materialise as [`attacher`][attacher]s.
693 module.exports = move
695 function move(options) {
696 var expected = (options || {}).extname
699 throw new Error('Missing `extname` in options')
704 function transformer(tree, file) {
705 if (file.extname && file.extname !== expected) {
706 file.extname = expected
715 var unified = require('unified')
716 var parse = require('remark-parse')
717 var remark2rehype = require('remark-rehype')
718 var stringify = require('rehype-stringify')
719 var vfile = require('to-vfile')
720 var report = require('vfile-reporter')
721 var move = require('./move')
726 .use(move, {extname: '.html'})
728 .process(vfile.readSync('index.md'), function(err, file) {
729 console.error(report(err || file))
731 vfile.writeSync(file) // Written to `index.html`.
736 ### `function attacher([options])`
738 An attacher is the thing passed to [`use`][use]. It configures the processor
739 and in turn can receive options.
741 Attachers can configure processors, such as by interacting with parsers and
742 compilers, linking them to other processors, or by specifying how the syntax
747 The context object is set to the invoked on [`processor`][processor].
751 * `options` (`*`, optional) — Configuration
755 [`transformer`][transformer] — Optional.
759 Attachers are invoked when the processor is [frozen][freeze]: either when
760 `.freeze()` is called explicitly, or when [`.parse()`][parse], [`.run()`][run],
761 [`.stringify()`][stringify], or [`.process()`][process] is called for the first
764 ### `function transformer(node, file[, next])`
766 Transformers modify the syntax tree or metadata of a file. A transformer is a
767 function which is invoked each time a file is passed through the transform
768 phase. If an error occurs (either because it’s thrown, returned, rejected, or
769 passed to [`next`][next]), the process stops.
771 The transformation process in **unified** is handled by [`trough`][trough], see
772 it’s documentation for the exact semantics of transformers.
776 * `node` ([`Node`][node])
777 * `file` ([`VFile`][file])
778 * `next` ([`Function`][next], optional)
782 * `Error` — Can be returned to stop the process
783 * [`Node`][node] — Can be returned and results in further transformations
784 and `stringify`s to be performed on the new tree
785 * `Promise` — If a promise is returned, the function is asynchronous, and
786 **must** be resolved (optionally with a [`Node`][node]) or rejected
787 (optionally with an `Error`)
789 #### `function next(err[, tree[, file]])`
791 If the signature of a transformer includes `next` (third argument), the
792 function **may** finish asynchronous, and **must** invoke `next()`.
796 * `err` (`Error`, optional) — Stop the process
797 * `node` ([`Node`][node], optional) — New syntax tree
798 * `file` ([`VFile`][file], optional) — New virtual file
802 Presets provide a potentially sharable way to configure processors. They can
803 contain multiple plugins and optionally settings as well.
810 exports.settings = {bullet: '*', fences: true}
813 require('remark-preset-lint-recommended'),
814 require('remark-comment-config'),
815 require('remark-preset-lint-markdown-style-guide'),
816 [require('remark-toc'), {maxDepth: 3, tight: true}],
817 require('remark-github')
824 var remark = require('remark')
825 var vfile = require('to-vfile')
826 var report = require('vfile-reporter')
827 var preset = require('./preset')
831 .process(vfile.readSync('index.md'), function(err, file) {
832 console.error(report(err || file))
835 vfile.writeSync(file)
842 **unified** is built by people just like you! Check out
843 [`contributing.md`][contributing] for ways to get started.
845 This project has a [Code of Conduct][coc]. By interacting with this repository,
846 organisation, or community you agree to abide by its terms.
848 Want to chat with the community and contributors? Join us in [Gitter][chat]!
850 Have an idea for a cool new utility or tool? That’s great! If you want
851 feedback, help, or just to share it with the world you can do so by creating
852 an issue in the [`unifiedjs/ideas`][ideas] repository!
856 Preliminary work for unified was done [in 2014][preliminary] for
857 [**retext**][retext] and inspired by [`ware`][ware]. Further incubation
858 happened in [**remark**][remark]. The project was finally [externalised][]
859 in 2015 and [published][] as `unified`. The project was authored by
860 [**@wooorm**](https://github.com/wooorm).
862 Although `unified` since moved it’s plugin architecture to [`trough`][trough],
863 thanks to [**@calvinfo**](https://github.com/calvinfo),
864 [**@ianstormtaylor**](https://github.com/ianstormtaylor), and others for their
865 work on [`ware`][ware], which was a huge initial inspiration.
869 [MIT][license] © [Titus Wormer][author]
873 [logo]: https://cdn.rawgit.com/unifiedjs/unified/0cd3a41/logo.svg
875 [travis-badge]: https://img.shields.io/travis/unifiedjs/unified.svg
877 [travis]: https://travis-ci.org/unifiedjs/unified
879 [codecov-badge]: https://img.shields.io/codecov/c/github/unifiedjs/unified.svg
881 [codecov]: https://codecov.io/github/unifiedjs/unified
883 [chat-badge]: https://img.shields.io/gitter/room/unifiedjs/Lobby.svg
885 [chat]: https://gitter.im/unifiedjs/Lobby
887 [npm]: https://docs.npmjs.com/cli/install
891 [author]: http://wooorm.com
893 [site]: https://unifiedjs.github.io
895 [guides]: https://unifiedjs.github.io/#guides
897 [rehype]: https://github.com/rehypejs/rehype
899 [remark]: https://github.com/remarkjs/remark
901 [retext]: https://github.com/retextjs/retext
903 [hast]: https://github.com/syntax-tree/hast
905 [mdast]: https://github.com/syntax-tree/mdast
907 [nlcst]: https://github.com/syntax-tree/nlcst
909 [unist]: https://github.com/syntax-tree/unist
911 [engine]: https://github.com/unifiedjs/unified-engine
913 [args]: https://github.com/unifiedjs/unified-args
915 [gulp]: https://github.com/unifiedjs/unified-engine-gulp
917 [atom]: https://github.com/unifiedjs/unified-engine-atom
919 [remark-rehype]: https://github.com/remarkjs/remark-rehype
921 [remark-retext]: https://github.com/remarkjs/remark-retext
923 [rehype-retext]: https://github.com/rehypejs/rehype-retext
925 [rehype-remark]: https://github.com/rehypejs/rehype-remark
927 [unist-utilities]: https://github.com/syntax-tree/unist#list-of-utilities
929 [vfile]: https://github.com/vfile/vfile
931 [vfile-contents]: https://github.com/vfile/vfile#vfilecontents
933 [vfile-utilities]: https://github.com/vfile/vfile#related-tools
939 [processor]: #processor
941 [process]: #processorprocessfilevalue-done
943 [parse]: #processorparsefilevalue
945 [parser]: #processorparser
947 [stringify]: #processorstringifynode-file
949 [run]: #processorrunnode-file-done
951 [compiler]: #processorcompiler
953 [use]: #processoruseplugin-options
955 [attacher]: #function-attacheroptions
957 [transformer]: #function-transformernode-file-next
959 [next]: #function-nexterr-tree-file
961 [freeze]: #processorfreeze
965 [run-done]: #function-doneerr-node-file
967 [process-done]: #function-doneerr-file
969 [trough]: https://github.com/wooorm/trough#function-fninput-next
971 [promise]: https://developer.mozilla.org/Web/JavaScript/Reference/Global_Objects/Promise
973 [remark-plugins]: https://github.com/remarkjs/remark/blob/master/doc/plugins.md#list-of-plugins
975 [rehype-plugins]: https://github.com/rehypejs/rehype/blob/master/doc/plugins.md#list-of-plugins
977 [retext-plugins]: https://github.com/retextjs/retext/blob/master/doc/plugins.md#list-of-plugins
979 [stream]: https://github.com/unifiedjs/unified-stream
981 [contributing]: contributing.md
983 [coc]: code-of-conduct.md
985 [ideas]: https://github.com/unifiedjs/ideas
987 [preliminary]: https://github.com/retextjs/retext/commit/8fcb1f#diff-168726dbe96b3ce427e7fedce31bb0bc
989 [externalised]: https://github.com/remarkjs/remark/commit/9892ec#diff-168726dbe96b3ce427e7fedce31bb0bc
991 [published]: https://github.com/unifiedjs/unified/commit/2ba1cf
993 [ware]: https://github.com/segmentio/ware