Unified place to Parse the options.

The argument passing has the following priority:

1. options.argv passed as parameter.

2. Arguments passed to the command line.

3. Options inside a JSON config file. Must be passed in 2. or

   1. as --config-file FILENAME.

Get one setting.

Indicates whether the hatch will be compressed into a tar.gz file after it is finished. The original hatch directory will not be removed.

Return an array of URLs if passed as settings.

The ingesters can use this URLs to reingest a batch of articles.

Waits for any pending downloads, creates the hatch manifests, and optionally compresses the archive into a tar.gz file.

Downloads and processes asset into metadata and data objects which will be saved in the hatch.

Explicitly save asset as failed.

Check if asset has failed.

Copies the generated hatch directory to directory. This can be used for automating the ingestion of content.

Copies the generated hatch directory to an S3 bucket on AWS.

20-byte hex ID of the document

Set multiple metadata at once.

Sets the text that will be shown as the synopsis of a document. (The synopsis is often shown in the UI on "cards" if there is enough space.)

Sets an image asset to be the article's thumbnail. (An image asset is returned e.g. from util.download_img or util.download_image.)

Pass the date that the post or article was published. You can pass a Date object here. If it is not a Date object, then new Date() will be called on it, so you can also pass e.g. a timestamp.

Pass in an array of ID strings for "sets" (categories, tags). The ID string is not the human-readable string (though it's possible for the two to be identical); you can create a human-readable name for the set in your app using other tools.

Compare NewsArticle#set_section.

Set multiple metadata at once.

Extends BaseAsset#set_metadata

Pass in the name of the post author here. (The Blog article format currently only is designed to handle one author.) For the News article format, see NewsArticle#set_authors.

Marks an image as the "main" image of the post, which is marked up specially. (Make sure to remove this image from the HTML that you pass to NewsArticle#set_body, otherwise it will be present twice in the rendered post.)

Sets a caption to go with the image set by BlogArticle#set_main_image.

Pass the body of the article as HTML here. Note that this will be rendered mostly unchanged, inside an <article> element. The ingester itself is responsible for cleaning up this HTML such that the stylesheet applies cleanly to it.

If you use BlogArticle#set_main_image, make sure to remove the main image from this HTML.

The ingester is responsible for passing cleaned-up HTML that the stylesheet's rules will apply to. In many cases this will not be necessary, but in other cases quite a lot of cleanup might be needed. Some hints to take into account:

• Figures should be inside <figure> elements, not paragraphs
• Figures with captions should also contain <figcaption> elements
• Quotes should be inside <blockquote> elements, properly formatted inside with <p> (and <cite> if applicable)
• Paragraphs should be in <p> elements, not denoted with <br> linebreaks

Here is where to customize the stylesheet. Most customizations should be able to be accomplished just by tweaking some SCSS variables which the stylesheet exposes. Here is a list of the variables:

• $primary-light-color
• $primary-dark-color
• $accent-light-color
• $accent-dark-color
• $background-light-color
• $background-dark-color
• $title-font
• $body-font
• $context-font
• $support-font

The default stylesheet is included with @import '_default';. (Note that you can also leave out this import and start from a blank stylesheet to completely do your own thing.) If you are adding rules as well as customizing variables, make sure that the variables are specified before the import, and the rules are specified after it.

Sets the text rendered at the bottom of the post. Unlike NewsArticle#set_read_more_link, this takes text rather than HTML, because the blog article format is a bit more strict about what it renders there. An example might be "Original article at Planet GNOME" which is turned into a link during rendering.

Pass in an array of ID strings for "sets" (categories, tags). The ID string is not the human-readable string (though it's possible for the two to be identical); you can create human-readable name for the set in your app using other tools. (Compare NewsArticle#set_section.)

Marks the page as a "static" page so that it will show up on the main menu of an app. Use this for e.g. "About the author" or "FAQ" pages.

Takes the metadata that has been set on this asset so far, and renders an HTML page internally in preparation for saving into a hatch.

You must call render() before passing this asset to Hatch#save_asset. No other methods should be called on the asset after calling render().

Set multiple metadata at once.

Extends BaseAsset#set_metadata

Set multiple metadata at once.

Extends BaseAsset#set_metadata

Pass an array of names of article authors. If value is a string instead of an array, it is assumed there is only one author. For the blog article format, see BlogArticle#set_author.

Marks an image as the "main" image of the article, which is marked up specially. Includes an optional caption. Compare BlogArticle.set_main_image and BlogArticle.set_main_image_caption.

(Make sure to remove this image from the HTML that you pass to NewsArticle#set_body, otherwise it will be present twice in the rendered post.)

Pass in the ID string of a "set" (category, tag). This is not the human-readable string (though it's possible for the two to be identical); you can create human-readable name for the set in your app using other tools.

(Compare BlogArticle#set_tags for the blog article format; news articles are usually in one section of the news site at a time.)

Marks the page as a "static" page so that it will show up on the main menu of an app. Use this for e.g. "About the author" or "FAQ" pages.

Name of the publication, e.g. "New York Times". This shows up in a link in the rendered news article.

Sets the lede (opening paragraph) of the news article. This paragraph is marked up specially.

(Make sure to remove this paragraph from the HTML that you pass to NewsArticle#set_body, otherwise it will be present twice in the rendered article.)

Pass the body of the article here. Note that this will be rendered mostly unchanged, inside an <article> element. The ingester itself is responsible for cleaning up this HTML such that the stylesheet applies cleanly to it.

Note that if you use NewsArticle#set_lede, you should make sure the lede is not included in value, and likewise for NewsArticle#set_main_image.

The ingester is responsible for passing cleaned-up HTML that the stylesheet's rules will apply to. In many cases this will not be necessary, but in other cases quite a lot of cleanup might be needed. Some hints to take into account:

• Figures should be inside <figure> elements, not paragraphs
• Figures with captions should also contain <figcaption> elements
• Quotes should be inside <blockquote> elements, properly formatted inside with <p> (and <cite> if applicable)
• Paragraphs should be in <p> elements, not denoted with <br> linebreaks

Here is where to customize the stylesheet. Most customizations should be able to be accomplished just by tweaking some SCSS variables which the stylesheet exposes. Here is a list of the variables:

• $primary-light-color
• $primary-dark-color
• $accent-light-color
• $accent-dark-color
• $background-light-color
• $background-dark-color
• $title-font
• $body-font
• $context-font
• $support-font

The default stylesheet is included with @import '_default';. (Note that you can also leave out this import and start from a blank stylesheet to completely do your own thing.) If you are adding rules as well as customizing variables, make sure that the variables are specified before the import, and the rules are specified after it.

Sets the HTML rendered at the bottom of the article, which can include a link back to the original source. An example might be:

Read more at <a href="http://...">Planet GNOME</a>

Use this for crediting original sources, for example.

Takes the metadata that has been set on this asset so far, and renders an HTML page internally in preparation for saving into a hatch.

You must call render() before passing this asset to Hatch#save_asset. No other methods should be called on the asset after calling render().

Set multiple metadata at once.

Extends BaseAsset#set_metadata

Set multiple metadata at once.

Extends VideoAsset#set_metadata

Set the author of this document to the given value. The author should be the "creator" of the video, being the person or organization responsible for its production.

Returns a DOM object and a NewsArticle corresponding to the URI provided. If fetching the DOM fails, the NewsArticle is marked as failed in the hatch.

Returns a DOM object and a BlogArticle corresponding to the URI provided. If fetching the DOM fails, the BlogArticle is marked as failed in the hatch.

Downloads a thumbnail image for the Youtube video specified by $embedUrl.

Downloads an image specified by the src attribute of an <img> element represented by $img. It modifies the <img> element to associate it with the image asset, which will be used later on in the ingestion process.

This is a convenience function which saves you from having to extract the image URI yourself and pass it to download_image.

Pass baseUri to handle relative links in the src attribute.

Get parsed settings to limit the ingested items.

The maxItems and maxDaysOld parameters can be overridden by the LIBINGESTER_MAX_ITEMS and LIBINGESTER_MAX_DAYS_OLD environment variables, respectively.

A utility for fetching entries from an RSS feed for further processing.

The max_items and max_days_old parameters can be overridden by the LIBINGESTER_MAX_ITEMS and LIBINGESTER_MAX_DAYS_OLD environment variables, respectively.

A utility to create a pagination function for Wordpress-generated feeds, since they are so common. Make sure to verify that your feed is generated by Wordpress before using this!

Default options for utilcleanup_body

Utility to extend utilcleanup_body defaults.

Utility to remove the article body.

The cleanup should be among the last operations done with the body. For example, if your ingester tries to make Video assets from iframes, the cleanup utility removes iframe elements by default.