User:Riviera/Podcast rss: Difference between revisions
No edit summary |
|||
(12 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
__TOC__ | |||
< | <span id="podcasts-are-rss-feeds"></span> | ||
</ | = Podcasts are RSS feeds = | ||
On Monday 2nd, we discussed podcasts as the antithesis of radio. That radio is live whereas podcasts are prerecorded and that modes of engaging with podcasts and radio differ. Podcasts are built upon Really Simple Syndication (RSS). In other words, in terms of code, there's little difference between the XML for an blog feed and the XML for a podcast feed. The following command combines a regular expression with the grep command to retrieve a list of all the XML tags in RSS feed for the ''Call Me Mother'' podcast. | <syntaxhighlight lang="fish">$ wget http://feeds.libsyn.com/330110/rss | ||
</syntaxhighlight> | |||
On Monday 2nd, we briefly discussed podcasts as the antithesis of radio. That radio is live whereas podcasts are prerecorded and that modes of engaging with podcasts and radio differ. Podcasts are built upon Really Simple Syndication (RSS). In other words, in terms of code, there's little difference between the XML for an blog feed and the XML for a podcast feed. The following command combines a regular expression with the grep command to retrieve a list of all the XML tags in RSS feed for the ''Call Me Mother'' podcast. | |||
<syntaxhighlight lang="fish">$ grep -E "<[[:alpha:]]+" rss | |||
</syntaxhighlight> | |||
This command prints results such as | This command prints results such as | ||
<syntaxhighlight lang="xml"><pubDate>Fri, 02 Apr 2021 15:25:34 GMT</pubDate> | |||
</syntaxhighlight> | |||
<syntaxhighlight lang="xml"><title>Stephen Whittle</title> | |||
</syntaxhighlight> | |||
and | |||
< | <syntaxhighlight lang="xml"><link>https://www.novel.audio</link> | ||
</ | </syntaxhighlight> | ||
These tags also appear in RSS feeds for written blogs. However, the command also prints results such as | |||
<syntaxhighlight lang="xml"><itunes:explicit>yes</itunes:explicit> | |||
</syntaxhighlight> | |||
and | and | ||
< | <syntaxhighlight lang="xml"><acast:showId>62b087ec4f1d1f0014025b79</acast:showId> | ||
</div> | </syntaxhighlight> | ||
To make two files which record the different <code>itunes</code> and <code>acast</code> tags: | |||
<syntaxhighlight lang="fish">$ grep -E "<itunes:[[:alpha:]]+>" rss > itunes.txt | |||
$ grep -E "<acast:[[:alpha:]]+>" rss > acast.txt | |||
</syntaxhighlight> | |||
Or, to create one file with both sets of tags: | |||
<syntaxhighlight lang="fish">$ grep -E "<itunes:[[:alpha:]]+>|<acast:[[:alpha:]]+>" rss > both.txt | |||
</syntaxhighlight> | |||
<span id="editing"></span> | |||
= Editing = | |||
What I want to retrieve is a list of the tags only. At the moment, I'm not interested in what's in between the tags. Ideally I'd like to use built in commands to generate a text which contains an outline of tags along the lines of the following listing. | |||
<syntaxhighlight lang="xml">1 <tag> | |||
2 <subtag> | |||
3 </subtag> | |||
4 </tag> | |||
</syntaxhighlight> | |||
<span id="first-attempt"></span> | |||
== First Attempt == | |||
Can this be achieved by making two files, <code>open.txt</code> and <code>close.txt</code>? The first file should contain all the opening tags and the latter all the closing tags. | |||
<syntaxhighlight lang="bash">$ grep -Eon "<[[:alpha:]]+>" rss > open.txt | |||
$ grep -Eon "</[[:alpha:]]+>" rss > close.txt | |||
</syntaxhighlight> | |||
It should then be possible to <code>cat</code> both files and <code>sort</code> the result by line number producing the desired outcome. | |||
<syntaxhighlight lang="shell">$ cat open.txt close.txt | sort -n > sorted.txt && cat sorted.txt | less | |||
</syntaxhighlight> | |||
<span id="second-attempt"></span> | |||
== Second Attempt == | |||
Whilst reading through the output of the first attempt (<code>sorted.txt</code>), I discovered that | |||
# the <code></guid></code> tags had no correlating opening tag | |||
# That the closing tags were sorted before the opening tags | |||
<span id="problem-1-some-information-was-missing"></span> | |||
=== Problem 1: Some information was missing === | |||
I had already deliberately omitted <code>acast</code> and <code>itunes</code> tags. In doing so I worked on the assumption that there were no other relevant, colon-separated tags in the <code>rss</code> file. Fortunately, retrieving the additional data was a quick fix: | |||
<syntaxhighlight lang="shell">$ grep -Eon "<[[:alpha:]]+>|<[[:alpha:]]+ [^z-A]*>" rss > open.txt | |||
</syntaxhighlight> | |||
Figuring out a way to sort the file such that the closing tags followed the opening tags was another matter. The general outline was there, but if I could sort out the details this could become a template for a podcast RSS generator. And that could potentially be useful in relation to Worm's sonic archive. | |||
<span id="problem-2-adding-whitespace-with-sed"></span> | |||
=== Problem 2: Adding whitespace with <code>sed</code> === | |||
The following command prepends a new line to each instance of '</' in the <code>rss</code> file. | |||
<syntaxhighlight lang="shell">sed 's/<\\//\n<\\//' rss >rss-out; | |||
</syntaxhighlight> | |||
Now it's time to write the <code>open.txt</code> and <code>close.txt</code> files. | |||
<syntaxhighlight lang="shell">$ grep -Eon "<[[:alpha:]]+>|<[[:alpha:]]+ [^z-A]*>" rss-out > open.txt | |||
$ grep -Eon "</[[:alpha:]]+>>" rss-out > close.txt | |||
</syntaxhighlight> | |||
Placing a newline before each closing tag increases the line number on which each closing tag appears. Now it's time to concatenate <code>open.txt</code> and <code>close.txt</code> and view the output. | |||
<syntaxhighlight lang="shell">$ cat open.txt close.txt | sort -n > sorted.txt && cat sorted.txt | less | |||
</syntaxhighlight> | |||
<span id="third-attempt"></span> | |||
== Third Attempt == | |||
On closer inspection of the output of sorted.txt I noticed a further lack of information. Various closing anchor tags had no corresponding opening tag. I therefore had to write a regular expression capable of matching the additional data in these tags. This might make the output of <code>sorted.txt</code> a little less readable, Although there may be a way to tidy up this information. For completeness, it would also make sense to include the <code>acast</code> and <code>itunes</code> tags. | |||
<syntaxhighlight lang="shell">$ grep -Eon "<[[:alpha:]]+>|<[[:alpha:]]+ [^z-A]*>|<[[:alpha:]]+ [[:alpha:]]+([=\":/'.;_ ]|[[:alnum:]])*>|<itunes:[[:alpha:]]+>|<acast:[[:alpha:]]+>" rss-out > open.txt | |||
$ grep -Eon "</[[:alpha:]]+>|</itunes:[[:alpha:]]+>|</acast:[[:alpha:]]+>" rss-out > close.txt | |||
</syntaxhighlight> | |||
Then, as above | |||
<syntaxhighlight lang="shell">$ cat open.txt close.txt | sort -n > sorted.txt && cat sorted.txt | less | |||
</syntaxhighlight> | |||
<span id="fourth-attempt"></span> | |||
== Fourth Attempt == | |||
I refined the regular expression. It now matches every opening and closing tag. Also, it's necessary to pass the global option to the sed substitution command. The second sed command prepends whitespace to every opening tag. This improves the structure of the output | |||
<syntaxhighlight lang="shell">sed 's/<\\//\n<\\//g' rss >rss-out | |||
sed "s/<\([[:alpha:]][^>]*>\)/\n<\1/g" rss-out > rss-out-out | |||
grep -Eon "<[[:alpha:]]([^>]*>)" rss-out-out > open.txt | |||
grep -Eon "</([^>]*>)" rss-out-out > close.txt | |||
cat open.txt close.txt | sort -n > sorted.txt | |||
grep -Eo "<([^>]*>)" sorted.txt | less > skeleton.xml | |||
chromium sorted.xml | |||
</syntaxhighlight> | |||
The webpage complains about unmatched horizontal rule tags. I deleted all of these using the query replace function in Emacs. The webpage subsequently complained about <code><br></code> and <code><br /></code> tags, so I deleted all of these too. The result is what I wanted. It views well in the browser; I am able to collapse tags for an abbreviated overview of the document. | |||
<span id="fleshing-out-the-skeleton"></span> | |||
= Fleshing out the skeleton = | |||
<code>item</code> tags are the primary constituent parts of the <code>channel</code> tag in <code>skeleton.xml</code>. However, the channel tag also contains several other tags. I have decided to design the podcast generator around this feature. This raises questions about the possible options the command might take. How does the user provide input? There are three aspects to the overall design. | |||
# creating a system for generating items | |||
# Creating a system for generating channel information | |||
# Creating a system for adding items to a document containing channel information. | |||
'''Wishlist''' | |||
* Create a system for updating channel information in the form of <code>skeleton --add foo.mp3 bar.xml</code> | |||
<span id="a-closer-look-at-items"></span> | |||
== A closer look at items == | |||
Each Item tag is made up of 15 tags. Many of these belong to the <code>itunes</code> schema. | |||
<syntaxhighlight lang="xml"><itunes:title> | |||
<itunes:duration> | |||
<itunes:explicit> | |||
<itunes:episodeType> | |||
<itunes:season> | |||
<itunes:episode> | |||
<itunes:image href="https://assets.pippa.io/shows/62b087ec4f1d1f0014025b79/show-cover.jpg"/> | |||
<itunes:summary> | |||
</syntaxhighlight> | |||
Others belong to the <code>acast</code> schema. | |||
<syntaxhighlight lang="xml"><acast:episodeId> | |||
<acast:showId> | |||
<acast:episodeUrl> | |||
<acast:settings> | |||
</syntaxhighlight> | |||
And then there's the rest. | |||
<syntaxhighlight lang="xml"><title> | |||
<pubDate> | |||
<enclosure url="https://sphinx.acast.com/p/open/s/62b087ec4f1d1f0014025b79/e/62f2226a61f23900137394a3/media.mp3" length="56168488" type="audio/mpeg"/> | |||
<guid isPermaLink="false"> | |||
<description> | |||
<link> | |||
</syntaxhighlight> | |||
I intend only to make a basic podcast generator, so I'm not going to focus on the itunes and acast schemas. | |||
<span id="draft-one-of-script"></span> | |||
== Draft one of script == | |||
<span id="general-outline"></span> | |||
=== General Outline === | |||
The following shell script defines the function <code>skeleton</code>. It takes a directory as an argument, asks the user for some information and writes an xml file. | |||
<syntaxhighlight lang="fish">#!/usr/bin/fish | |||
function skeleton -d "Generate an RSS channel for a podcast" -a directory | |||
set -l showid (uuidgen); | |||
set -l options (fish_opt -s h -l help); | |||
argparse --name='skeleton' 'h/help' -- $argv; | |||
mkdir -p /tmp/skeleton/{$showid}/e/; | |||
set -l channelfile /tmp/skeleton/{$showid}/rss.xml; | |||
# write the channel data | |||
echo '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">' > $channelfile; | |||
echo '<channel>' >> $channelfile; | |||
echo '<ttl>60</ttl>' >> $channelfile; | |||
echo '<generator>skeleton</generator>' >> $channelfile; | |||
read -l channelname -P "Channel Title: "; | |||
echo '<title>'$channelname'</title>' >> $channelfile; | |||
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile; | |||
echo '<atom:link href="hub.xpub.nl/chopchop/worm/'$showid'" ref="self" type="application/rss+xml"/>' >> $channelfile; | |||
echo '<language>en</language>' >> $channelfile; | |||
read -l channeldesc -P "Channel Description: "; | |||
echo '<description><![CDATA['$channeldesc']]></description>' >> $channelfile; | |||
echo '<image><url>https://hub.xpub.nl/chopchop/worm/'$showid'/image.jpg</url>' >> $channelfile; | |||
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile; | |||
echo '<title>'$channelname'</title>' >> $channelfile; | |||
echo '</image>' >> $channelfile; | |||
# write the item data | |||
for file in (ls $argv); | |||
set -l guid (sha256sum {$argv}/{$file} | grep -Eo "[[:alnum:]]{64}"); | |||
mkdir -p /tmp/skeleton/{$showid}/e/{$guid}; | |||
ln -s {$argv}/{$file} /tmp/skeleton/{$showid}/e/{$guid}/{$file}; | |||
set i (math $i + 1); | |||
set -l itemfile (printf '/tmp/skeleton/%s/e/%s/item-%i' $showid $guid $i); | |||
echo "<item>" > $itemfile; | |||
read -l title -P "Item $i Title: "; | |||
read -l desc -P "Item $i Description: "; | |||
echo "<title>"$title"</title>" >> $itemfile; | |||
echo "<pubDate>"(date)"</pubDate>" >> $itemfile; | |||
echo '<enclosure url="https://hub.xpub.nl/chopchop/worm/'$showid'/e/'$guid'/'$file'" length="'(soxi -D {$argv}/{$file})'" type="'(file -b --mime-type {$argv}/{$file})'"/>' >> $itemfile; | |||
sed -i 's/\\/\\/[^.]*\(\\/[^\\/]*.mp3\)/\1/' $itemfile; | |||
echo '<guid isPermaLink="false">'$guid'</guid>' >> $itemfile; | |||
set -l link (grep -Eo "https://[^.].mp3" $itemfile); | |||
echo '<link>'$link'</link>' >> $itemfile; | |||
echo '<description><![CDATA['$desc']]></description>' >> $itemfile; | |||
echo '</item>' >> $itemfile; | |||
end | |||
# concatenate items in reverse order and append to the channel file | |||
for item in (ls -t /tmp/skeleton/*/e/*/item-*); | |||
cat $item >> $channelfile; | |||
end | |||
# close the rss and channel tags | |||
echo '</channel>' >> $channelfile; | |||
echo '</rss>' >> $channelfile; | |||
end | |||
</syntaxhighlight> | |||
<span id="analysis-of-draft-one"></span> | |||
== Analysis of draft one == | |||
<span id="lines-1---6"></span> | |||
=== Lines 1 - 6 === | |||
<syntaxhighlight lang="fish">#!/usr/bin/fish | |||
function skeleton -d "Generate an RSS channel for a podcast" -a directory | |||
set -l showid (uuidgen); | |||
set -l options (fish_opt -s h -l help); | |||
argparse --name='skeleton' 'h/help' -- $argv; | |||
mkdir -p /tmp/skeleton/{$showid}/e/; | |||
</syntaxhighlight> | |||
The function <code>skeleton</code> is given a description and takes a <code>directory</code> as an argument. Next two local variables are set <code>showid</code> and <code>options</code>. The former has the value of evaluating the command <code>uuidgen</code>. The latter allows for the user to pass <code>-h</code> or <code>--help</code> to <code>skeleton</code> which will display help information. I need to figure out how to enable more options than 'help' alone. <code>argparse</code> is a command which defines how the <code>skeleton</code> command takes both options and arguments. The script next calls the command <code>mkdir</code> to create a temporary file structure. The <code>showid</code> variable expands to the value which was set in line 3. | |||
<span id="lines-7---15"></span> | |||
=== Lines 7 - 15 === | |||
<syntaxhighlight lang="fish">set -l channelfile /tmp/skeleton/{$showid}/rss.xml; | |||
# write the channel data | |||
echo '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">' > $channelfile; | |||
echo '<channel>' >> $channelfile; | |||
echo '<ttl>60</ttl>' >> $channelfile; | |||
echo '<generator>skeleton</generator>' >> $channelfile; | |||
</syntaxhighlight> | |||
Previously, the programme took a directory as an argument. Now it takes a channel as an argument. Taking a directory as an argument resembles the <code>generate</code> option. Consequently, much of the code from before can be placed within a newly written if statement. This extends to, with some adjustments, some of the lines which were omitted above. | |||
<syntaxhighlight lang="fish">read -l channelname -P "Channel Title: "; | |||
echo '<title>'$channelname'</title>' >> $channelfile; | |||
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile; | |||
</syntaxhighlight> | |||
Next, the user is prompted to provide a title for the channel. The input is stored in the variable <code>channelname</code>. <code>echo</code> is used to encase the value of <code>channelname</code> within within <code>xml</code> <code>title</code> tags. This line of code is appended to the <code>channelfile</code>. No adjustments need to be made to these three lines. | |||
<span id="line-16"></span> | |||
=== Line 16 === | |||
<div class="captioned-content"> | |||
<div class="caption"> | |||
This line of code implicitly suggests a location for the overall output of the skeleton command. | |||
</div> | </div> | ||
<syntaxhighlight lang="fish">echo '<atom:link href="hub.xpub.nl/chopchop/worm/'$showid'" ref="self" type="application/rss+xml"/>' >> $channelfile; | |||
</syntaxhighlight> | |||
</div> | </div> | ||
This line of code suggests it will eventually be necessary to create the directory <code>/media/worm/radio/$showid</code> on chopchop. Doing so would intervene in the current structure of radio worm's archive and I am hesitant about doing so for two reasons: | |||
# Space | |||
# Risk Management | |||
'''Space''' | |||
What is the output of the <code>skeleton</code> command when it takes a directory as an argument? | |||
<pre>/tmp/skeleton/ | |||
/tmp/skeleton/$showid/ | |||
/tmp/skeleton/$showid/rss.xml | |||
/tmp/skeleton/$showid/e/ | |||
/tmp/skeleton/$showid/e/$guid/ | |||
/tmp/skeleton/$showid/e/$guid/item-n | |||
/tmp/skeleton/$showid/e/$guid/symbolic-link-to.mp3 | |||
</pre> | |||
The implication of line 16 is that eventually these temporary files are moved to <code>/media/worm/radio/</code> which corresponds to the public url <code>https://hub.xpub.nl/chopchop/worm/</code>. Adding files and directories to this location will take up space on the harddrive; perhaps more space than there is available. | |||
'''Risk Management''' | |||
The shell is powerful and there's no undo function. Consequently, caution and care must be exercised when executing shell commands on files. Whereas chopchop has a backup of Radio Worm's sonic archive it would be time consuming and undesirable to have to recopy data from the original hard drive in the event of data loss. Whilst this is unlikely as long as the <code>rm</code> command is not used, things can still go wrong. For this reason, it might be better to situate the final, non-temporary location of the files at a distance from Worm's archive itself. Perhaps in the home folder of a user called 'podcasts', or on a separate hard drive. | |||
<span id="line-17"></span> | |||
=== Line 17 === | |||
<syntaxhighlight lang="fish">echo '<language>en</language>' >> $channelfile; | |||
</syntaxhighlight> | |||
I have set the channel language to English. This could be changed or made into a user defined variable. However, I will not make these changes here. | |||
<span id="lines-18---23"></span> | |||
=== lines 18 - 23 === | |||
The following lines do not need adjusting. | |||
<syntaxhighlight lang="fish">read -l channeldesc -P "Channel Description: "; | |||
echo '<description><![CDATA['$channeldesc']]></description>' >> $channelfile; | |||
echo '<image><url>https://hub.xpub.nl/chopchop/worm/'$showid'/image.jpg</url>' >> $channelfile; | |||
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile; | |||
echo '<title>'$channelname'</title>' >> $channelfile; | |||
echo '</image>' >> $channelfile; | |||
</syntaxhighlight> | |||
<span id="lines-24---43"></span> | |||
=== Lines 24 - 43 === | |||
<syntaxhighlight lang="fish"># write the item data | |||
for file in (ls $argv); | |||
</syntaxhighlight> | |||
The for loop acts upon the result of calling <code>ls</code> on the value of <code>argv</code>. Currently, the value of <code>argv</code> is supposed to be a directory full of .mp3 files. The for loop uses the name of each file to write item data in an <code>xml</code> format for each mp3 file in a given directory. | |||
<syntaxhighlight lang="fish">set -l guid (sha256sum {$argv}/{$file} | grep -Eo "[[:alnum:]]{64}"); | |||
</syntaxhighlight> | |||
Each mp3 file is assigned a <code>guid</code> which is equivalent to the <code>sha256sum</code> of the mp3 file. This was Thijs' suggestion. Piping the result of calling <code>sha256sum</code> on the full path of the mp3 file through a grep command is required to retrieve the <code>sha256sum</code> alone. This was discussed above. Notably, <code>$argv</code> is used here to prefix the full path to the value of the mp3 <code>file</code>. I intend to change the value of <code>argv</code> to a channel and consequently the way in which the value of <code>guid</code> is set will have to change. | |||
<syntaxhighlight lang="fish">mkdir -p /tmp/skeleton/{$showid}/e/{$guid}; | |||
</syntaxhighlight> | |||
In any case, the value of guid is utilised to make a temporary directory, which is okay. | |||
<syntaxhighlight lang="fish">ln -s {$argv}/{$file} /tmp/skeleton/{$showid}/e/{$guid}/{$file}; | |||
</syntaxhighlight> | |||
A symbolic link is created. The idea is that the url specified by the enclosure tag is a symbolic link to the publicly available sound files. | |||
<syntaxhighlight lang="fish">set i (math $i + 1); | |||
</syntaxhighlight> | |||
An incremental counter is set. | |||
<syntaxhighlight lang="fish">set -l itemfile (printf '/tmp/skeleton/%s/e/%s/item-%i' $showid $guid $i); | |||
echo "<item>" > $itemfile; | |||
read -l title -P "Item $i Title: "; | |||
read -l desc -P "Item $i Description: "; | |||
echo "<title>"$title"</title>" >> $itemfile; | |||
echo "<pubDate>"(date)"</pubDate>" >> $itemfile; | |||
</syntaxhighlight> | |||
Those lines can stay the same. | |||
<syntaxhighlight lang="fish">echo '<enclosure url="https://hub.xpub.nl/chopchop/worm/'$showid'/e/'$guid'/'$file'" length="'(soxi -D {$argv}/{$file})'" type="'(file -b --mime-type {$argv}/{$file})'"/>' >> $itemfile; | |||
</syntaxhighlight> | |||
That line will have to change. <code>$argv</code> will be changed to <code>$_flag_generator</code> and the url will need to correspond to the final location of the output files. | |||
<syntaxhighlight lang="fish">sed -i 's/\\/\\/[^.]*\(\\/[^\\/]*.mp3\)/\1/' $itemfile; | |||
echo '<guid isPermaLink="false">'$guid'</guid>' >> $itemfile; | |||
set -l link (grep -Eo "https://[^.].mp3" $itemfile); | |||
echo '<link>'$link'</link>' >> $itemfile; | |||
echo '<description><![CDATA['$desc']]></description>' >> $itemfile; | |||
echo '</item>' >> $itemfile; | |||
end | |||
</syntaxhighlight> | |||
A change is needed here. The second regular expression currently matches nothing due to an absent asterisk. | |||
<span id="lines-44---47"></span> | |||
=== Lines 44 - 47 === | |||
<syntaxhighlight lang="fish"># concatenate items in reverse order and append to the channel file | |||
for item in (ls -t /tmp/skeleton/*/e/*/item-*); | |||
cat $item >> $channelfile; | |||
end | |||
</syntaxhighlight> | |||
This for loop is limited by the fact that it reads every item in <code>/tmp/skeleton/*/e/*/</code>. This could become problematic if left unattended. If the files remain in their temporary location too much data will be gathered next time the command is called. However, it's not an issue if the output files are moved to their permanent location after this part of the code has been executed. | |||
<span id="lines-48---50"></span> | |||
=== Lines 48 - 50 === | |||
<syntaxhighlight lang="fish"># close the rss and channel tags | |||
echo '</channel>' >> $channelfile; | |||
echo '</rss>' >> $channelfile; | |||
</syntaxhighlight> | |||
The final few lines of the script closes open tags and terminates the command. | |||
<span id="improvements-to-draft-one"></span> | |||
== Improvements to draft one == | |||
<span id="lines-1---6-1"></span> | |||
=== Lines 1 - 6 === | |||
<div class="captioned-content"> | |||
<div class="caption"> | |||
this is a test | |||
</div> | </div> | ||
<syntaxhighlight lang="fish">#!/usr/bin/fish | |||
function skeleton -d "Generate an RSS channel for a podcast" -a channel | |||
set -l options (fish_opt -s h -l help); | |||
set options $options (fish_opt --short=g --long=generate --required-val); | |||
set options $options (fish_opt --short=a --long=add --required-val --multiple-vals); | |||
argparse $options -- $argv; | |||
or return | |||
if set -ql _flag_help | |||
echo "skeleton [ -g DIR | -a FILE | -h ] CHANNEL | |||
-h --help Display this text | |||
-g --genenerate DIR Generate an RSS feed for DIR | |||
-a --add FILE Add an item to a channel | |||
Skeleton is a command line tool for generating and writing RSS feeds for podcasts." | |||
return 0 | |||
end | |||
if set -ql _flag_generate and set -ql _flag_add; | |||
echo 'ERROR: skeleton cannot add and generate simultaneously' | |||
return 1 | |||
end | |||
if set -ql _flag_generate | |||
# echo $_flag_generate | |||
end | |||
if set -ql _flag_add | |||
# echo $_flag_add | |||
end | |||
end | |||
</syntaxhighlight> | |||
</div> | </div> | ||
--> | I have adjusted the code such that different flags create different ways of interacting with the command. I have added three flags, <code>-h</code>, <code>-g</code> and <code>-a</code>, in full <code>--help</code>, <code>--generate</code> and <code>--add</code>. These are stored in the <code>options</code> variable. The <code>add</code> flag can be passed to <code>skeleton</code> multiple times. Furthermore, <code>skeleton</code> now takes a channel as an argument. This is more flexible than taking a directory as an argument. I have made use of <code>if</code> statements to make the function do things under certain conditions. The first if statement relates to the <code>help</code> flag. If that flag is passed to <code>skeleton</code>, then text detailing how to use the command is displayed. The second if statement prevents the user from adding items to rss feeds and generating rss feeds simultaneously. <code>--add</code> and <code>--generate</code> become mutually exclusive options. The remaining if statements control what the programme does when the <code>generate</code> and <code>add</code> flags are passed to <code>skeleton</code>. Some lines have been omitted because it makes sense to place them elsewhere following these changes. | ||
<span id="lines-7---15-1"></span> | |||
=== Lines 7 - 15 === | |||
<syntaxhighlight lang="fish">set -l showid (uuidgen); | |||
mkdir -p /tmp/skeleton/{$showid}; | |||
</syntaxhighlight> | |||
Due to the resemblance described above the remainder of the script (lines 7 - 50) will be placed inside the third if statement. The code goes on to create a temporary directory using the value of the <code>showid</code> variable. | |||
<syntaxhighlight lang="fish">touch /tmp/skeleton/index | |||
echo $_flag_generate $showid >> /tmp/skeleton/index | |||
</syntaxhighlight> | |||
Somehow, when using the <code>add</code> flag, <code>skeleton</code> will need to retrieve the value of <code>showid</code> for a given channel. This will ensure coherent and consistent output. I'm going to write a file which pairs the <code>showid</code> with the value of <code>flag_generate</code>. The code for the <code>add</code> flag will consult this file when it sets the value of <code>showid</code>. | |||
<syntaxhighlight lang="fish">set -l channelfile /tmp/skeleton/{$showid}/rss.xml; | |||
# write the channel data | |||
echo '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">' > $channelfile; | |||
echo '<channel>' >> $channelfile; | |||
echo '<ttl>60</ttl>' >> $channelfile; | |||
echo '<generator>skeleton</generator>' >> $channelfile; | |||
</syntaxhighlight> | |||
A <code>channelfile</code> is then set. Redirecting the output of <code>echo</code> to the value of <code>channelfile</code> creates the file and stores the output there. | |||
<span id="line-16-1"></span> | |||
=== Line 16 === | |||
<syntaxhighlight lang="fish">echo '<atom:link href="hub.xpub.nl/chopchop/river/public_html/podcasts/'$showid'" ref="self" type="application/rss+xml"/>' >> $channelfile; | |||
</syntaxhighlight> | |||
It's evident that line 16 is bound up with the final location of the generated files. I'll eventually place the <code>rss.xml</code> file in <code>~/public_html/podcasts/$showid/</code>. Therefore line 16 can be changed to the above. | |||
<span id="lines-24---43-1"></span> | |||
=== Lines 24 - 43 === | |||
<syntaxhighlight lang="fish">for file in (ls $_flag_generate); | |||
set -l guid (sha256sum {$_flag_generate}/{$file} | grep -Eo "[[:alnum:]]{64}"); | |||
mkdir -p /tmp/skeleton/{$showid}/e/{$guid}; | |||
ln -s {$argv}/{$file} /tmp/skeleton/{$showid}/e/{$guid}/{$file}; | |||
set i (math $i + 1); | |||
set -l itemfile (printf '/tmp/skeleton/%s/e/%s/item-%i' $showid $guid $i); | |||
echo "<item>" > $itemfile; | |||
read -l title -P "Item $i Title: "; | |||
read -l desc -P "Item $i Description: "; | |||
echo "<title>"$title"</title>" >> $itemfile; | |||
echo "<pubDate>"(date)"</pubDate>" >> $itemfile; | |||
echo '<enclosure url="https://hub.xpub.nl/chopchop/river/public_html/podcasts/'$showid'/e/'$guid'/'$file'" length="'(soxi -D {$_flag_generate}/{$file})'" type="'(file -b --mime-type {$_flag_generate}/{$file})'"/>' >> $itemfile; | |||
sed -i 's/\\/\\/[^.]*\(\\/[^\\/]*.mp3\)/\1/' $itemfile; | |||
echo '<guid isPermaLink="false">'$guid'</guid>' >> $itemfile; | |||
set -l link "https://hub.xpub.nl/chopchop/river/public_html/podcasts/$showid/e/$guid/$file"; | |||
echo '<link>'$link'</link>' >> $itemfile; | |||
echo '<description><![CDATA['$desc']]></description>' >> $itemfile; | |||
echo '</item>' >> $itemfile; | |||
</syntaxhighlight> | |||
<span id="removing-temporary-files"></span> | |||
=== (Re)Moving temporary files === | |||
<div class="captioned-content"> | |||
<div class="caption"> | |||
To avoid errors and save the temporary files, it's important to move them to a different location. | |||
</div> | |||
<syntaxhighlight lang="fish">mkdir -p ~/public_html/podcasts/ | |||
mv /tmp/skeleton/* ~/public_html/podcasts/ | |||
touch ~/public_html/podcasts/index; | |||
cat /tmp/skeleton/index >> ~/public_html/podcasts/index; | |||
rm -r /tmp/skeleton/; | |||
</syntaxhighlight> | |||
</div> | </div> | ||
<span id="repositioning-lines-7---50"></span> | |||
=== Repositioning lines 7 - 50 === | |||
<div class="captioned-content"> | |||
<div class="caption"> | |||
Having made these changes, lines 7 - 50 should be placed within the third if statement in the rewriting of lines 1 - 6. | |||
</div> | |||
<syntaxhighlight lang="fish">if set -ql _flag_generate | |||
... | |||
end | |||
</syntaxhighlight> | |||
</div> | </div> | ||
<span id="draft-two-of-script"></span> | |||
== Draft two of script == | |||
<syntaxhighlight lang="fish">#!/usr/bin/fish | |||
function skelegen -d "Generate an RSS channel for a podcast" | |||
set -l options (fish_opt -s h -l help); | |||
set options $options (fish_opt --short=g --long=generate --required-val); | |||
set options $options (fish_opt --short=a --long=add --required-val --multiple-vals); | |||
argparse $options -- $argv; | |||
or return | |||
if set -ql _flag_help | |||
echo "skelegen [ -g DIR | -a FILE | -h ] CHANNEL | |||
-h --help Display this text | |||
-g --genenerate DIR Generate an RSS feed for DIR | |||
-a --add FILE Add an item to a channel | |||
Skelegen is a command line tool for generating and writing RSS feeds for podcasts." | |||
return 0 | |||
end | |||
if set -ql _flag_generate and set -ql _flag_add; | |||
echo 'ERROR: skelegen cannot add and generate simultaneously' | |||
return 1 | |||
end | |||
if set -ql _flag_generate | |||
set -l showid (uuidgen); | |||
mkdir -p /tmp/skeleton/{$showid}; | |||
touch /tmp/skeleton/index; | |||
echo $_flag_generate $showid >> /tmp/skeleton/index; | |||
set -l channelfile /tmp/skeleton/{$showid}/rss.xml; | |||
# write the channel data | |||
echo '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">' > $channelfile; | |||
echo '<channel>' >> $channelfile; | |||
echo '<ttl>60</ttl>' >> $channelfile; | |||
echo '<generator>skeleton</generator>' >> $channelfile; | |||
read -l channelname -P "Channel Title: "; | |||
echo '<title>'$channelname'</title>' >> $channelfile; | |||
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile; | |||
echo '<atom:link href="hub.xpub.nl/chopchop/river/public_html/podcasts/'$showid'" ref="self" type="application/rss+xml"/>' >> $channelfile; | |||
echo '<language>en</language>' >> $channelfile; | |||
read -l channeldesc -P "Channel Description: "; | |||
echo '<description><![CDATA['$channeldesc']]></description>' >> $channelfile; | |||
echo '<image><url>https://hub.xpub.nl/chopchop/worm/'$showid'/image.jpg</url>' >> $channelfile; | |||
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile; | |||
echo '<title>'$channelname'</title>' >> $channelfile; | |||
echo '</image>' >> $channelfile; | |||
for file in (ls $_flag_generate); | |||
set -l guid (sha256sum {$_flag_generate}/{$file} | grep -Eo "[[:alnum:]]{64}"); | |||
mkdir -p /tmp/skeleton/{$showid}/e/{$guid}; | |||
ln -s {$_flag_generate}/{$file} /tmp/skeleton/{$showid}/e/{$guid}/{$file}; | |||
set i (math $i + 1); | |||
set -l itemfile (printf '/tmp/skeleton/%s/e/%s/item-%i' $showid $guid $i); | |||
echo "<item>" > $itemfile; | |||
read -l title -P "Item $i Title: "; | |||
read -l desc -P "Item $i Description: "; | |||
echo "<title>"$title"</title>" >> $itemfile; | |||
echo "<pubDate>"(date)"</pubDate>" >> $itemfile; | |||
echo '<enclosure url="https://hub.xpub.nl/chopchop/river/public_html/podcasts/'$showid'/e/'$guid'/'$file'" length="'(soxi -D {$_flag_generate}/{$file})'" type="'(file -b --mime-type {$_flag_generate}/{$file})'"/>' >> $itemfile; | |||
sed -i 's/\\/\\/[^.]*\(\\/[^\\/]*.mp3\)/\1/' $itemfile; | |||
echo '<guid isPermaLink="false">'$guid'</guid>' >> $itemfile; | |||
set -l link "https://hub.xpub.nl/chopchop/river/public_html/podcasts/$showid/e/$guid/$file"; | |||
echo '<link>'$link'</link>' >> $itemfile; | |||
echo '<description><![CDATA['$desc']]></description>' >> $itemfile; | |||
echo '</item>' >> $itemfile; | |||
end | |||
# concatenate items in reverse order and append to the channel file | |||
for item in (ls -t /tmp/skeleton/*/e/*/item-*); | |||
cat $item >> $channelfile; | |||
end | |||
# close the rss and channel tags | |||
echo '</channel>' >> $channelfile; | |||
echo '</rss>' >> $channelfile; | |||
# (re)move temporary files | |||
mkdir -p ~/public_html/podcasts/{$showid}; | |||
mv /tmp/skeleton/{$showid}/* ~/public_html/podcasts/{$showid}/; | |||
touch ~/public_html/podcasts/index; | |||
cat /tmp/skeleton/index >> ~/public_html/podcasts/index; | |||
rm -r /tmp/skeleton/; | |||
end | |||
if set -ql _flag_add | |||
# echo $_flag_add | |||
end | |||
end | |||
</syntaxhighlight> | |||
<span id="draft-three-of-script"></span> | |||
== Draft three of script == | |||
<syntaxhighlight lang="fish">#!/usr/bin/fish | |||
function skelegen -d "Generate an RSS channel for a podcast" | |||
set -l options (fish_opt -s h -l help); | |||
set options $options (fish_opt --short=g --long=generate --required-val); | |||
set options $options (fish_opt --short=a --long=add --required-val --multiple-vals); | |||
set options $options (fish_opt --short=c --long=channel --required-val); | |||
argparse $options -- $argv; | |||
or return | |||
if set -ql _flag_help | |||
echo "skelegen [ -g DIR | -a FILE | -h ] -c XML_FILE | |||
-h --help Display this text | |||
-g --genenerate DIR Generate an RSS feed for DIR | |||
-a --add FILE Add an item to a channel | |||
-c --channel XML_FILE Specify an output channel | |||
Skelegen is a command line application for generating and writing RSS feeds for podcasts. Skelegen is capable of generating an RSS feed for a directory of audio recordings. It is also possible to add audio recordings to a channel. | |||
EXAMPLE USAGE | |||
skelegen --generate ~/Music/podcast --channel ~/public_html/podcasts/mypodcast.xml | |||
skelegen --add ~/Music/podcast/episode.mp3 --chamnel ~/public_html/podcasts/myotherpodcast.xml" | |||
end | |||
if set -ql _flag_generate and set -ql _flag_add; | |||
echo 'ERROR: skelegen cannot add and generate simultaneously'; | |||
return 1; | |||
end | |||
if not set -ql _flag_channel; | |||
echo "ERROR: channel must be specified."; | |||
return 1; | |||
end | |||
if set -ql _flag_channel; | |||
set -l channelprefix (echo $_flag_channel | sed -E 's/\\/([^\\/]*.xml)/\\/\n\1/' | grep -Ev "[^/]*.xml"); | |||
mkdir -p $channelprefix; | |||
end | |||
#the < | if set -ql _flag_generate | ||
set -l showid (uuidgen); | |||
set -l channelprefix (echo $_flag_channel | sed -E 's/\\/([^\\/]*.xml)/\\/\n\1/' | grep -Ev "[^/]*.xml"); | |||
set -l publicprefix (echo $channelprefix | sed -E 's/\\/home\\/riviera\\/public_html\\/([^\\/]+\\/)/https:\\/\\/hub.xpub.nl\\/chopchop\\/~river\\/\1/'); | |||
set -l channelfile (echo $_flag_channel | sed -E 's/\\/([^\\/]*.xml)/\\/\n\1/' | grep -Eo "[^/]*.xml"); | |||
mkdir -p {$channelprefix}{$showid}; | |||
set -l channel {$channelprefix}{$showid}/{$channelfile}; | |||
# # write the channel data | |||
echo '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">' > $channel; | |||
echo '<channel>' >> $channel; | |||
echo '<ttl>60</ttl>' >> $channel; | |||
echo '<generator>skeleton</generator>' >> $channel; | |||
read -l channelname -P "Channel Title: "; | |||
echo '<title>'$channelname'</title>' >> $channel; | |||
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channel; | |||
echo '<atom:link href="'$publicprefix$showid'" ref="self" type="application/rss+xml"/>' >> $channel; | |||
echo '<language>en</language>' >> $channel; | |||
< | read -l channeldesc -P "Channel Description: "; | ||
echo '<description><![CDATA['$channeldesc']]></description>' >> $channel; | |||
echo '<image><url>'{$publicprefix}{$showid}'/image.jpg</url>' >> $channel; | |||
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channel; | |||
echo '<title>'$channelname'</title>' >> $channel; | |||
echo '</image>' >> $channel; | |||
for file in (ls $_flag_generate); | |||
set -l guid (sha256sum {$_flag_generate}/{$file} | grep -Eo "[[:alnum:]]{64}"); | |||
mkdir -p {$channelprefix}{$showid}/e/{$guid}; | |||
ln -s {$_flag_generate}/{$file} {$channelprefix}{$showid}/e/{$guid}/{$file}; | |||
set i (math $i + 1); | |||
set -l itemfile (printf '%s%s/e/%s/item-%i' $channelprefix $showid $guid $i); | |||
echo "<item>" > $itemfile; | |||
read -l title -P "Item $i Title: "; | |||
read -l desc -P "Item $i Description: "; | |||
echo "<title>"$title"</title>" >> $itemfile; | |||
echo "<pubDate>"(date)"</pubDate>" >> $itemfile; | |||
echo '<enclosure url="'$publicprefix$showid'/e/'$guid'/'$file'" length="'(soxi -D {$_flag_generate}/{$file})'" type="'(file -b --mime-type {$_flag_generate}/{$file})'"/>' >> $itemfile; | |||
sed -i 's/\\/\\/[^.]*\(\\/[^\\/]*.mp3\)/\1/' $itemfile; | |||
echo '<guid isPermaLink="false">'$guid'</guid>' >> $itemfile; | |||
set -l link "https://hub.xpub.nl/chopchop/~river/podcasts/$showid/e/$guid/$file"; | |||
echo '<link>'$link'</link>' >> $itemfile; | |||
echo '<description><![CDATA['$desc']]></description>' >> $itemfile; | |||
echo '</item>' >> $itemfile; | |||
end | |||
# concatenate items in reverse order and append to the channel file | |||
for item in (ls -t {$channelprefix}*/e/*/item-*); | |||
cat $item >> $channel; | |||
end | |||
# close the rss and channel tags | |||
echo '</channel>' >> $channel; | |||
echo '</rss>' >> $channel; | |||
mkdir -p ~/.skeleton; | |||
touch ~/.skeleton/channellist; | |||
echo $channel >> ~/.skeleton/channellist; | |||
end | |||
if set -ql _flag_add | |||
... | |||
end | |||
end | |||
</syntaxhighlight> | |||
<span id="the-add-flag"></span> | |||
== The add flag == |
Latest revision as of 08:23, 9 October 2023
Podcasts are RSS feeds
$ wget http://feeds.libsyn.com/330110/rss
On Monday 2nd, we briefly discussed podcasts as the antithesis of radio. That radio is live whereas podcasts are prerecorded and that modes of engaging with podcasts and radio differ. Podcasts are built upon Really Simple Syndication (RSS). In other words, in terms of code, there's little difference between the XML for an blog feed and the XML for a podcast feed. The following command combines a regular expression with the grep command to retrieve a list of all the XML tags in RSS feed for the Call Me Mother podcast.
$ grep -E "<[[:alpha:]]+" rss
This command prints results such as
<pubDate>Fri, 02 Apr 2021 15:25:34 GMT</pubDate>
<title>Stephen Whittle</title>
and
<link>https://www.novel.audio</link>
These tags also appear in RSS feeds for written blogs. However, the command also prints results such as
<itunes:explicit>yes</itunes:explicit>
and
<acast:showId>62b087ec4f1d1f0014025b79</acast:showId>
To make two files which record the different itunes
and acast
tags:
$ grep -E "<itunes:[[:alpha:]]+>" rss > itunes.txt
$ grep -E "<acast:[[:alpha:]]+>" rss > acast.txt
Or, to create one file with both sets of tags:
$ grep -E "<itunes:[[:alpha:]]+>|<acast:[[:alpha:]]+>" rss > both.txt
Editing
What I want to retrieve is a list of the tags only. At the moment, I'm not interested in what's in between the tags. Ideally I'd like to use built in commands to generate a text which contains an outline of tags along the lines of the following listing.
1 <tag>
2 <subtag>
3 </subtag>
4 </tag>
First Attempt
Can this be achieved by making two files, open.txt
and close.txt
? The first file should contain all the opening tags and the latter all the closing tags.
$ grep -Eon "<[[:alpha:]]+>" rss > open.txt
$ grep -Eon "</[[:alpha:]]+>" rss > close.txt
It should then be possible to cat
both files and sort
the result by line number producing the desired outcome.
$ cat open.txt close.txt | sort -n > sorted.txt && cat sorted.txt | less
Second Attempt
Whilst reading through the output of the first attempt (sorted.txt
), I discovered that
- the
</guid>
tags had no correlating opening tag - That the closing tags were sorted before the opening tags
Problem 1: Some information was missing
I had already deliberately omitted acast
and itunes
tags. In doing so I worked on the assumption that there were no other relevant, colon-separated tags in the rss
file. Fortunately, retrieving the additional data was a quick fix:
$ grep -Eon "<[[:alpha:]]+>|<[[:alpha:]]+ [^z-A]*>" rss > open.txt
Figuring out a way to sort the file such that the closing tags followed the opening tags was another matter. The general outline was there, but if I could sort out the details this could become a template for a podcast RSS generator. And that could potentially be useful in relation to Worm's sonic archive.
Problem 2: Adding whitespace with sed
The following command prepends a new line to each instance of '</' in the rss
file.
sed 's/<\\//\n<\\//' rss >rss-out;
Now it's time to write the open.txt
and close.txt
files.
$ grep -Eon "<[[:alpha:]]+>|<[[:alpha:]]+ [^z-A]*>" rss-out > open.txt
$ grep -Eon "</[[:alpha:]]+>>" rss-out > close.txt
Placing a newline before each closing tag increases the line number on which each closing tag appears. Now it's time to concatenate open.txt
and close.txt
and view the output.
$ cat open.txt close.txt | sort -n > sorted.txt && cat sorted.txt | less
Third Attempt
On closer inspection of the output of sorted.txt I noticed a further lack of information. Various closing anchor tags had no corresponding opening tag. I therefore had to write a regular expression capable of matching the additional data in these tags. This might make the output of sorted.txt
a little less readable, Although there may be a way to tidy up this information. For completeness, it would also make sense to include the acast
and itunes
tags.
$ grep -Eon "<[[:alpha:]]+>|<[[:alpha:]]+ [^z-A]*>|<[[:alpha:]]+ [[:alpha:]]+([=\":/'.;_ ]|[[:alnum:]])*>|<itunes:[[:alpha:]]+>|<acast:[[:alpha:]]+>" rss-out > open.txt
$ grep -Eon "</[[:alpha:]]+>|</itunes:[[:alpha:]]+>|</acast:[[:alpha:]]+>" rss-out > close.txt
Then, as above
$ cat open.txt close.txt | sort -n > sorted.txt && cat sorted.txt | less
Fourth Attempt
I refined the regular expression. It now matches every opening and closing tag. Also, it's necessary to pass the global option to the sed substitution command. The second sed command prepends whitespace to every opening tag. This improves the structure of the output
sed 's/<\\//\n<\\//g' rss >rss-out
sed "s/<\([[:alpha:]][^>]*>\)/\n<\1/g" rss-out > rss-out-out
grep -Eon "<[[:alpha:]]([^>]*>)" rss-out-out > open.txt
grep -Eon "</([^>]*>)" rss-out-out > close.txt
cat open.txt close.txt | sort -n > sorted.txt
grep -Eo "<([^>]*>)" sorted.txt | less > skeleton.xml
chromium sorted.xml
The webpage complains about unmatched horizontal rule tags. I deleted all of these using the query replace function in Emacs. The webpage subsequently complained about <br>
and <br />
tags, so I deleted all of these too. The result is what I wanted. It views well in the browser; I am able to collapse tags for an abbreviated overview of the document.
Fleshing out the skeleton
item
tags are the primary constituent parts of the channel
tag in skeleton.xml
. However, the channel tag also contains several other tags. I have decided to design the podcast generator around this feature. This raises questions about the possible options the command might take. How does the user provide input? There are three aspects to the overall design.
- creating a system for generating items
- Creating a system for generating channel information
- Creating a system for adding items to a document containing channel information.
Wishlist
- Create a system for updating channel information in the form of
skeleton --add foo.mp3 bar.xml
A closer look at items
Each Item tag is made up of 15 tags. Many of these belong to the itunes
schema.
<itunes:title>
<itunes:duration>
<itunes:explicit>
<itunes:episodeType>
<itunes:season>
<itunes:episode>
<itunes:image href="https://assets.pippa.io/shows/62b087ec4f1d1f0014025b79/show-cover.jpg"/>
<itunes:summary>
Others belong to the acast
schema.
<acast:episodeId>
<acast:showId>
<acast:episodeUrl>
<acast:settings>
And then there's the rest.
<title>
<pubDate>
<enclosure url="https://sphinx.acast.com/p/open/s/62b087ec4f1d1f0014025b79/e/62f2226a61f23900137394a3/media.mp3" length="56168488" type="audio/mpeg"/>
<guid isPermaLink="false">
<description>
<link>
I intend only to make a basic podcast generator, so I'm not going to focus on the itunes and acast schemas.
Draft one of script
General Outline
The following shell script defines the function skeleton
. It takes a directory as an argument, asks the user for some information and writes an xml file.
#!/usr/bin/fish
function skeleton -d "Generate an RSS channel for a podcast" -a directory
set -l showid (uuidgen);
set -l options (fish_opt -s h -l help);
argparse --name='skeleton' 'h/help' -- $argv;
mkdir -p /tmp/skeleton/{$showid}/e/;
set -l channelfile /tmp/skeleton/{$showid}/rss.xml;
# write the channel data
echo '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">' > $channelfile;
echo '<channel>' >> $channelfile;
echo '<ttl>60</ttl>' >> $channelfile;
echo '<generator>skeleton</generator>' >> $channelfile;
read -l channelname -P "Channel Title: ";
echo '<title>'$channelname'</title>' >> $channelfile;
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile;
echo '<atom:link href="hub.xpub.nl/chopchop/worm/'$showid'" ref="self" type="application/rss+xml"/>' >> $channelfile;
echo '<language>en</language>' >> $channelfile;
read -l channeldesc -P "Channel Description: ";
echo '<description><![CDATA['$channeldesc']]></description>' >> $channelfile;
echo '<image><url>https://hub.xpub.nl/chopchop/worm/'$showid'/image.jpg</url>' >> $channelfile;
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile;
echo '<title>'$channelname'</title>' >> $channelfile;
echo '</image>' >> $channelfile;
# write the item data
for file in (ls $argv);
set -l guid (sha256sum {$argv}/{$file} | grep -Eo "[[:alnum:]]{64}");
mkdir -p /tmp/skeleton/{$showid}/e/{$guid};
ln -s {$argv}/{$file} /tmp/skeleton/{$showid}/e/{$guid}/{$file};
set i (math $i + 1);
set -l itemfile (printf '/tmp/skeleton/%s/e/%s/item-%i' $showid $guid $i);
echo "<item>" > $itemfile;
read -l title -P "Item $i Title: ";
read -l desc -P "Item $i Description: ";
echo "<title>"$title"</title>" >> $itemfile;
echo "<pubDate>"(date)"</pubDate>" >> $itemfile;
echo '<enclosure url="https://hub.xpub.nl/chopchop/worm/'$showid'/e/'$guid'/'$file'" length="'(soxi -D {$argv}/{$file})'" type="'(file -b --mime-type {$argv}/{$file})'"/>' >> $itemfile;
sed -i 's/\\/\\/[^.]*\(\\/[^\\/]*.mp3\)/\1/' $itemfile;
echo '<guid isPermaLink="false">'$guid'</guid>' >> $itemfile;
set -l link (grep -Eo "https://[^.].mp3" $itemfile);
echo '<link>'$link'</link>' >> $itemfile;
echo '<description><![CDATA['$desc']]></description>' >> $itemfile;
echo '</item>' >> $itemfile;
end
# concatenate items in reverse order and append to the channel file
for item in (ls -t /tmp/skeleton/*/e/*/item-*);
cat $item >> $channelfile;
end
# close the rss and channel tags
echo '</channel>' >> $channelfile;
echo '</rss>' >> $channelfile;
end
Analysis of draft one
Lines 1 - 6
#!/usr/bin/fish
function skeleton -d "Generate an RSS channel for a podcast" -a directory
set -l showid (uuidgen);
set -l options (fish_opt -s h -l help);
argparse --name='skeleton' 'h/help' -- $argv;
mkdir -p /tmp/skeleton/{$showid}/e/;
The function skeleton
is given a description and takes a directory
as an argument. Next two local variables are set showid
and options
. The former has the value of evaluating the command uuidgen
. The latter allows for the user to pass -h
or --help
to skeleton
which will display help information. I need to figure out how to enable more options than 'help' alone. argparse
is a command which defines how the skeleton
command takes both options and arguments. The script next calls the command mkdir
to create a temporary file structure. The showid
variable expands to the value which was set in line 3.
Lines 7 - 15
set -l channelfile /tmp/skeleton/{$showid}/rss.xml;
# write the channel data
echo '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">' > $channelfile;
echo '<channel>' >> $channelfile;
echo '<ttl>60</ttl>' >> $channelfile;
echo '<generator>skeleton</generator>' >> $channelfile;
Previously, the programme took a directory as an argument. Now it takes a channel as an argument. Taking a directory as an argument resembles the generate
option. Consequently, much of the code from before can be placed within a newly written if statement. This extends to, with some adjustments, some of the lines which were omitted above.
read -l channelname -P "Channel Title: ";
echo '<title>'$channelname'</title>' >> $channelfile;
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile;
Next, the user is prompted to provide a title for the channel. The input is stored in the variable channelname
. echo
is used to encase the value of channelname
within within xml
title
tags. This line of code is appended to the channelfile
. No adjustments need to be made to these three lines.
Line 16
This line of code suggests it will eventually be necessary to create the directory /media/worm/radio/$showid
on chopchop. Doing so would intervene in the current structure of radio worm's archive and I am hesitant about doing so for two reasons:
- Space
- Risk Management
Space
What is the output of the skeleton
command when it takes a directory as an argument?
/tmp/skeleton/ /tmp/skeleton/$showid/ /tmp/skeleton/$showid/rss.xml /tmp/skeleton/$showid/e/ /tmp/skeleton/$showid/e/$guid/ /tmp/skeleton/$showid/e/$guid/item-n /tmp/skeleton/$showid/e/$guid/symbolic-link-to.mp3
The implication of line 16 is that eventually these temporary files are moved to /media/worm/radio/
which corresponds to the public url https://hub.xpub.nl/chopchop/worm/
. Adding files and directories to this location will take up space on the harddrive; perhaps more space than there is available.
Risk Management
The shell is powerful and there's no undo function. Consequently, caution and care must be exercised when executing shell commands on files. Whereas chopchop has a backup of Radio Worm's sonic archive it would be time consuming and undesirable to have to recopy data from the original hard drive in the event of data loss. Whilst this is unlikely as long as the rm
command is not used, things can still go wrong. For this reason, it might be better to situate the final, non-temporary location of the files at a distance from Worm's archive itself. Perhaps in the home folder of a user called 'podcasts', or on a separate hard drive.
Line 17
echo '<language>en</language>' >> $channelfile;
I have set the channel language to English. This could be changed or made into a user defined variable. However, I will not make these changes here.
lines 18 - 23
The following lines do not need adjusting.
read -l channeldesc -P "Channel Description: ";
echo '<description><![CDATA['$channeldesc']]></description>' >> $channelfile;
echo '<image><url>https://hub.xpub.nl/chopchop/worm/'$showid'/image.jpg</url>' >> $channelfile;
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile;
echo '<title>'$channelname'</title>' >> $channelfile;
echo '</image>' >> $channelfile;
Lines 24 - 43
# write the item data
for file in (ls $argv);
The for loop acts upon the result of calling ls
on the value of argv
. Currently, the value of argv
is supposed to be a directory full of .mp3 files. The for loop uses the name of each file to write item data in an xml
format for each mp3 file in a given directory.
set -l guid (sha256sum {$argv}/{$file} | grep -Eo "[[:alnum:]]{64}");
Each mp3 file is assigned a guid
which is equivalent to the sha256sum
of the mp3 file. This was Thijs' suggestion. Piping the result of calling sha256sum
on the full path of the mp3 file through a grep command is required to retrieve the sha256sum
alone. This was discussed above. Notably, $argv
is used here to prefix the full path to the value of the mp3 file
. I intend to change the value of argv
to a channel and consequently the way in which the value of guid
is set will have to change.
mkdir -p /tmp/skeleton/{$showid}/e/{$guid};
In any case, the value of guid is utilised to make a temporary directory, which is okay.
ln -s {$argv}/{$file} /tmp/skeleton/{$showid}/e/{$guid}/{$file};
A symbolic link is created. The idea is that the url specified by the enclosure tag is a symbolic link to the publicly available sound files.
set i (math $i + 1);
An incremental counter is set.
set -l itemfile (printf '/tmp/skeleton/%s/e/%s/item-%i' $showid $guid $i);
echo "<item>" > $itemfile;
read -l title -P "Item $i Title: ";
read -l desc -P "Item $i Description: ";
echo "<title>"$title"</title>" >> $itemfile;
echo "<pubDate>"(date)"</pubDate>" >> $itemfile;
Those lines can stay the same.
echo '<enclosure url="https://hub.xpub.nl/chopchop/worm/'$showid'/e/'$guid'/'$file'" length="'(soxi -D {$argv}/{$file})'" type="'(file -b --mime-type {$argv}/{$file})'"/>' >> $itemfile;
That line will have to change. $argv
will be changed to $_flag_generator
and the url will need to correspond to the final location of the output files.
sed -i 's/\\/\\/[^.]*\(\\/[^\\/]*.mp3\)/\1/' $itemfile;
echo '<guid isPermaLink="false">'$guid'</guid>' >> $itemfile;
set -l link (grep -Eo "https://[^.].mp3" $itemfile);
echo '<link>'$link'</link>' >> $itemfile;
echo '<description><![CDATA['$desc']]></description>' >> $itemfile;
echo '</item>' >> $itemfile;
end
A change is needed here. The second regular expression currently matches nothing due to an absent asterisk.
Lines 44 - 47
# concatenate items in reverse order and append to the channel file
for item in (ls -t /tmp/skeleton/*/e/*/item-*);
cat $item >> $channelfile;
end
This for loop is limited by the fact that it reads every item in /tmp/skeleton/*/e/*/
. This could become problematic if left unattended. If the files remain in their temporary location too much data will be gathered next time the command is called. However, it's not an issue if the output files are moved to their permanent location after this part of the code has been executed.
Lines 48 - 50
# close the rss and channel tags
echo '</channel>' >> $channelfile;
echo '</rss>' >> $channelfile;
The final few lines of the script closes open tags and terminates the command.
Improvements to draft one
Lines 1 - 6
I have adjusted the code such that different flags create different ways of interacting with the command. I have added three flags, -h
, -g
and -a
, in full --help
, --generate
and --add
. These are stored in the options
variable. The add
flag can be passed to skeleton
multiple times. Furthermore, skeleton
now takes a channel as an argument. This is more flexible than taking a directory as an argument. I have made use of if
statements to make the function do things under certain conditions. The first if statement relates to the help
flag. If that flag is passed to skeleton
, then text detailing how to use the command is displayed. The second if statement prevents the user from adding items to rss feeds and generating rss feeds simultaneously. --add
and --generate
become mutually exclusive options. The remaining if statements control what the programme does when the generate
and add
flags are passed to skeleton
. Some lines have been omitted because it makes sense to place them elsewhere following these changes.
Lines 7 - 15
set -l showid (uuidgen);
mkdir -p /tmp/skeleton/{$showid};
Due to the resemblance described above the remainder of the script (lines 7 - 50) will be placed inside the third if statement. The code goes on to create a temporary directory using the value of the showid
variable.
touch /tmp/skeleton/index
echo $_flag_generate $showid >> /tmp/skeleton/index
Somehow, when using the add
flag, skeleton
will need to retrieve the value of showid
for a given channel. This will ensure coherent and consistent output. I'm going to write a file which pairs the showid
with the value of flag_generate
. The code for the add
flag will consult this file when it sets the value of showid
.
set -l channelfile /tmp/skeleton/{$showid}/rss.xml;
# write the channel data
echo '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">' > $channelfile;
echo '<channel>' >> $channelfile;
echo '<ttl>60</ttl>' >> $channelfile;
echo '<generator>skeleton</generator>' >> $channelfile;
A channelfile
is then set. Redirecting the output of echo
to the value of channelfile
creates the file and stores the output there.
Line 16
echo '<atom:link href="hub.xpub.nl/chopchop/river/public_html/podcasts/'$showid'" ref="self" type="application/rss+xml"/>' >> $channelfile;
It's evident that line 16 is bound up with the final location of the generated files. I'll eventually place the rss.xml
file in ~/public_html/podcasts/$showid/
. Therefore line 16 can be changed to the above.
Lines 24 - 43
for file in (ls $_flag_generate);
set -l guid (sha256sum {$_flag_generate}/{$file} | grep -Eo "[[:alnum:]]{64}");
mkdir -p /tmp/skeleton/{$showid}/e/{$guid};
ln -s {$argv}/{$file} /tmp/skeleton/{$showid}/e/{$guid}/{$file};
set i (math $i + 1);
set -l itemfile (printf '/tmp/skeleton/%s/e/%s/item-%i' $showid $guid $i);
echo "<item>" > $itemfile;
read -l title -P "Item $i Title: ";
read -l desc -P "Item $i Description: ";
echo "<title>"$title"</title>" >> $itemfile;
echo "<pubDate>"(date)"</pubDate>" >> $itemfile;
echo '<enclosure url="https://hub.xpub.nl/chopchop/river/public_html/podcasts/'$showid'/e/'$guid'/'$file'" length="'(soxi -D {$_flag_generate}/{$file})'" type="'(file -b --mime-type {$_flag_generate}/{$file})'"/>' >> $itemfile;
sed -i 's/\\/\\/[^.]*\(\\/[^\\/]*.mp3\)/\1/' $itemfile;
echo '<guid isPermaLink="false">'$guid'</guid>' >> $itemfile;
set -l link "https://hub.xpub.nl/chopchop/river/public_html/podcasts/$showid/e/$guid/$file";
echo '<link>'$link'</link>' >> $itemfile;
echo '<description><![CDATA['$desc']]></description>' >> $itemfile;
echo '</item>' >> $itemfile;
(Re)Moving temporary files
Repositioning lines 7 - 50
Draft two of script
#!/usr/bin/fish
function skelegen -d "Generate an RSS channel for a podcast"
set -l options (fish_opt -s h -l help);
set options $options (fish_opt --short=g --long=generate --required-val);
set options $options (fish_opt --short=a --long=add --required-val --multiple-vals);
argparse $options -- $argv;
or return
if set -ql _flag_help
echo "skelegen [ -g DIR | -a FILE | -h ] CHANNEL
-h --help Display this text
-g --genenerate DIR Generate an RSS feed for DIR
-a --add FILE Add an item to a channel
Skelegen is a command line tool for generating and writing RSS feeds for podcasts."
return 0
end
if set -ql _flag_generate and set -ql _flag_add;
echo 'ERROR: skelegen cannot add and generate simultaneously'
return 1
end
if set -ql _flag_generate
set -l showid (uuidgen);
mkdir -p /tmp/skeleton/{$showid};
touch /tmp/skeleton/index;
echo $_flag_generate $showid >> /tmp/skeleton/index;
set -l channelfile /tmp/skeleton/{$showid}/rss.xml;
# write the channel data
echo '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">' > $channelfile;
echo '<channel>' >> $channelfile;
echo '<ttl>60</ttl>' >> $channelfile;
echo '<generator>skeleton</generator>' >> $channelfile;
read -l channelname -P "Channel Title: ";
echo '<title>'$channelname'</title>' >> $channelfile;
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile;
echo '<atom:link href="hub.xpub.nl/chopchop/river/public_html/podcasts/'$showid'" ref="self" type="application/rss+xml"/>' >> $channelfile;
echo '<language>en</language>' >> $channelfile;
read -l channeldesc -P "Channel Description: ";
echo '<description><![CDATA['$channeldesc']]></description>' >> $channelfile;
echo '<image><url>https://hub.xpub.nl/chopchop/worm/'$showid'/image.jpg</url>' >> $channelfile;
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channelfile;
echo '<title>'$channelname'</title>' >> $channelfile;
echo '</image>' >> $channelfile;
for file in (ls $_flag_generate);
set -l guid (sha256sum {$_flag_generate}/{$file} | grep -Eo "[[:alnum:]]{64}");
mkdir -p /tmp/skeleton/{$showid}/e/{$guid};
ln -s {$_flag_generate}/{$file} /tmp/skeleton/{$showid}/e/{$guid}/{$file};
set i (math $i + 1);
set -l itemfile (printf '/tmp/skeleton/%s/e/%s/item-%i' $showid $guid $i);
echo "<item>" > $itemfile;
read -l title -P "Item $i Title: ";
read -l desc -P "Item $i Description: ";
echo "<title>"$title"</title>" >> $itemfile;
echo "<pubDate>"(date)"</pubDate>" >> $itemfile;
echo '<enclosure url="https://hub.xpub.nl/chopchop/river/public_html/podcasts/'$showid'/e/'$guid'/'$file'" length="'(soxi -D {$_flag_generate}/{$file})'" type="'(file -b --mime-type {$_flag_generate}/{$file})'"/>' >> $itemfile;
sed -i 's/\\/\\/[^.]*\(\\/[^\\/]*.mp3\)/\1/' $itemfile;
echo '<guid isPermaLink="false">'$guid'</guid>' >> $itemfile;
set -l link "https://hub.xpub.nl/chopchop/river/public_html/podcasts/$showid/e/$guid/$file";
echo '<link>'$link'</link>' >> $itemfile;
echo '<description><![CDATA['$desc']]></description>' >> $itemfile;
echo '</item>' >> $itemfile;
end
# concatenate items in reverse order and append to the channel file
for item in (ls -t /tmp/skeleton/*/e/*/item-*);
cat $item >> $channelfile;
end
# close the rss and channel tags
echo '</channel>' >> $channelfile;
echo '</rss>' >> $channelfile;
# (re)move temporary files
mkdir -p ~/public_html/podcasts/{$showid};
mv /tmp/skeleton/{$showid}/* ~/public_html/podcasts/{$showid}/;
touch ~/public_html/podcasts/index;
cat /tmp/skeleton/index >> ~/public_html/podcasts/index;
rm -r /tmp/skeleton/;
end
if set -ql _flag_add
# echo $_flag_add
end
end
Draft three of script
#!/usr/bin/fish
function skelegen -d "Generate an RSS channel for a podcast"
set -l options (fish_opt -s h -l help);
set options $options (fish_opt --short=g --long=generate --required-val);
set options $options (fish_opt --short=a --long=add --required-val --multiple-vals);
set options $options (fish_opt --short=c --long=channel --required-val);
argparse $options -- $argv;
or return
if set -ql _flag_help
echo "skelegen [ -g DIR | -a FILE | -h ] -c XML_FILE
-h --help Display this text
-g --genenerate DIR Generate an RSS feed for DIR
-a --add FILE Add an item to a channel
-c --channel XML_FILE Specify an output channel
Skelegen is a command line application for generating and writing RSS feeds for podcasts. Skelegen is capable of generating an RSS feed for a directory of audio recordings. It is also possible to add audio recordings to a channel.
EXAMPLE USAGE
skelegen --generate ~/Music/podcast --channel ~/public_html/podcasts/mypodcast.xml
skelegen --add ~/Music/podcast/episode.mp3 --chamnel ~/public_html/podcasts/myotherpodcast.xml"
end
if set -ql _flag_generate and set -ql _flag_add;
echo 'ERROR: skelegen cannot add and generate simultaneously';
return 1;
end
if not set -ql _flag_channel;
echo "ERROR: channel must be specified.";
return 1;
end
if set -ql _flag_channel;
set -l channelprefix (echo $_flag_channel | sed -E 's/\\/([^\\/]*.xml)/\\/\n\1/' | grep -Ev "[^/]*.xml");
mkdir -p $channelprefix;
end
if set -ql _flag_generate
set -l showid (uuidgen);
set -l channelprefix (echo $_flag_channel | sed -E 's/\\/([^\\/]*.xml)/\\/\n\1/' | grep -Ev "[^/]*.xml");
set -l publicprefix (echo $channelprefix | sed -E 's/\\/home\\/riviera\\/public_html\\/([^\\/]+\\/)/https:\\/\\/hub.xpub.nl\\/chopchop\\/~river\\/\1/');
set -l channelfile (echo $_flag_channel | sed -E 's/\\/([^\\/]*.xml)/\\/\n\1/' | grep -Eo "[^/]*.xml");
mkdir -p {$channelprefix}{$showid};
set -l channel {$channelprefix}{$showid}/{$channelfile};
# # write the channel data
echo '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">' > $channel;
echo '<channel>' >> $channel;
echo '<ttl>60</ttl>' >> $channel;
echo '<generator>skeleton</generator>' >> $channel;
read -l channelname -P "Channel Title: ";
echo '<title>'$channelname'</title>' >> $channel;
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channel;
echo '<atom:link href="'$publicprefix$showid'" ref="self" type="application/rss+xml"/>' >> $channel;
echo '<language>en</language>' >> $channel;
read -l channeldesc -P "Channel Description: ";
echo '<description><![CDATA['$channeldesc']]></description>' >> $channel;
echo '<image><url>'{$publicprefix}{$showid}'/image.jpg</url>' >> $channel;
echo '<link>https://hub.xpub.nl/chopchop/worm/</link>' >> $channel;
echo '<title>'$channelname'</title>' >> $channel;
echo '</image>' >> $channel;
for file in (ls $_flag_generate);
set -l guid (sha256sum {$_flag_generate}/{$file} | grep -Eo "[[:alnum:]]{64}");
mkdir -p {$channelprefix}{$showid}/e/{$guid};
ln -s {$_flag_generate}/{$file} {$channelprefix}{$showid}/e/{$guid}/{$file};
set i (math $i + 1);
set -l itemfile (printf '%s%s/e/%s/item-%i' $channelprefix $showid $guid $i);
echo "<item>" > $itemfile;
read -l title -P "Item $i Title: ";
read -l desc -P "Item $i Description: ";
echo "<title>"$title"</title>" >> $itemfile;
echo "<pubDate>"(date)"</pubDate>" >> $itemfile;
echo '<enclosure url="'$publicprefix$showid'/e/'$guid'/'$file'" length="'(soxi -D {$_flag_generate}/{$file})'" type="'(file -b --mime-type {$_flag_generate}/{$file})'"/>' >> $itemfile;
sed -i 's/\\/\\/[^.]*\(\\/[^\\/]*.mp3\)/\1/' $itemfile;
echo '<guid isPermaLink="false">'$guid'</guid>' >> $itemfile;
set -l link "https://hub.xpub.nl/chopchop/~river/podcasts/$showid/e/$guid/$file";
echo '<link>'$link'</link>' >> $itemfile;
echo '<description><![CDATA['$desc']]></description>' >> $itemfile;
echo '</item>' >> $itemfile;
end
# concatenate items in reverse order and append to the channel file
for item in (ls -t {$channelprefix}*/e/*/item-*);
cat $item >> $channel;
end
# close the rss and channel tags
echo '</channel>' >> $channel;
echo '</rss>' >> $channel;
mkdir -p ~/.skeleton;
touch ~/.skeleton/channellist;
echo $channel >> ~/.skeleton/channellist;
end
if set -ql _flag_add
...
end
end