aaronmanning/podcast-hoarder
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

ipod syncing; updated readme; rust 2024; version bump

Browse Source
This commit is contained in:
Aaron Manning
2025-09-21 09:07:18 +10:00
parent fcc6acda56
commit 19330e66c9
11 changed files with 419 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
Cargo.lock generated
View File

@@ -522,7 +522,7 @@ checksum = "e3148f5046208a5d56bcfc03053e3ca6334e51da8dfb19b6cdc8b306fae3283e"
[[package]]
name = "podcast-hoarder"
version = "0.0.0"
version = "0.1.0"
dependencies = [
"anyhow",
"audiotags",

4
Cargo.toml
View File

@@ -1,7 +1,7 @@
[package]
name = "podcast-hoarder"
version = "0.0.0"
edition = "2021"
version = "0.1.0"
edition = "2024"
authors = ["Aaron Manning <contact@aaronmanning.net>"]
license = "GPL-2.0-only"

2
LICENSE
View File

@@ -337,4 +337,4 @@ This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Library General
Public License instead of this License.
Public License instead of this License.

51
README.md
View File

@@ -4,22 +4,20 @@ I listen to a lot of podcasts, and I am also a bit of a data hoarder when it com
To avoid being at the whim of the podcast host, I created this downloading tool to bulk download podcast episodes and store show notes. When using this tool, if an episode is removed from a podcast feed the file will stay downloaded and the show notes will be preserved, with the episode simply marked as no longer being available in the current version of the feed. Episodes which are changed and the author has marked the episode as changed as per a change in the guid will be downloaded as separate episodes.
This is not a podcast player, it is a data hoarding tool.
> **Note**:
> This program is in version `0.0.0` because it is the first version that I am testing for problems and issues. Once I have used it for a little while with lots of different feeds, I will stabilize the format for the way metadata is stored and prevent breaking changes.
Over time, this then evolved as a way to manage podcasts for my [iPod](https://support.apple.com/en-lb/docs/ipod/131146) running [Rockbox](https://www.rockbox.org/), including generating playlist files, syncing played episodes from the iPod back to the master copy, and loading the latest episodes on to the iPod.
## Installation
```
git clone https://git.aaronmanning.net/aaronmanning/podcast-hoarder.git
cd podcast-hoarder
cargo install --path .
cargo install --git https://git.aaronmanning.net/aaronmanning/podcast-hoarder.git
```
## Usage
Open an empty folder for storing all podcast episodes and metadata and create a file called `podcasts.toml`. Here is an example of this file containing a few podcasts
### Feed Setup
Open an empty folder for storing all podcast episodes and metadata and create a file called `podcasts.toml`. Below is an example of this file containing a few podcasts.
```
[podcasts]
99-percent-invisible = "https://feeds.simplecast.com/BqbsxVfO"
@@ -27,21 +25,40 @@ this-american-life = "https://www.thisamericanlife.org/podcast/rss.xml"
hello-internet = "https://www.hellointernet.fm/podcast?format=rss"
```
Each line is in the format
Each line is in the following format.
```
alias = "rss_feed"
```
It is also possible to replace the right hand side with an object `{ source = "rss_feed", skip_download = true }` if you wish to ignore the feed when downloading but still include it in playlist generation. This is particularly useful when a feed no longer becomes available, and you don't want to hit the download error every time.
The alias for the podcast is the name given to the folder where they are stored and the identifier for the sake of this app. This is simply to make downloads resistent to name changes in the feed. Changing an alias will cause a redownload of the episodes.
Then simply run
```
podcast-hoarder
```
to bulk download all current episodes.
Subsequently running the program will check for updates and download any new episodes as required.
Note that RSS feeds which start with `http` are fetched from the web, anything else is treated as a file path which allows local RSS files to be used as well.
### Downloading
Once everything is set up, you can run the following line to download all the episodes.
```
podcast-hoarder download
```
Subsequently running the program will check for updates and download any new episodes as required.
### iPod Sync
To sync podcasts to a Rockbox iPod, simply run
```
podcast-hoarder <ipod-root>
```
replacing `<ipod-root>` with the root directory of the iPod. This will generate playlists for all podcasts, sync the playlists to the iPod, and sync the episodes to the iPod.
You can also include the `--download` flag to include downloading within this command.
All podcasts have the prefix `[PC]` attached to their name. To each podcast `<podcast>`, there are two playlists, `[PC] <podcast>` and `[PC] <podcast> (unlistened)`. There is also a playlist `[PC] [Master Feed]` which contains all podcast episodes ordered from most recent to oldest.
When syncing played episodes from the iPod to the local copy, this program looks for the playlist named `[PC] [Listened]`, and if the file doesn't exist, it will be created the first time round. As such, simply add episodes to this playlist when they are listened to in order to sync the played status back and have them no longer appear in the `(unlistened)` version of the playlist.

72
src/download.rs
View File

@@ -10,6 +10,7 @@ use sanitise_file_name::sanitise;
use crate::{
rss,
input,
folders,
manage::{
Specification,
@@ -33,9 +34,14 @@ fn download_to_file(url: &str, path: &path::Path) -> anyhow::Result<()> {
pub(crate) fn update_podcast(
alias: &str,
root: &path::Path,
feed_location: &str,
source: &input::Source,
) -> anyhow::Result<()> {
if source.skip_download() {
println!(r#"[info] skipping download for "{}""#, alias);
return Ok(())
}
// Create output directory
let output = folders::podcast_folder(root, alias);
if !output.exists() {
@@ -45,32 +51,32 @@ pub(crate) fn update_podcast(
println!(r#"[info] scanning feed for "{}""#, alias);
if feed_location.starts_with("http") {
let feed_url = feed_location;
match source.source() {
input::SourceKind::Url(feed_url) => {
// Get the podcast feed
let response = minreq::get(feed_url)
// For SquareSpace which refuses requests with no User-Agent
.with_header("User-Agent", "podcast-downloader")
.with_header("Accept", "*/*")
.send()
.context(format!(r#"error when requesting feed url "{}" for {}"#, feed_url, alias))?;
// Get the podcast feed
let response = minreq::get(feed_url)
// For SquareSpace which refuses requests with no User-Agent
.with_header("User-Agent", "podcast-downloader")
.with_header("Accept", "*/*")
.send()
.context(format!(r#"error when requesting feed url "{}" for {}"#, feed_url, alias))?;
if response.status_code != 200 {
eprintln!(r#"[error] feed "{}" for alias {} responded with non-200 ({}) status code"#, feed_url, alias, response.status_code);
return Ok(());
}
if response.status_code != 200 {
eprintln!(r#"[error] feed "{}" for alias {} responded with non-200 ({}) status code"#, feed_url, alias, response.status_code);
return Ok(());
let feed = response.as_str()?.to_owned();
update_podcast_from_feed(&output, &feed)
}
let feed = response.as_str()?.to_owned();
update_podcast_from_feed(&output, &feed)
} else {
let feed_path = root.join(feed_location);
match fs::read_to_string(&feed_path) {
Ok(feed) => update_podcast_from_feed(&output, &feed),
Err(err) => {
eprintln!(r#"[error] failed to read path "{}" with error {}"#, feed_path.display(), err);
Ok(())
input::SourceKind::Path(feed_location) => {
let feed_path = root.join(feed_location);
match fs::read_to_string(&feed_path) {
Ok(feed) => update_podcast_from_feed(&output, &feed),
Err(err) => {
eprintln!(r#"[error] failed to read path "{}" with error {}"#, feed_path.display(), err);
Ok(())
}
}
}
}
@@ -149,7 +155,7 @@ pub(crate) fn update_podcast_from_feed(
let channel = feed.rss.channel;
let spec_file = output.join("spec.toml");
let spec_file = output.join(folders::SPEC_FILE);
let mut spec = Specification::read_from_with_default(&spec_file)?;
@@ -243,11 +249,21 @@ pub(crate) fn update_podcast_from_feed(
match download_to_file(enclosure.url.as_ref(), &file_path) {
Ok(()) => {
let file_path = file_path.canonicalize().unwrap();
let relative_path = file_path.strip_prefix(&output).unwrap();
spec.insert_into_files(
if let Some(previous) = spec.insert_into_files(
id.to_owned(),
file_path.strip_prefix(&output).unwrap().to_owned(),
)?;
relative_path.to_owned(),
) {
println!("[warning] duplicate id {:?} for episodes {:?} and {:?}", id, previous, relative_path);
// Revert to the previous file
spec.insert_into_files(id.to_owned(), previous);
// Delete the newly downloaded file
fs::remove_file(file_path)?;
// Skip
continue;
}
let episode = Episode::new_downloaded(title, description, id.to_owned());

11
src/folders.rs
View File

@@ -2,11 +2,18 @@ use std::path;
use sanitise_file_name::sanitise;
pub(crate) const PODCASTS_DIR: &str = "Podcasts";
pub(crate) const SPEC_FILE: &str = "spec.toml";
pub(crate) const LOCAL_PODCASTS_DIR: &str = "podcasts";
pub(crate) const LOCAL_PLAYLISTS_DIR: &str = "playlists";
pub(crate) const IPOD_PODCASTS_DIR: &str = "Podcasts";
pub(crate) const LISTENED_PLAYLIST_PATH: &str = "[PC] [Listened].m3u";
pub(crate) const MASTER_PLAYLIST_PATH: &str = "[PC] [Master Feed].m3u";
pub(crate) fn podcast_folder(
root: &path::Path,
alias: &str,
) -> path::PathBuf {
root.join(PODCASTS_DIR).join(sanitise(alias))
root.join(LOCAL_PODCASTS_DIR).join(sanitise(alias))
}

58
src/input.rs
View File

@@ -1,5 +1,5 @@
use std::path;
use std::collections::HashMap;
use std::collections::BTreeMap;
#[derive(clap::Parser)]
pub(crate) struct Args {
@@ -48,12 +48,66 @@ pub(crate) enum Command {
#[arg(long, short)]
podcast: Option<String>,
},
/// Syncs local copy of podcasts to Rockbox iPod.
Sync {
/// Directory to the root of the iPod running Rockbox.
ipod_dir: path::PathBuf,
/// Download any new episodes before syncing.
#[clap(short, long)]
download: bool,
},
}
/// Struct modelling configuration file format.
#[derive(serde::Deserialize)]
pub(crate) struct Config {
/// Map from podcast alias to RSS feed either as a url (prefix: http) or file path.
pub(crate) podcasts: HashMap<String, String>,
pub(crate) podcasts: BTreeMap<String, Source>,
}
#[derive(serde::Deserialize)]
#[serde(untagged)]
pub(crate) enum Source {
String(String),
Object {
source: String,
#[serde(rename = "skip-download")]
skip_download: bool,
},
}
impl Source {
pub(crate) fn source_raw(&self) -> &str {
match self {
Self::String(source) | Self::Object { source, .. } => source,
}
}
pub(crate) fn skip_download(&self) -> bool {
match self {
Self::String(_) => false,
Self::Object { skip_download, .. } => *skip_download,
}
}
fn is_url(&self) -> bool {
self.source_raw().starts_with("http")
}
pub(crate) fn source(&self) -> SourceKind<'_> {
if self.is_url() {
SourceKind::Url(self.source_raw())
} else {
SourceKind::Path(self.source_raw())
}
}
}
pub(crate) enum SourceKind<'a> {
Url(&'a str),
Path(&'a str),
}

46
src/main.rs
View File

@@ -1,4 +1,5 @@
mod rss;
mod sync;
mod input;
mod manage;
mod folders;
@@ -34,8 +35,8 @@ fn main() -> anyhow::Result<()> {
Command::Download { podcast } => {
// Updating single podcast
if let Some(alias) = podcast {
if let Some(feed_url) = config.podcasts.get(&alias) {
download::update_podcast(&alias, root, feed_url)?;
if let Some(source) = config.podcasts.get(&alias) {
download::update_podcast(&alias, root, source)?;
}
else {
anyhow::bail!(r#"podcast "{}" not found in configuration file"#, alias)
@@ -43,14 +44,14 @@ fn main() -> anyhow::Result<()> {
}
// Updating all podcasts
else {
for (alias, feed_url) in config.podcasts {
download::update_podcast(&alias, root, &feed_url)?;
for (alias, source) in config.podcasts {
download::update_podcast(&alias, root, &source)?;
}
}
},
}
Command::List { status, podcast } => {
let output = folders::podcast_folder(root, &podcast);
let spec_file = output.join("spec.toml");
let spec_file = output.join(folders::SPEC_FILE);
let spec = manage::Specification::read_from(&spec_file)?;
@@ -63,10 +64,10 @@ fn main() -> anyhow::Result<()> {
}
}
}
},
}
Command::Mark { status, podcast } => {
let output = folders::podcast_folder(root, &podcast);
let spec_file = output.join("spec.toml");
let spec_file = output.join(folders::SPEC_FILE);
let mut spec = manage::Specification::read_from(&spec_file)?;
@@ -77,7 +78,7 @@ fn main() -> anyhow::Result<()> {
}
spec.write_to(&spec_file)?;
},
}
Command::Playlist { podcast } => {
if let Some(alias) = podcast {
playlist::generate_podcast_m3u(alias.as_str(), root, false)?;
@@ -89,7 +90,32 @@ fn main() -> anyhow::Result<()> {
}
playlist::generate_master_m3u(&config, root)?;
}
},
}
Command::Sync { ipod_dir, download } => {
// Sync listened podcasts
sync::sync_listened(
root,
&ipod_dir.join(folders::LOCAL_PLAYLISTS_DIR).join(folders::LISTENED_PLAYLIST_PATH),
true,
)?;
if download {
// Download all new episodes
for (alias, source) in &config.podcasts {
download::update_podcast(&alias, root, &source)?;
}
}
// Generate updated playlist files
for (alias, _) in &config.podcasts {
playlist::generate_podcast_m3u(alias.as_str(), root, true)?;
playlist::generate_podcast_m3u(alias.as_str(), root, false)?;
}
playlist::generate_master_m3u(&config, root)?;
// Sync podcasts and playlists
sync::sync(root, &config, &ipod_dir)?;
}
};
Ok(())

15
src/manage.rs
View File

@@ -8,8 +8,6 @@ use std::{
},
};
use anyhow::Context;
#[derive(Debug, Default, serde::Serialize, serde::Deserialize)]
pub(crate) struct Specification<'a> {
files: HashMap<Cow<'a, str>, Cow<'a, path::Path>>,
@@ -74,13 +72,11 @@ impl<'a> Specification<'a> {
&mut self,
id: impl Into<Cow<'a, str>>,
path: impl Into<Cow<'a, path::Path>>,
) -> anyhow::Result<()> {
) -> Option<Cow<'a, path::Path>> {
self.files.insert(
id.into(),
path.into()
)
.context("insertion of episode with duplicate id")
.map(|_| ())
}
pub(crate) fn insert_into_feed(
@@ -100,6 +96,15 @@ impl<'a> Specification<'a> {
}
}
}
pub(crate) fn files_and_feed_mut(
&mut self,
) -> (
&mut BTreeMap<chrono::NaiveDateTime, Vec<Episode<'a>>>,
&HashMap<Cow<'a, str>, Cow<'a, path::Path>>,
) {
(&mut self.feed, &self.files)
}
}
#[derive(Debug, serde::Serialize, serde::Deserialize)]

50
src/playlist.rs
View File

@@ -1,5 +1,6 @@
use std::{
fs,
io,
iter,
path,
collections::BTreeMap,
@@ -28,12 +29,16 @@ impl<'a> Playlist<'a> {
}
}
/// Writes the playlist file based on the specified filename.
///
/// Output boolean indicates if the playlist was written (if it was
/// different from the existing file)
fn write_as(
&self,
name: &str,
reverse: bool,
) -> anyhow::Result<()> {
let playlists_folder = self.root.join("Playlists");
) -> anyhow::Result<bool> {
let playlists_folder = self.root.join(folders::LOCAL_PLAYLISTS_DIR);
if !playlists_folder.exists() {
fs::create_dir(&playlists_folder)
.context(format!("failed to create output directory for playlists"))?;
@@ -41,9 +46,8 @@ impl<'a> Playlist<'a> {
let mut path = playlists_folder.join(sanitise(name));
path.set_extension("m3u");
let mut file = fs::File::create(path)?;
let mut writer = m3u::Writer::new(&mut file);
let mut output = io::BufWriter::new(Vec::new());
let mut writer = m3u::Writer::new(&mut output);
let entries =
self
.files
@@ -61,7 +65,15 @@ impl<'a> Playlist<'a> {
}
};
Ok(())
drop(writer);
let output = output.into_inner()?;
if fs::read(&path)? != output {
fs::write(path, output.as_slice())?;
Ok(true)
} else {
Ok(false)
}
}
fn insert(
@@ -71,9 +83,9 @@ impl<'a> Playlist<'a> {
) {
let entry = m3u::path_entry({
let relative = absolute_path.strip_prefix(
&self.root.join(folders::PODCASTS_DIR)
&self.root.join(folders::LOCAL_PODCASTS_DIR)
).unwrap();
path::Path::new("/Podcasts").join(relative)
path::Path::new(folders::IPOD_PODCASTS_DIR).join(relative)
});
match self.files.get_mut(&published) {
@@ -98,7 +110,7 @@ pub(crate) fn generate_master_m3u(
for (podcast, _) in &config.podcasts {
let output = folders::podcast_folder(root, podcast.as_str());
let spec_file = output.join("spec.toml");
let spec_file = output.join(folders::SPEC_FILE);
let spec = Specification::read_from(&spec_file)?;
let (feed, files) = spec.into_feed_and_files();
@@ -117,7 +129,11 @@ pub(crate) fn generate_master_m3u(
}
}
playlist.write_as("[Podcast Master Feed]", true)
if playlist.write_as(folders::MASTER_PLAYLIST_PATH, true)? {
println!("[info] generated master playlist");
}
Ok(())
}
@@ -129,7 +145,7 @@ pub(crate) fn generate_podcast_m3u(
let mut playlist = Playlist::new(root);
let output = folders::podcast_folder(root, alias);
let spec_file = output.join("spec.toml");
let spec_file = output.join(folders::SPEC_FILE);
let spec = Specification::read_from(&spec_file)?;
let episodes =
@@ -153,9 +169,15 @@ pub(crate) fn generate_podcast_m3u(
);
}
if unlistened_only {
playlist.write_as(&format!("{} (unlistened)", alias), false)
let written = if unlistened_only {
playlist.write_as(&format!("[PC] {} (unlistened)", alias), false)
} else {
playlist.write_as(alias, false)
playlist.write_as(&format!("[PC] {}", alias), false)
}?;
if written {
println!("[info] generated playlist for podcast {:?}.", alias);
}
Ok(())
}

190
src/sync.rs Normal file
View File

@@ -0,0 +1,190 @@
use std::{
collections::{
BTreeMap,
BTreeSet,
},
fs,
io,
path,
};
use crate::{
folders, input, manage::Specification
};
use anyhow::Context;
/// Rockbox writes a BOM to playlist files it creates, so we use this function
/// to extract the BOM if it exists, from a reader.
fn extract_bom<R: io::Read>(
reader: &mut io::BufReader<R>,
) -> anyhow::Result<()> {
use io::Read;
let mut bom = Vec::with_capacity(3);
let mut bom_reader = reader.take(3);
bom_reader.read_to_end(&mut bom)?;
if &bom[..] == &[0xEF, 0xBB, 0xBF] {
let mut dummy = [0u8; 3];
reader.read_exact(&mut dummy)?;
}
Ok(())
}
/// Takes a playlist of listened files and updates the status of the files in
/// the master copy.
pub(crate) fn sync_listened(
// The folder containing the podcasts.toml file.
root: &path::Path,
// The path to the m3u playlist file.
playlist: &path::Path,
// Whether the playlist file given should be emptied when done.
empty_file: bool,
) -> anyhow::Result<()> {
// The way the specification file was written makes this operation much
// slower than it needs to be, since there isn't a convenient lookup from
// file to episodes, only the other way around.
//
// In practice it hardly matters though, the playlist can never really be
// that big.
if !playlist.exists() {
println!("[warning] specified playlist file does not exist. skipping.");
if empty_file {
fs::write(playlist, [])?;
}
return Ok(());
}
let mut reader =
io::BufReader::new(fs::File::open(playlist)?);
// Extract the BOM if it exists
extract_bom(&mut reader)?;
let mut reader = m3u::Reader::new(
reader
);
// We first extract all of the played files as a map from the podcast name
// to all of the played episodes of that podcast.
let mut played = BTreeMap::<String, BTreeSet<_>>::new();
for entry in reader.entries() {
if let m3u::Entry::Path(entry) = entry? {
let mut components: Vec<_> = entry.components().rev().take(2).collect();
let Some(podcast) = components.pop() else {
println!("[warning] could not identify podcast for file {:?}.", entry);
continue;
};
let Some(file_name) = components.pop() else {
println!("[warning] empty file name found in playlist. skipping.");
continue;
};
let podcast = podcast.as_os_str().to_str().context("podcast name was not a valid string")?;
let file_name = path::PathBuf::from(file_name.as_os_str());
// We append the episode to the appropriate entry of the map, or
// create a new one if we are looking at the first episode of a
// given podcast.
match played.get_mut(podcast) {
Some(existing) => {
existing.insert(file_name.to_owned());
}
None => {
played.insert(
podcast.to_owned(),
BTreeSet::from([file_name.to_owned()]),
);
}
}
}
}
for (podcast, played) in played {
println!("[info] syncing played episodes for {}", podcast);
let output = folders::podcast_folder(root, podcast.as_str());
let spec_file = output.join(folders::SPEC_FILE);
let mut spec = Specification::read_from(&spec_file)?;
let (feed, files) = spec.files_and_feed_mut();
for (_, episodes) in feed.iter_mut() {
for episode in episodes {
if played.contains(files.get(episode.id()).as_deref().unwrap().as_ref()) {
println!("[info] marking {:?} as played", episode.title());
episode.listened = true;
}
}
}
spec.write_to(&spec_file)?;
}
if empty_file {
fs::write(playlist, [])?;
}
Ok(())
}
pub(crate) fn sync(
root: &path::Path,
config: &input::Config,
target_dir: &path::Path,
) -> anyhow::Result<()> {
if !target_dir.exists() {
anyhow::bail!("specified target directory does not exist");
}
if !target_dir.join(folders::LOCAL_PODCASTS_DIR).exists() {
anyhow::bail!("specified target directory does not contain a folder {:?}", folders::LOCAL_PODCASTS_DIR);
}
for (podcast, _) in &config.podcasts {
let output = folders::podcast_folder(root, podcast.as_str());
let spec_file = output.join(folders::SPEC_FILE);
let spec = Specification::read_from(&spec_file)?;
for episode in spec.feed_iter().map(|(_, eps)| eps.iter()).flatten() {
let episode_local_path = spec.path_from_id(episode.id()).unwrap();
// let relative_path = path::PathBuf::from().join(podcast.as_str()).join(path);
let source = output.join(&episode_local_path);
let target = target_dir.join(source.strip_prefix(root).unwrap());
if !target.exists() {
println!("[info] copying from {:?} to {:?}.", source, target);
fs::copy(source, target)?;
}
}
}
if !target_dir.join(folders::LOCAL_PLAYLISTS_DIR).exists() {
anyhow::bail!("specified target directory does not contain a folder {:?}", folders::LOCAL_PLAYLISTS_DIR);
}
for source in fs::read_dir(root.join(folders::LOCAL_PLAYLISTS_DIR))? {
let source = source?.path();
let target = target_dir
.join(folders::LOCAL_PLAYLISTS_DIR)
.join(source.file_name().unwrap());
if !target.exists() || fs::metadata(&target)?.modified()? < fs::metadata(&source)?.modified()? {
println!("[info] copying playlist {:?}.", source.file_name().unwrap());
fs::copy(source, target)?;
}
}
Ok(())
}