Configuring a job¶

Jobs live in arrowloop.json. The jobs tab in the interface writes the same file, through the same validator, so neither way can produce something the other refuses.

{
  "bwlimit": "",
  "parallelJobs": 1,
  "notify": {
    "matrix": { "homeserver": "https://matrix.example.org", "room": "!abc:example.org", "token": "syt_..." },
    "onSuccess": false
  },
  "jobs": [
    {
      "name": "photos",
      "left": "/data/Photos",
      "right": "sftp:backup/photos",
      "state": "state/photos.db",
      "schedule": "*/15 * * * *",
      "watch": true,
      "exclude": ["*.xmp", "**/.thumbnails/**"],
      "emptyDirs": true
    }
  ]
}

Everything is checked when the file is read, including cron expressions, durations and misspelled field names. JSON normally ignores a field it does not recognise, so "excludes" instead of "exclude" would leave the filter empty and sync exactly the files you thought you had excluded.

Paths resolve against the file itself, not against whatever directory a service manager happened to start the process in.

Per job¶

Field	Default	What it does
`name`	required	How the job is asked for and how its history is kept apart. Two jobs cannot share one.
`left`, `right`	required	A local path or any rclone remote. Which is which makes no difference to the engine.
`state`	required	Where this job remembers what the two sides agreed on.
`direction`	both ways	`leftToRight` or `rightToLeft` makes one side the source and never writes to it. Anything else, including an empty field, is both ways.
`schedule`	none	A cron expression. Empty means the job only runs when somebody asks.
`watch`	`false`	Run when a local side changes. Needs a schedule as well; see below.
`watchSettle`	`2s`	How long the tree must go quiet before a change counts.
`runAtStart`	`false`	Run once as soon as the program starts, before waiting for the schedule. Fires once per start and never on a configuration reload.
`disabled`	`false`	Keeps the job in the file without running it.
`exclude`	none	Globs. Without a slash they match the file name at any depth; with one, the whole path. `**` crosses directories.
`noDefaultExcludes`	`false`	Also sync half-written files such as `*.part` and Office owner files.
`quietPeriod`	`5s`	How long a file must sit unchanged before it is touched.
`modWindow`	`2s`	How far modification times may differ and still count as equal. Only applies where a side cannot produce a hash.
`transfers`	`4`	How many files may be copied at once.
`emptyDirs`	`false`	Carry folders that hold no files.
`metadata`	`false`	Carry permissions, ownership and extended attributes.
`noTrash`	`false`	Delete outright instead of moving into the side's own trash. See the note below before switching it on.
`reportOnly`	`false`	Compare on every automatic turn and apply nothing. A run somebody starts by hand still applies.
`brakePercent`	`50`	Refuse a run deleting more than this share of known files. `0` switches the brake off.
`brakeFloor`	`10`	Never trip the brake below this many deletions.
`before`	none	A shell command run before every run. If it fails, the run does not start. See below.
`after`	none	A shell command run after every run, successful or not. If it fails, the run is marked failed.

noTrash makes a deletion final

The trash is what makes the promise at the top of Safety hold: nothing this program does destroys anything by itself. Switching it off withdraws that promise for this job, and it also applies to the losing side of a conflict, which is the case people forget. What you get back is the .arrowloop folder never being created, which matters on a share other people can see. Reasonable for a folder of downloads. A bad idea for documents.

The field is spelled as the negative so that a configuration written before it existed, or one where somebody forgot the line, keeps its trash. In the interface the switch reads the other way round and says "keep a bin", since a switch labelled with a negative is one people set backwards.

Zero is not the same as leaving it out

brakePercent and brakeFloor distinguish an explicit 0 from an absent field on purpose. Switching off the mass-delete brake has to be something you typed, never something you got by forgetting a line.

A one-way job restores, it does not skip

Both sides are still compared, because comparing is how the engine knows what changed. What the direction decides is what may be done with the answer. A file edited on the destination is restored from the source and a file deleted there is copied back, so the destination is made to agree. Skipping those changes instead would report them again on every run and the two sides would drift further apart for ever. What the source never had is left alone: nothing there says it should exist.

Watching¶

"watch": true starts a run when a local side changes, instead of waiting for the next tick. The reason is not latency: a schedule has to list both sides in full on every tick, which on a large tree or over a network is most of what a run costs.

It never replaces the schedule, and a watching job without one is refused. Only a local side can be watched, most remote backends have no way to tell anyone anything, and a watcher that missed an event has no way to know it did. The schedule is what eventually notices what the watcher did not.

Running at start¶

"runAtStart": true runs the job once as soon as the program starts, before its schedule is next due.

It exists because of what a schedule cannot say. A machine that was off overnight missed every turn a daily job had, and the job's next run is tomorrow: the two sides stay apart for a whole day for no reason other than the clock. This is also the setting that makes autostart worth switching on, since a program that starts with the session and then sits there until 03:00 has not helped anybody who has just turned their computer on.

Three things it does not do. It fires once per program start and never on a configuration reload, so saving one job in the interface does not set every job in the file running. A disabled job does not run, because disabled has to mean disabled everywhere. And the runs go one after another in file order rather than all at once, for the same reason ordinary runs are serialised: they share one uplink and one disk, and a start-up burst is where that matters most.

Commands before and after a run¶

{
  "name": "database",
  "left": "/srv/dump",
  "right": "b2:backup/dump",
  "state": "state/database.db",
  "before": "pg_dump -U app app > /srv/dump/app.sql",
  "after": "/usr/local/bin/report-backup.sh"
}

before runs before the two sides are compared. It is the place to write a database dump, stop a service that holds files open, or mount a drive. If it exits with anything other than zero, the run does not start, and the run record carries the last lines the command printed, since that is usually where a script says why it gave up.

after runs once the run is over, whether it went well or not, so it can restart what before stopped. It learns how the run went from its environment:

Variable	What it holds
`ARROWLOOP_JOB`	The job's name. Also set for `before`.
`ARROWLOOP_LEFT`, `ARROWLOOP_RIGHT`	The two sides as written in the file. Also set for `before`.
`ARROWLOOP_RESULT`	`ok` or `failed`.
`ARROWLOOP_ERROR`	Why the run failed, empty otherwise.
`ARROWLOOP_COPIED`, `ARROWLOOP_MOVED`, `ARROWLOOP_DELETED`	How many files the run copied, moved and deleted.
`ARROWLOOP_CONFLICTS`, `ARROWLOOP_SKIPPED`	How many files were in conflict and how many were left for later.

A command that exits with an error after a successful run marks the run failed, so a notification goes out for it. The commands run through sh -c, or cmd /C on Windows, in the program's working directory. Each may take fifteen minutes; after that it is stopped, because a hanging command would hold its job's place and every job queued behind it.

Only the file can set them

Anybody who can reach the interface could otherwise run anything on this machine, with the rights of the process. So the interface shows the commands but cannot set or change them: every save from the interface puts back what the file on disk says for a job of the same name. A job created or renamed in the interface starts without commands. The jobs tab and the file agree on everything else.

Whole file¶

Field	Default	What it does
`bwlimit`	none	rclone syntax, `1M` or a timetable like `08:00,512k 19:00,off`.
`parallelJobs`	`1`	How many jobs may run at once.
`history`	beside the file	Where run records go.
`notify.matrix`	none	A room to post into: homeserver, room id, access token.
`notify.webhook`	none	A URL that receives a small JSON document.
`notify.onSuccess`	`false`	Report every run rather than only the failures.

The bandwidth limit is not per job

rclone's token bucket is process-wide, so a per-job limit would be a promise the mechanism underneath cannot keep. Two jobs on one machine share one uplink either way.

Notifications default to failures only, because a tool that announces every successful sync teaches you to ignore it, and then the one message that mattered is ignored with the rest.