merged updates from main

README.md

@@ -16,20 +16,24 @@
 
 <h4 align="center">For Podman - see the fork <a href="https://github.com/sudo-kraken/podcheck">sudo-kraken/podcheck</a>!</h4>
 
+<h4 align="center">:whale: Docker Hub pull limit :chart_with_downwards_trend: not an issue for checks but for actual pulls - <a href="#whale-docker-hub-pull-limit-chart_with_downwards_trend-not-an-issue-for-checks-but-for-actual-pulls">read more</a></h4>
+
 ___
 ## :bell: Changelog
 
+Made MaxAsync=1 the default - edit to change.
+Added -x option to pass a MaxAsync value on runtime.
+Made it possible to disable xargs -P-flag by setting MaxAsync=0 or passing -x 0 option.
+
+- **v0.5.6.1**: Async xargs hotfix - due to errors `failed to request manifest head ... context canceled`
+    - Defaulted subprocess to 1 with `MaxAsync=1`, increase to find a stable value in your environment.
+    - Added `-x N` option to pass `MaxAsync` value at runtime.
+    - To disable xargs `-P` flag (max processes) all together, set `MaxAsync` to 0.
+- **v0.5.6.0**: Heavily improved performance due to async checking for updates.
+- **v0.5.5.0**: osx and bsd compatibility changes + rewrite of dependency installer
 - **v0.5.4.0**: Added support for a Prometheus+node_exporter metric collection through a file collector.
 - **v0.5.3.0**: Local image check changed (use imageId instead of name) and Gotify-template fixed (whale icon removed).
 - **v0.5.2.1**: Rewrite of dependency downloads, jq can be installed with package manager or static binary.
-- **v0.5.1**: DEPENDENCY WARNING: now requires **jq**. + Upstreaming changes from [sudo-kraken/podcheck](https://github.com/sudo-kraken/podcheck)
-- **v0.5.0**: Rewritten notify logic - all templates are adjusted and should be migrated!
-    - Copy the custom settings from your current template to the new version of the same template.
-    - Look into, copy and customize the `urls.list` file if that's of interest.
-    - Other changes:
-        - Added Discord notify template.
-        - Verbosity changed of `regctl`.
-- **v0.4.9**: Added a function to enrich the notify-message with release note URLs. See [Release notes addon](https://github.com/mag37/dockcheck#date-release-notes-addon-to-notifications)
 ___
 
 
@@ -83,6 +87,7 @@ ___
 ## :nut_and_bolt: Dependencies
 - Running docker (duh) and compose, either standalone or plugin. (see [Podman fork](https://github.com/sudo-kraken/podcheck)  
 - Bash shell or compatible shell of at least v4.3
+  - POSIX `xargs`, usually default but can be installed with the `findutils` package - to enable async.
 - [jq](https://github.com/jqlang/jq)
   - User will be prompted to install with package manager or download static binary.
 - [regclient/regctl](https://github.com/regclient/regclient) (Licensed under [Apache-2.0 License](http://www.apache.org/licenses/LICENSE-2.0))  
@@ -91,6 +96,7 @@ ___
 
 ## :tent: Install Instructions
 Download the script to a directory in **PATH**, I'd suggest using `~/.local/bin` as that's usually in **PATH**.   
+For OSX/macOS preferably use `/usr/local/bin`.
 ```sh
 # basic example with curl:
 curl -L https://raw.githubusercontent.com/mag37/dockcheck/main/dockcheck.sh -o ~/.local/bin/dockcheck.sh
@@ -98,6 +104,9 @@ chmod +x ~/.local/bin/dockcheck.sh
 
 # or oneliner with wget:
 wget -O ~/.local/bin/dockcheck.sh "https://raw.githubusercontent.com/mag37/dockcheck/main/dockcheck.sh" && chmod +x ~/.local/bin/dockcheck.sh
+
+# OSX or macOS version with curl:
+ curl -L https://raw.githubusercontent.com/mag37/dockcheck/main/dockcheck.sh -o /usr/local/bin/dockcheck.sh && chmod +x /usr/local/bin/dockcheck.sh
 ```
 Then call the script anywhere with just `dockcheck.sh`.
 Add preferred `notify.sh`-template to the same directory - this will not be touched by the scripts self-update function.
@@ -179,7 +188,15 @@ chmod 755 regctl
 ```
 Test it with `./regctl --help` and then either add the file to the same path as *dockcheck.sh* or in your path (eg. `~/.local/bin/regctl`).
 
-## :guardsman: Function to auth with docker hub before running
+## :whale: Docker Hub pull limit :chart_with_downwards_trend: not an issue for checks but for actual pulls
+Due to recent changes in [Docker Hub usage and limits](https://docs.docker.com/docker-hub/usage/)
+>Unauthenticated users: 10 pulls/hour   
+>Authenticated users with a free account: 100 pulls/hour
+
+This is not an issue for registry checks. But if you have a large stack and pull more than 10 updates at once consider updating more often or to create a free account.
+You could use/modify the login-wrapper function in the example below to automate the login prior to running `dockcheck.sh`.
+
+### :guardsman: Function to auth with docker hub before running
 **Example** - Change names, paths, and remove cat+password flag if you rather get prompted:
 ```sh
 function dchk {

dockcheck.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
-VERSION="v0.5.4.0"
+VERSION="v0.5.6.1"
-### ChangeNotes: Added support for a Prometheus+node_exporter metric collection through a file collector.
+### ChangeNotes: Async hotfix, 1 subprocess default, modify MaxAsync variable or pass -x N option to increase.
 Github="https://github.com/mag37/dockcheck"
 RawUrl="https://raw.githubusercontent.com/mag37/dockcheck/main/dockcheck.sh"
 
@@ -13,6 +13,10 @@ ScriptWorkDir="$(dirname "$ScriptPath")"
 LatestRelease="$(curl -s -r 0-50 $RawUrl | sed -n "/VERSION/s/VERSION=//p" | tr -d '"')"
 LatestChanges="$(curl -s -r 0-200 $RawUrl | sed -n "/ChangeNotes/s/# ChangeNotes: //p")"
 
+# User customizable defaults
+MaxAsync=1
+Timeout=10
+
 # Help Function
 Help() {
   echo "Syntax:     dockcheck.sh [OPTION] [part of name to filter]"
@@ -34,6 +38,7 @@ Help() {
   echo "-s     Include stopped containers in the check. (Logic: docker ps -a)."
   echo "-t     Set a timeout (in seconds) per container for registry checkups, 10 is default."
   echo "-v     Prints current version."
+  echo "-x N   Set max asynchronous subprocesses, 1 default, 0 to disable, 32+ tested."
   echo
   echo "Project source: $Github"
 }
@@ -46,9 +51,8 @@ c_blue="\033[0;34m"
 c_teal="\033[0;36m"
 c_reset="\033[0m"
 
-Timeout=10
 Stopped=""
-while getopts "aynpfrhlisvmc:e:d:t:" options; do
+while getopts "aynpfrhlisvmc:e:d:t:x:" options; do
   case "${options}" in
     a|y) AutoUp="yes" ;;
     c)   CollectorTextFileDirectory="${OPTARG}"
@@ -64,6 +68,7 @@ while getopts "aynpfrhlisvmc:e:d:t:" options; do
     s)   Stopped="-a" ;;
     t)   Timeout="${OPTARG}" ;;
     v)   printf "%s\n" "$VERSION" ; exit 0 ;;
+    x)   MaxAsync=${OPTARG} ;;
     d)   DaysOld=${OPTARG}
          if ! [[ $DaysOld =~ ^[0-9]+$ ]] ; then { printf "Days -d argument given (%s) is not a number.\n" "${DaysOld}" ; exit 2 ; } ; fi ;;
     h|*) Help ; exit 2 ;;
@@ -131,7 +136,8 @@ choosecontainers() {
 
 datecheck() {
   ImageDate=$($regbin -v error image inspect "$RepoUrl" --format='{{.Created}}' | cut -d" " -f1 )
-  ImageAge=$(( ( $(date +%s) - $(date -d "$ImageDate" +%s) )/86400 ))
+  ImageEpoch=$(date -d "$ImageDate" +%s 2>/dev/null) || ImageEpoch=$(date -f "%Y-%m-%d" -j "$ImageDate" +%s)
+  ImageAge=$(( ( $(date +%s) - $ImageEpoch )/86400 ))
   if [ "$ImageAge" -gt "$DaysOld" ] ; then
     return 0
   else
@@ -169,7 +175,7 @@ if [[ "$VERSION" != "$LatestRelease" ]] ; then
     read -r -p "Would you like to update? y/[n]: " SelfUpdate
     [[ "$SelfUpdate" =~ [yY] ]] && self_update
   else
-    [[ -n "$Notify" ]] && { [[ $(type -t dockcheck_notification) == function ]] && dockcheck_notification "$VERSION" "$LatestRelease" "$LatestChanges" ; }
+    [[ -n "$Notify" ]] && { [[ $(type -t dockcheck_notification) == function ]] && dockcheck_notification "$VERSION" "$LatestRelease" "$LatestChanges" || printf "Could not source notification function.\n" ; }
   fi
 fi
 
@@ -182,7 +188,7 @@ IFS=',' read -r -a Excludes <<< "$Exclude" ; unset IFS
 binary_downloader() {
   BinaryName="$1"
   BinaryUrl="$2"
-  case "$(uname --machine)" in
+  case "$(uname -m)" in
     x86_64|amd64) architecture="amd64" ;;
     arm64|aarch64) architecture="arm64";;
     *) printf "\n%bArchitecture not supported, exiting.%b\n" "$c_red" "$c_reset" ; exit 1;;
@@ -197,49 +203,49 @@ binary_downloader() {
 
 distro_checker() {
   if [[ -f /etc/arch-release ]] ; then PkgInstaller="pacman -S"
-  elif [[ -f /etc/redhat-release ]] ; then PkgInstaller="dnf install"
+  elif [[ -f /etc/redhat-release ]] ; then PkgInstaller="sudo dnf install"
-  elif [[ -f /etc/SuSE-release ]] ; then PkgInstaller="zypper install"
+  elif [[ -f /etc/SuSE-release ]] ; then PkgInstaller="sudo zypper install"
-  elif [[ -f /etc/debian_version ]] ; then PkgInstaller="apt-get install"
+  elif [[ -f /etc/debian_version ]] ; then PkgInstaller="sudo apt-get install"
+  elif [[ $(uname -s) == "Darwin" ]] ; then PkgInstaller="brew install"
   else PkgInstaller="ERROR" ; printf "\n%bNo distribution could be determined%b, falling back to static binary.\n" "$c_yellow" "$c_reset"
   fi
 }
 
-# Dependency check for jq in PATH or directory
+# Dependency check + installer function
-if [[ $(command -v jq) ]]; then jqbin="jq" ;
+dependency_check() {
-elif [[ -f "$ScriptWorkDir/jq" ]]; then jqbin="$ScriptWorkDir/jq" ;
+  AppName="$1"
-else
+  AppVar="$2"
-  printf "%s\n" "Required dependency 'jq' missing, do you want to install it?"
+  AppUrl="$3"
-  read -r -p "y: With packagemanager (sudo). / s: Download static binary. y/s/[n] " GetJq
+  if [[ $(command -v $AppName) ]]; then export $AppVar="$AppName" ;
-  GetJq=${GetJq:-no} # set default to no if nothing is given
+  elif [[ -f "$ScriptWorkDir/$AppName" ]]; then export $AppVar="$ScriptWorkDir/$AppName" ;
-  if [[ "$GetJq" =~ [yYsS] ]] ; then
+  else
-    [[ "$GetJq" =~ [yY] ]] && distro_checker
+    printf "%s\n" "Required dependency '$AppName' missing, do you want to install it?"
-    if [[ -n "$PkgInstaller" && "$PkgInstaller" != "ERROR" ]] ; then
+    read -r -p "y: With packagemanager (sudo). / s: Download static binary. y/s/[n] " GetBin
-      (sudo $PkgInstaller jq) ; PkgExitcode="$?"
+    GetBin=${GetBin:-no} # set default to no if nothing is given
-      [[ "$PkgExitcode" == 0 ]] && jqbin="jq" || printf "\n%bPackagemanager install failed%b, falling back to static binary.\n" "$c_yellow" "$c_reset"
+    if [[ "$GetBin" =~ [yYsS] ]] ; then
+      [[ "$GetBin" =~ [yY] ]] && distro_checker
+      if [[ -n "$PkgInstaller" && "$PkgInstaller" != "ERROR" ]] ; then
+        [[ $(uname -s) == "Darwin" && "$AppName" == "regctl" ]] && AppName="regclient"
+        ($PkgInstaller $AppName) ; PkgExitcode="$?" && AppName="$1"
+        if [[ "$PkgExitcode" == 0 ]] ; then { export $AppVar="$AppName" && printf "\n%b$AppName installed.%b\n" "$c_green" "$c_reset"; }
+        else printf "\n%bPackagemanager install failed%b, falling back to static binary.\n" "$c_yellow" "$c_reset"
+        fi
+      fi
+      if [[ "$GetBin" =~ [sS] || "$PkgInstaller" == "ERROR" || "$PkgExitcode" != 0 ]] ; then
+          binary_downloader "$AppName" "$AppUrl"
+          [[ -f "$ScriptWorkDir/$AppName" ]] && { export $AppVar="$ScriptWorkDir/$1" && printf "\n%b$AppName downloaded.%b\n" "$c_green" "$c_reset"; }
+      fi
+    else printf "\n%bDependency missing, exiting.%b\n" "$c_red" "$c_reset" ; exit 1 ;
     fi
-    if [[ "$GetJq" =~ [nN] || "$PkgInstaller" == "ERROR" || "$PkgExitcode" != 0 ]] ; then
-        binary_downloader "jq" "https://github.com/jqlang/jq/releases/latest/download/jq-linux-TEMP"
-        [[ -f "$ScriptWorkDir/jq" ]] && jqbin="$ScriptWorkDir/jq"
-    fi
-  else printf "\n%bDependency missing, exiting.%b\n" "$c_red" "$c_reset" ; exit 1 ;
   fi
-fi
+  # Final check if binary is correct
-# Final check if binary is correct
+  [[ "$1" == "jq" ]] && VerFlag="--version"
-$jqbin --version &> /dev/null  || { printf "%s\n" "jq is not working - try to remove it and re-download it, exiting."; exit 1; }
+  [[ "$1" == "regctl" ]] && VerFlag="version"
+  ${!AppVar} $VerFlag &> /dev/null  || { printf "%s\n" "$AppName is not working - try to remove it and re-download it, exiting."; exit 1; }
+}
 
-# Dependency check for regctl in PATH or directory
+dependency_check "regctl" "regbin" "https://github.com/regclient/regclient/releases/latest/download/regctl-linux-TEMP"
-if [[ $(command -v regctl) ]]; then regbin="regctl" ;
+dependency_check "jq" "jqbin" "https://github.com/jqlang/jq/releases/latest/download/jq-linux-TEMP"
-elif [[ -f "$ScriptWorkDir/regctl" ]]; then regbin="$ScriptWorkDir/regctl" ;
-else
-  read -r -p "Required dependency 'regctl' missing, do you want it downloaded? y/[n] " GetRegctl
-  if [[ "$GetRegctl" =~ [yY] ]] ; then
-    binary_downloader "regctl" "https://github.com/regclient/regclient/releases/latest/download/regctl-linux-TEMP"
-    [[ -f "$ScriptWorkDir/regctl" ]] && regbin="$ScriptWorkDir/regctl"
-  else printf "\n%bDependency missing, exiting.%b\n" "$c_red" "$c_reset" ; exit 1 ;
-  fi
-fi
-# Final check if binary is correct
-$regbin version &> /dev/null  || { printf "%s\n" "regctl is not working - try to remove it and re-download it, exiting."; exit 1; }
 
 # Check docker compose binary
 if docker compose version &> /dev/null ; then DockerBin="docker compose" ;
@@ -283,31 +289,70 @@ if [[ $t_out ]]; then
 else t_out=""
 fi
 
-# Check the image-hash of every running container VS the registry
+check_image() {
-for i in $(docker ps $Stopped --filter "name=$SearchName" --format '{{.Names}}') ; do
+  i="$1"
-  ((RegCheckQue+=1))
+  local Excludes=($Excludes_string)
-  progress_bar "$RegCheckQue" "$ContCount"
+  for e in "${Excludes[@]}" ; do 
-  # Looping every item over the list of excluded names and skipping
+    if [[ "$i" == "$e" ]]; then
-  for e in "${Excludes[@]}" ; do [[ "$i" == "$e" ]] && continue 2 ; done
+      echo Skip $i
+      return
+    fi
+  done
+
+  local NoUpdates GotUpdates GotErrors
   ImageId=$(docker inspect "$i" --format='{{.Image}}')
   RepoUrl=$(docker inspect "$i" --format='{{.Config.Image}}')
   LocalHash=$(docker image inspect "$ImageId" --format '{{.RepoDigests}}')
+
   # Checking for errors while setting the variable
   if RegHash=$(${t_out} $regbin -v error image digest --list "$RepoUrl" 2>&1) ; then
     if [[ "$LocalHash" = *"$RegHash"* ]] ; then
-      NoUpdates+=("$i")
+      echo NoUpdates "$i"
     else
       if [[ -n "$DaysOld" ]] && ! datecheck ; then
-        NoUpdates+=("+$i ${ImageAge}d")
+        echo NoUpdates "+$i ${ImageAge}d"
       else
-        GotUpdates+=("$i")
+        echo GotUpdates "$i"
       fi
     fi
   else
     # Here the RegHash is the result of an error code
-    GotErrors+=("$i - ${RegHash}")
+    echo GotErrors "$i - ${RegHash}"
   fi
-done
+}
+
+# Make required functions and variables available to subprocesses
+export -f check_image datecheck
+export Excludes_string="${Excludes[@]}" # Can only export scalar variables
+export t_out regbin RepoUrl DaysOld
+
+# Check for POSIX xargs with -P option, fallback without async 
+if (echo "test" | xargs -P 2 >/dev/null 2>&1) && [[ "$MaxAsync" != 0 ]]; then
+  XargsAsync="-P $MaxAsync"
+else
+  XargsAsync=""
+  [[ "$MaxAsync" != 0 ]] && printf "%bMissing POSIX xargs, consider installing 'findutils' for asynchronous lookups.%b\n" "$c_red" "$c_reset"
+fi
+
+# Asynchronously check the image-hash of every running container VS the registry
+while read -r line; do
+  ((RegCheckQue+=1))
+  progress_bar "$RegCheckQue" "$ContCount"
+
+  Got=${line%% *}  # Extracts the first word (NoUpdates, GotUpdates, GotErrors)
+  item=${line#* }
+
+  case "$Got" in
+    NoUpdates) NoUpdates+=("$item") ;;
+    GotUpdates) GotUpdates+=("$item") ;;
+    GotErrors) GotErrors+=("$item") ;;
+    Skip) ;;
+    *) echo "Error! Unexpected output from subprocess: ${line}" ;;
+  esac
+done < <( \
+  docker ps $Stopped --filter "name=$SearchName" --format '{{.Names}}' | \
+  xargs ${XargsAsync} -I {} bash -c 'check_image "{}"' \
+)
 
 # Sort arrays alphabetically
 IFS=$'\n'
@@ -317,7 +362,7 @@ unset IFS
 
 # Run the prometheus exporter function
 if [ -n "$CollectorTextFileDirectory" ] ; then
-  source "$ScriptWorkDir"/addons/prometheus/prometheus_collector.sh && prometheus_exporter ${#NoUpdates[@]} ${#GotUpdates[@]} ${#GotError[@]}
+  source "$ScriptWorkDir"/addons/prometheus/prometheus_collector.sh && prometheus_exporter ${#NoUpdates[@]} ${#GotUpdates[@]} ${#GotErrors[@]}
 fi
 
 # Define how many updates are available
@@ -412,3 +457,4 @@ else
 fi
 
 exit 0
+