dhcpcd and dhcpcd-run-hooks on Raspbian

Written by pmd - - no comments

If you are using Raspbian GNU/Linux 9 (stretch), then you are most likely using dhcpcd as your DHCP and DHCPv6 client. How to know your OS version? Here :

$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
  • dhcpcd gets the host information (IP address, routes, etc) from a DHCP server and configures the network interface of the machine on which it is running. Manual of DHCPCD(8).
  • dhcpcd-run-hooks is a DHCP client configuration script. It is used by dhcpcd to run any system and user defined hook scripts. Manual of DHCPCD-RUN-HOOKS(8).

dhcpcd

It seems that to restart all network interfaces this works :

$ sudo systemctl restart dhcpcd.service

To understand what happened, better read dhcpcd logs, here is what we can see:

$ journalctl -u dhcpcd
Dec 21 15:38:04 raspberrypi systemd[1]: Stopping dhcpcd on all interfaces...
Dec 21 15:38:04 raspberrypi dhcpcd[15174]: sending signal TERM to pid 14888
Dec 21 15:38:04 raspberrypi dhcpcd[15174]: waiting for pid 14888 to exit
Dec 21 15:38:04 raspberrypi dhcpcd[14888]: received SIGTERM, stopping
Dec 21 15:38:04 raspberrypi dhcpcd[14888]: eth0: removing interface
Dec 21 15:38:04 raspberrypi dhcpcd[14888]: wlan0: removing interface
Dec 21 15:38:04 raspberrypi dhcpcd[15174]: sending signal TERM to pid 14888
Dec 21 15:38:04 raspberrypi dhcpcd[15174]: waiting for pid 14888 to exit
Dec 21 15:38:05 raspberrypi systemd[1]: Stopped dhcpcd on all interfaces.
Dec 21 15:38:05 raspberrypi systemd[1]: Starting dhcpcd on all interfaces...
Dec 21 15:38:05 raspberrypi dhcpcd[15180]: dev: loaded udev
Dec 21 15:38:05 raspberrypi systemd[1]: Started dhcpcd on all interfaces.
Dec 21 15:38:05 raspberrypi dhcpcd-run-hooks[15203]: wlan0: starting wpa_supplicant
Dec 21 15:38:05 raspberrypi dhcpcd[15185]: eth0: waiting for carrier
Dec 21 15:38:05 raspberrypi dhcpcd[15185]: wlan0: waiting for carrier
Dec 21 15:38:06 raspberrypi dhcpcd[15185]: wlan0: carrier acquired
Dec 21 15:38:06 raspberrypi dhcpcd[15185]: DUID 00:01:00:01:21:b0:de:63:b8:27:eb:4d:6c:87
Dec 21 15:38:06 raspberrypi dhcpcd[15185]: wlan0: IAID eb:5d:6d:97
Dec 21 15:38:06 raspberrypi dhcpcd[15185]: wlan0: adding address fe80::2882:e851:cf0c:9242
Dec 21 15:38:06 raspberrypi dhcpcd[15185]: wlan0: soliciting an IPv6 router
Dec 21 15:38:06 raspberrypi dhcpcd[15185]: wlan0: rebinding lease of 192.168.254.200
Dec 21 15:38:06 raspberrypi dhcpcd[15185]: wlan0: leased 192.168.254.200 for 86400 seconds
Dec 21 15:38:06 raspberrypi dhcpcd[15185]: wlan0: adding route to 192.168.254.0/24
Dec 21 15:38:06 raspberrypi dhcpcd[15185]: wlan0: adding default route via 192.168.254.1
Dec 21 15:38:19 raspberrypi dhcpcd[15185]: wlan0: no IPv6 Routers available

Source

dhcpcd-run-hooks

dhcpcd runs /lib/dhcpcd/dhcpcd-run-hooks. This script runs each script found in /lib/dhcpcd/dhcpcd-hooks in a lexical order.
dhcpcd-run-hooks is used by dhcpcd to run any system and user defined hook scripts. System hook scripts are found in /lib/dhcpcd/dhcpcd-hooks.
There is a list of reasons why dhcpcd-run-hooks could be invoked, such as BOUND (= dhcpcd obtained a new lease from a DHCP server = computer has an IP assigned on the interface).
Many variables will then be set, along with any protocol supplied ones, such as :

  • $interface : the name of the interface
  • $reason : reason why dhcpcd-run-hooks was invoked
  • $if_up : true if the interface is up, otherwise false.

To know what is the current IPv4 address after an interface BOUNDed dhcpcd obtained a new lease from a DHCP server.:

ifconfig $interface | grep "inet " | awk '{print $2}'

I created /lib/dhcpcd/dhcpcd-hooks/60-test_by_pmd in order to play with this functionality. Shell script I used in order to check IP after new DHCP lease was obtained (writing output in /home/pi/test.txt in order to see what happened afterward):

if [ "$reason" = "BOUND" ] && [ "$if_up" = "true" ]; then
     echo "$(date +"%d-%m-%y %H:%M:%S ") $reason $interface IP:$if_up DOWN:$if_down" >> /home/pi/test.txt
     ifconfig $interface | grep "inet " | awk '{print $2}' >> /home/pi/test.txt
fi

Sources : both manuals.

AUTO-Update of openvpn configuration

Written by pmd - - no comments

It happens that NordVPN openvpn configuration files don't work anymore after a while.

I made a bash script to update the configuration easy and fast:

#!/bin/bash
# Mise a jour de la configuration NordVPN
# Dossier temporaire
DOSSIER_TEMP="/home/pi/NordVPNautoUpdate"
#Dossier des configurations openvpn
DOSSIER_OPENVPN="/etc/openvpn"
CONF_OPENVPN="server.conf"
AUTOLOGIN_OPENVPN="login.txt"
# Pays
PAYS="ua"
# TCP ou UDP?
PROTO="udp"
# Archive de configuration NordVPN
ARCHIVE_NORDVPN="https://downloads.nordcdn.com/configs/archives/servers/ovpn.zip"
FICHIER=$(echo -n $ARCHIVE_NORDVPN | awk -F "/" '{printf $NF}')

# On supprime et on crée le dossier temp
rm -rf $DOSSIER_TEMP
mkdir $DOSSIER_TEMP

# On télécharge tous les fichiers de conf NordVPN
wget $ARCHIVE_NORDVPN -P $DOSSIER_TEMP

# On dézip l'archive téléchargée
mkdir $DOSSIER_TEMP/temp
unzip -q "$DOSSIER_TEMP/$FICHIER" -d $DOSSIER_TEMP/temp

# On garde que les conf d'un certain pays
mv $DOSSIER_TEMP/temp/ovpn_$PROTO/$PAYS[0-9]*$PROTO* $DOSSIER_TEMP/
rm -r $DOSSIER_TEMP/temp

# On supprime les conf double-VPN et l'archive zip
NbConf=0
for ConfOpenVPN in $DOSSIER_TEMP/*; do
        if [[ $ConfOpenVPN != */$PAYS[0-9]*$PROTO* ]]; then
                rm $ConfOpenVPN
        else
                # On compte le nombre de conf restantes
                NbConf=$((NbConf+1))
        fi
done

# On choisi une conf au hasard
NbConf=$((1 + RANDOM % $NbConf))
COMPTEUR=0
for ConfOpenVPN in $DOSSIER_TEMP/*; do
        COMPTEUR=$((COMPTEUR+1))
        if [[ $COMPTEUR -eq $NbConf ]]; then
            # On modifie la configuration pour authentification automatique
            sed -i "s@auth-user-pass@auth-user-pass $DOSSIER_OPENVPN/$AUTOLOGIN_OPENVPN@" $ConfOpenVPN
            # On déplace la conf dans le dossier openvpn
            sudo cp -f $ConfOpenVPN $DOSSIER_OPENVPN
            sudo cp -f $ConfOpenVPN $DOSSIER_OPENVPN/$CONF_OPENVPN
                        # On informe
                        echo "Configuration installée :"
                        echo $ConfOpenVPN | awk -F "/" '{printf $NF}'
                        echo ""
        fi
done

# On supprime le dossier temporaire
rm -rf $DOSSIER_TEMP
# On redémarre openvpn avec la nouvelle configuration
sudo service openvpn restart

Then simply execute the script:

bash NordVPNautoUpdate.sh

Unlock standalone Airsonic database

Written by pmd - - no comments

If your airsonic is stucked for some reasons such as Liquibase Update Failed and Waiting for changelog lock...

liquibase.exception.LockException: Could not acquire change log lock. Currently locked by 192.168.254.200 (192.168.254.200) since 18/11/18 16:39
  1. Stop Airsonic for sure :
    sudo service airsonic stop
  2. Read logs /var/airsonic/airsonic.log :
    nano /var/airsonic/airsonic.log
  3. Backup your database folder /var/airsonic/db :
    sudo cp /var/airsonic/db /var/airsonic/db_backup
  4. Edit /var/airsonic/db/airsonic.script :
    sudo nano /var/airsonic/db/airsonic.script
    • remove all lines where you can find DATABASECHANGELOGLOCK by example :
      • CREATE MEMORY TABLE DATABASECHANGELOGLOCK.....
      • INSERT INTO DATABASECHANGELOGLOCK VALUES(1,TRUE,'2018-11-18 16:39:56.266000000','192.168.254.200 (192.168.254.200)')
  5. Save file.
  6. Start Airsonic :
    sudo service airsonic start
  7. Check Airsonic status :
    sudo service airsonic status

If you wanna reset your Airsonic music server, just delete the database /var/airsonic/db ...

Source 1, 2, 3.

Update Airsonic on Raspberry Pi 3

Written by pmd - - no comments

How to update a standalone Airsonic

  • Stop the current running Airsonic:
$ sudo systemctl stop airsonic.service
$ cd /var/airsonic/
$ sudo mv airsonic.war airsonic.war.10.1.1.bak
$ sudo wget https://github.com/airsonic/airsonic/releases/download/v10.1.2/airsonic.war --output-document=/var/airsonic/airsonic.war
  • Put good rights to all folder (new airsonic.war is not owned by airsonic), and verify it has been applied:
$ sudo chown -R airsonic:airsonic /var/airsonic
$ ls
total 151M
-rw-r--r--  1 airsonic airsonic 760K Oct 11 19:52 airsonic.log
-rw-r--r--  1 airsonic airsonic  11M Oct  3 00:20 airsonic.log.1
-rw-r--r--  1 airsonic airsonic  978 Oct 11 04:00 airsonic.properties
-rw-r--r--  1 airsonic airsonic  70M Jul 28 22:10 airsonic.war
-rw-r--r--  1 airsonic airsonic  70M Dec 16  2017 airsonic.war.10.1.1.bak
drwxr-xr-x  2 airsonic airsonic 4.0K Oct 11 19:49 db
drwxr-xr-x  2 airsonic airsonic  20K Sep 14 16:08 lastfmcache
drwxr-xr-x  7 airsonic airsonic 4.0K Jul 22 10:36 lucene2
-rw-r--r--  1 airsonic airsonic  741 Oct 11 19:50 rollback.sql
drwxr-xr-x 16 airsonic airsonic 4.0K Oct  4 17:45 thumbs
drwxr-xr-x  2 airsonic airsonic 4.0K Jul 22 10:54 transcode
  • Start updated Airsonic and restart proxy server:
$ sudo systemctl start airsonic.service
$ sudo service nginx restart

Nothing too hard, basically, you need to replace the airsonic.war by the updated one.

Unstable Wifi connection to Raspberry Pi 3

Written by pmd - - no comments

Original post

I am loosing connection to my Raspberry Pi 3 in a very much random way.

I recently replaced Subsonic by Airsonic, and connection to Pi seemed to be much more stable since that exact day. But today, it happened one more time... as I was expecting! Time to troubleshoot has come.

Since I am using 2 tables to root my traffic, one of them is not updated on some interface up/down events. I made a small script called by cron every 5 minutes to understand things and rebooting Pi once a day to get the access back:

#!/usr/bin/env bash

# Script to monitor and restart wireless access point when needed

# Logging
date >> NetworkCheckLog
echo "wap_check_pmd: Checking if network connection is down..." >> NetworkCheckLog
echo "$ ip route show table main" >> NetworkCheckLog
ip route show table main >> NetworkCheckLog
echo "$ ip route show table 42" >> NetworkCheckLog
ip route show table 42 >> NetworkCheckLog

# First make sure we can resolve google, otherwise 'ping -w' would hang
if ! $(host -W5 www.google.com > /dev/null 2>&1); then
        #Make a note in syslog
        logger "wap_check_pmd: Network connection is down..."
        date >> NetworkCheckLog
        ifconfig >> NetworkCheckLog
        echo "wap_check_pmd: Network connection is down..." >> NetworkCheckLog
        exit
fi

Time when it got fucked:

Mon  6 Aug 08:55:01 CEST 2018
wap_check_pmd: Checking if network connection is down...
$ ip route show table main
0.0.0.0/1 via 10.8.7.1 dev tun0
default via 192.168.1.1 dev wlan0 src 192.168.1.200 metric 303
10.8.8.0/24 dev tun0 proto kernel scope link src 10.8.7.12
31.192.112.170 via 192.168.1.1 dev wlan0
128.0.0.0/1 via 10.8.7.1 dev tun0
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.200 metric 303
$ ip route show table 42
default via 192.168.1.1 dev wlan0 src 192.168.1.200 metric 303
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.200 metric 303

Mon  6 Aug 09:00:01 CEST 2018
wap_check_pmd: Checking if network connection is down...
$ ip route show table main
0.0.0.0/1 via 10.8.7.1 dev tun0
default via 192.168.1.1 dev wlan0 src 192.168.1.200 metric 303
10.8.8.0/24 dev tun0 proto kernel scope link src 10.8.7.12
128.0.0.0/1 via 10.8.7.1 dev tun0
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.200 metric 303
$ ip route show table 42
Mon  6 Aug 09:00:15 CEST 2018
wap_check_pmd: Network connection is down...

I then checked /var/log/syslog:

Aug  6 08:59:26 raspberrypi dhcpcd[382]: wlan0: carrier lost
[...]
Aug  6 08:59:26 raspberrypi dhcpcd[382]: wlan0: deleting default route via 192.168.1.1
Aug  6 08:59:26 raspberrypi dhcpcd[382]: wlan0: deleting route to 192.168.1.0/24
[...]
Aug  6 08:59:51 raspberrypi dhcpcd[382]: wlan0: carrier acquired

After Googling, I found an interesting article. Which provide interesting tips.

  1. Use this command to read the current power saving mode of your Pi:
    $ sudo iw wlan0 get power_save
    Power save: on
  2. And this one to turn power_save off:
    $ sudo iw wlan0 set power_save off
  3. To make this permanent add the following line to /etc/rc.local:
    /sbin/iw dev wlan0 set power_save off

We will see if it helps :)

Update 27/10/2018

I choose to turn the power_save off as it originally was. But I came up with this script:

#!/bin/bash
#/home/pi/wap_check
# Script to monitor and restart wireless access point when needed
maxPloss=10 #Maximum percent packet loss before a restart

# Logging
# echo "$(date +"%d-%m-%y %H:%M:%S ") wap_check_pmd: Checking if network connection is down..." >> /home/pi/NetworkCheckLog

# If table 42 is empty, we lost raspberry
TABLE_42_eth0=$(ip route show table 42 | grep eth0)
TABLE_42_wlan0=$(ip route show table 42 | grep wlan0)
# addapt following "if" and TABLE_42_xxx variables above depending on your interfaces
if [[ -z "${TABLE_42_eth0}" ]] || [[ -z "${TABLE_42_wlan0}" ]]; then
  # Loging
  echo  "$(date +"%d-%m-%y %H:%M:%S ") wap_check_pmd: Table 42 is (partialy?) empty! Trying to populate it:" >> /home/pi/NetworkCheckLog
  logger "wap_check_pmd: Table 42 is (partialy?) empty! Trying to populate it."
  # Purge table 42 just in case
  sudo ip route flush table 42
  # Filling up the table 42
  ip route show table main | grep -v tun0 | while read LINE; do sudo ip route add $LINE table 42; done
  sleep 1
  ip route show table 42 >> /home/pi/NetworkCheckLog
fi

# IP cannot be the same, debian-transmission can go only through tun0
IP_VPN=$(sudo -u debian-transmission wget -qO- ifconfig.co)
IP_PUBLIC=$(wget -qO- ifconfig.co)
if [[ "${IP_VPN}" = "${IP_PUBLIC}" ]]; then
  echo "$(date +"%d-%m-%y %H:%M:%S ") wap_check_pmd: Same IP for debian-transmission and pi: $(echo $IP_VPN) and $(echo $IP_PUBLIC)" >> /home/pi/NetworkCheckLog
  logger "wap_check_pmd: Same IP for debian-transmission and pi..."
  if [[ -n "${IP_PUBLIC}"  ]]; then
    echo "$(date +"%d-%m-%y %H:%M:%S ") wap_check_pmd: Rebooting. [1]" >> /home/pi/NetworkCheckLog
        logger "wap_check_pmd: Rebooting. [1]"
    sleep 1
    sudo /sbin/reboot
        exit
  fi
fi

# Make sure we can resolve google, otherwise 'ping -w' would hang
if ! $(host -W5 www.google.com > /dev/null 2>&1); then
    # Make a note in syslog
    logger "wap_check_pmd: Network connection is down..."
        echo "$(date +"%d-%m-%y %H:%M:%S ") wap_check_pmd: Rebooting. [2]" >> /home/pi/NetworkCheckLog
        logger "wap_check_pmd: Rebooting. [2]"
        # Restart device
        sleep 1
        sudo /sbin/reboot
    exit
fi

# Initialize to a value that would force a restart
# (just in case ping gives an error and ploss doesn't get set)
ploss=101
# now ping google for 10 seconds and count packet loss
ploss=$(ping -q -w10 www.google.com | grep -o "[0-9]*%" | tr -d %) > /dev/null 2>&1
if [ "$ploss" -gt "$maxPloss" ]; then
    logger "wap_check_pmd: Packet loss ($ploss%) exceeded $maxPloss"
    echo "$(date +"%d-%m-%y %H:%M:%S ") wap_check_pmd: Packet loss ($ploss%) exceeded $maxPloss" >> /home/pi/NetworkCheckLog
    # restart_networking
fi

I run this script every 5 minutes, line in my crontab file:

*/5 * * * * /bin/bash /home/pi/NetworkCheck.sh

Log file looks like this:

22-09-18 22:25:02  wap_check_pmd: Table 42 is empty! Trying to populate it:
22-09-18 22:26:55  wap_check_pmd: Same IP for debian-transmission and pi:  and
22-09-18 22:27:09  wap_check_pmd: Rebooting. [2]
24-09-18 23:20:16  wap_check_pmd: Rebooting. [2]
25-09-18 22:26:10  wap_check_pmd: Packet loss (30%) exceeded 10
25-09-18 23:11:07  wap_check_pmd: Packet loss (16%) exceeded 10
04-10-18 02:55:12  wap_check_pmd: Packet loss (70%) exceeded 10
04-10-18 15:50:01  wap_check_pmd: Table 42 is empty! Trying to populate it:
default via 192.168.1.1 dev wlan0 src 192.168.1.200 metric 303
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.200 metric 303
04-10-18 19:20:12  wap_check_pmd: Packet loss (20%) exceeded 10
04-10-18 20:40:27  wap_check_pmd: Packet loss (30%) exceeded 10
04-10-18 20:55:21  wap_check_pmd: Packet loss (16%) exceeded 10
05-10-18 20:20:01  wap_check_pmd: Table 42 is empty! Trying to populate it:
default via 192.168.1.1 dev wlan0 src 192.168.1.200 metric 303
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.200 metric 303
05-10-18 20:52:25  wap_check_pmd: Packet loss (30%) exceeded 10
06-10-18 15:15:15  wap_check_pmd: Packet loss (30%) exceeded 10
07-10-18 08:55:02  wap_check_pmd: Same IP for debian-transmission and pi:  and
16-10-18 19:21:53  wap_check_pmd: Same IP for debian-transmission and pi:  and
16-10-18 19:22:07  wap_check_pmd: Rebooting. [2]
27-10-18 11:30:20  wap_check_pmd: Packet loss (20%) exceeded 10

youtube-dl, now yt-dlp

Written by pmd - - no comments

youtube-dl is a usefull packet to download youtube video through command line. GitHub.

List what to download

First step is to list what youtube video you wanna download and convert to mp3 file:

2 files to put on a web server which are:

  1. liste.txt which must be chmod 777 (index.php will write links in this file)
  2. index.php
<!DOCTYPE html>
<html>
<head>
<title>youtube-dl</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
body {
width: 550px;
font-size: 75%;
}
p {
font-family: "Lucida Console", "Courier New", monospace;
white-space: nowrap
}
li {
font-family: "Lucida Console", "Courier New", monospace;
white-space: nowrap
}
hr {
border: 0;
border-bottom: 1px dashed #CCCCCC;
background: #FFFFFF;
}
input[type=text], select {
width: 80%;
padding: 12px 20px;
margin: 8px 0;
display: inline-block;
border: 1px solid #ccc;
border-radius: 4px;
box-sizing: border-box;
}

input[type=submit] {
width: 80%;
background-color: #4CAF50;
color: white;
padding: 14px 20px;
margin: 8px 0;
border: none;
border-radius: 4px;
cursor: pointer;
}

input[type=submit]:hover {
background-color: #45a049;
}

div {
border-radius: 5px;
background-color: #f2f2f2;
padding: 20px;
overflow:auto;
width: 500px;
}
</style>
</head>
<body>
<span style="white-space: nowrap">
<div>
<p>
<?php
if(isset($_POST['field1']))
{
$data = $_POST['field1'] . "\r\n";
$ret = file_put_contents('liste.txt', $data, FILE_APPEND | LOCK_EX);
if($ret === false) {
echo "There was an error writing this file.";
}
else {
echo "$ret bytes written to file.";
}
}
else {
echo "No post data to process.";
}
?>
</p>
</div>
<hr>

<div>

<p>Format to be respected:</p>
<ol>
<li>« https://www.youtube.com/watch?v=1YRW1QRKTBc »</li>
<li>« https://youtu.be/9cBtJYI6itg »</li>
<li>« https://soundcloud.com/hungry-music/nto-alter-ego »</li>
</ol>

<form action="index.php" method="POST">
<input type="text" id="fname" name="field1" placeholder="https://www.youtube.com/watch?v=1YRW1QRKTBc"><br />
<input type="submit" value="Submit">
</form>
</div>

<hr>

<div>
<p>Next link(s) to be downloaded and saved to mp3 files:</p>
<p>
<?php
echo nl2br(file_get_contents( "liste.txt" ));
?>
</p>
</div>

<hr>

<div>
<p>
Logs (last 200 lines): <br />
<?php
//echo nl2br(file_get_contents( "/path/to/history.txt" ));
echo nl2br(trim(implode("", array_slice(file("/path/to/history.txt"), -200))));
?>
</p>
</div>
</body>
</html>

Sources 1, 2

Secure input form

https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/
http://www.htaccesstools.com/htpasswd-generator/

location ^~ /youtube-dl {
     auth_basic           "Иди на хуй мозгоёб!!!";
     auth_basic_user_file /path/to/.htpasswd;
     location ~ \.php$ {
         include snippets/fastcgi-php.conf;
         fastcgi_pass unix:/run/php/php7.0-fpm.sock;
         }
 }

Restart nginx:

$ sudo service nginx restart

Download script

He depends on:

  • /path/to/liste_en_cours.txt
  • /path/to/history.txt
#!/usr/bin/bash
filename="$1"
liste_en_cours=/path/to/Downloads/youtube_auto/liste_en_cours.txt

cp -f $1 $liste_en_cours
sleep 1
echo -n > $1

echo "  " >> /path/to/Downloads/youtube_auto/history.txt
date >> /path/to/Downloads/youtube_auto/history.txt

while read -r line
do
    name="$(echo $line | sed 's/\r//g')"
    echo -n "$name | " >> /path/to/Downloads/youtube_auto/history.txt
    page="$(wget -O - $name)"
    echo $page | sed -n 's/.*<title>\(.*\)<\/title>.*/\1/ip;T;q' >> /path/to/Downloads/youtube_auto/history.txt

    /usr/local/bin/youtube-dl -f bestaudio $name --extract-audio --audio-format mp3 -o "/path/to/Downloads/youtube_auto/downloads/%(title)s.%(ext)s" | grep ffmpeg | grep mp3 | sed 's/.*\//===> /' | sed 's/mp3.*/mp3/' >> /path/to/Downloads/youtube_auto/history.txt

done < "$liste_en_cours"

mv -f /path/to/Downloads/youtube_auto/downloads/* /path/to/Music/youtube

#All big files goes to folder Mixes
for file in /path/to/Music/youtube/*; do
    SIZE="$(stat --printf="%s" "$file")"
    if (( $SIZE > 25000000 )); then
        mv -f "$file" /path/to/Music/Mixes
    fi
done

Source homemade. Now execute it with cron job by example.

Switching to yt-dlp

youtube-dl is notmaintained anymore. It now fails with "Unable to extract uploader id" error using version 2021.12.17. yt-dlp is a youtube-dl fork based on the now inactive youtube-dlc. The main focus of this project is adding new features and patches while also keeping up to date with the original project.

Use SSH to connect to Raspberry Pi. Run the following command to download the latest version of yt-dlp from GitHub repository:

$ sudo wget -qO /usr/local/bin/yt-dlp https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp

Set read and execute permissions for a file:

$ sudo chmod a+rx /usr/local/bin/yt-dlp

Now yt-dlp command will be available for all users as a system-wide command.
We can check yt-dlp version as follows:

yt-dlp --version

We can update yt-dlp to the latest version by using the following command:

sudo yt-dlp -U

So in above script I replace /usr/local/bin/youtube-dl by /usr/local/bin/yt-dlp.

Source.

Install Airsonic on Raspberry Pi 3

Written by pmd - - no comments

Stand-alone WAR installation integrated within systemd

Let's see how to install a stand-alone Airsonic on a Raspberry 3 integrated with Systemd.

First of all, we need Java to be installed:

$ sudo apt-get install openjdk-8-jre

By following these systemd setup instructions, Airsonic will be available at http://localhost:8080/airsonic.

  1. Setup dedicated airsonic user:
    $ sudo useradd airsonic
  2. Setup Airsonic data dir:
    $ sudo mkdir /var/airsonic
    $ sudo chown airsonic /var/airsonic
  3. Download the stand-alone WAR:
    $ sudo wget https://github.com/airsonic/airsonic/releases/download/v10.1.1/airsonic.war --output-document=/var/airsonic/airsonic.war
  4. Setup systemd service:
    $ sudo wget https://raw.githubusercontent.com/airsonic/airsonic/master/contrib/airsonic.service -O /etc/systemd/system/airsonic.service
    $ sudo systemctl daemon-reload
    $ sudo systemctl start airsonic.service
    $ sudo systemctl enable airsonic.service
    $ sudo wget https://raw.githubusercontent.com/airsonic/airsonic/master/contrib/airsonic-systemd-env -O /etc/sysconfig/airsonic

    Note: On Debian systems you need to replace /etc/sysconfig with /etc/default)

  5. Go configure http://localhost:8080/airsonic.

Set up a reverse proxy: nginx

  1. Go edit nginx configuration:
    $ sudo nano /etc/nginx/sites-available/default
  2.         location ^~ /airsonic {
                    proxy_pass http://127.0.0.1:8080;
                    proxy_set_header Host $host;
                    proxy_set_header X-Real-IP $remote_addr;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            }
  3. Restart nginx:
    $ sudo service nginx restart

Now the Airsonic server should be accissible from outside uside http://publicadress.com/airsonic

Source 1 : Stand-alone WAR installation
Source 2 : The url I'm opening is http://publicadress.com/airsonic but it will be redirected to http://publicadress.com/login, which brings up error 404.

Set up the transcoder

$ sudo apt-get install ffmpeg

Then we need to link Airsonic with the transcoder ffmpeg.

$ sudo mkdir /var/airsonic/transcode
$ cd /var/airsonic/transcode
$ sudo ln -s /usr/bin/ffmpeg /var/aisonic/transcode/ffmpeg
$ ls -alh
total 8.0K
drwxr-xr-x 2 airsonic airsonic 4.0K Jul 22 10:54 .
drwxr-xr-x 5 airsonic tomcat8  4.0K Jul 22 10:48 ..
lrwxrwxrwx 1 root     root       15 Jul 22 10:54 ffmpeg -> /usr/bin/ffmpeg
$ sudo chown -R airsonic:airsonic /var/airsonic
$ ls -alh
total 8.0K
drwxr-xr-x 2 airsonic airsonic 4.0K Jul 22 10:54 .
drwxr-xr-x 5 airsonic airsonic 4.0K Jul 22 10:48 ..
lrwxrwxrwx 1 airsonic airsonic   15 Jul 22 10:54 ffmpeg -> /usr/bin/ffmpeg

Interact with server from Android

I use the Ultrasonic application:
ultrasonic

Source 1, 2, 3,

Startup log example

$ sudo systemctl status airsonic
● airsonic.service - Airsonic Media Server
     Loaded: loaded (/etc/systemd/system/airsonic.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-01-18 21:09:56 CET; 20min ago
   Main PID: 538 (java)
      Tasks: 33 (limit: 1597)
        CPU: 3min 15.474s
     CGroup: /system.slice/airsonic.service
             └─538 /usr/bin/java -Xmx700m -Dairsonic.home=/var/airsonic -Dserver.context-path=/airsonic -Dserver.port=8080 -jar /var/airsonic/airsonic.war

Jan 18 21:11:50 raspberrypi java[538]: 2022-01-18 21:11:50.554  INFO --- l.l.StandardLockService                  : Successfully released change log lock
Jan 18 21:11:51 raspberrypi java[538]: 2022-01-18 21:11:51.540  INFO --- o.a.p.service.SettingsService            : Java: 1.8.0_312, OS: Linux
Jan 18 21:11:53 raspberrypi java[538]: 2022-01-18 21:11:53.290  INFO --- org.airsonic.player.Application          : Detected Tomcat web server
Jan 18 21:12:31 raspberrypi java[538]: 2022-01-18 21:12:31.557  INFO --- o.a.p.service.PodcastService             : Automatic Podcast update scheduled to run every 24 hour(s), starting at Tue Jan 18 21:17:31 CET 2022
Jan 18 21:12:33 raspberrypi java[538]: 2022-01-18 21:12:33.840  INFO --- o.a.p.s.search.IndexManager              : Index was found (index version 18).
Jan 18 21:12:33 raspberrypi java[538]: 2022-01-18 21:12:33.900  INFO --- o.a.p.s.MediaScannerService              : Automatic media library scanning scheduled to run every 1 day(s), starting at 2022-01-19T03:00:00.887
Jan 18 21:13:03 raspberrypi java[538]: 2022-01-18 21:13:03.363  INFO --- org.airsonic.player.Application          : Started Application in 154.011 seconds (JVM running for 170.609)
Jan 18 21:17:31 raspberrypi java[538]: 2022-01-18 21:17:31.553  INFO --- o.a.p.service.PodcastService             : Starting scheduled Podcast refresh.
Jan 18 21:17:31 raspberrypi java[538]: 2022-01-18 21:17:31.565  INFO --- o.a.p.service.PodcastService             : Completed scheduled Podcast refresh.
Jan 18 21:18:01 raspberrypi java[538]: 2022-01-18 21:18:01.772  INFO --- o.a.p.f.BootstrapVerificationFilter      : Servlet container: Apache Tomcat/8.5.51
$

NTFS HDD plugged to Raspberry Pi 3

Written by pmd - - no comments

Choose where to mount the disk:

mkdir /home/pi/HDD

Partition is NTFS, so if we wanna write on it:

sudo apt-get install ntfs-3g

To know the UUID of the HDD:

sudo blkid

Edit how to mount disk partition:

sudo nano /etc/fstab

Add this line and save:

UUID=AADEEA03DEE9C7A1 /home/pi/HDD ntfs-3g uid=1000,gid=1000,auto,umask=000 0 0

Mount  all filesystems mentioned in fstab to be mounted as indicated :

sudo mount -a

 

Backup ftp server + pluxml on Raspberry

Written by pmd - - no comments

Information from shellhacks.com.

Download

The following command recursively downloads your site with all its files and folders from FTP server and saves them to the current directory.

wget -r -l 0 -nH -X folder_to_skip ftp://user:pass@ftp.server.com

 

option description
user FTP username
pass FTP password
ftp.server.com IP address or domain name of an FTP server
-r Recursive retrieving
-l Maximum recursion depth (0 = unlimit) (default = 5)
-nH Disable generation of host-prefixed directories
-X exclude a list of directories

Backup

Now you can compress the folder with your site as follows:

tar -czf site-backup-$(date +%Y%m%d-%H%M%S).tar.gz yoursite.com

Install pluxml on raspberry pi

I am using nginx + php7 + pluxml following these 3 links:

  1. https://www.raspberrypi.org/documentation/remote-access/web-server/nginx.md
  2. https://blog.norore.fr/index.php?article13/pluxml-nginx-et-php-7-sont-dans-un-bateau
  3. https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/

To find which php version should be installed, you can search what is available:

$ apt-cache search php7

Install php and nginx (see here to install nginx):

$ sudo apt-get install php7.3-fpm php7.3-xml php7.3-gd

Once all is installed, configure php :

$ sudo nano /etc/php/7.3/fpm/pool.d/www.conf
[www]
user = www-data
group = www-data
listen = /run/php/php7.3-fpm.sock
listen.owner = www-data
listen.group = www-data
;listen.mode = 0660
$ sudo nano /etc/nginx/sites-available/default
server {
        listen 80 default_server;
        listen [::]:80 default_server;

        # Add index.php to the list if you are using PHP
        index index.php index.html index.htm index.nginx-debian.html;

        server_name _;

        location @handler {
                rewrite ^/(.*)$ /index.php? last;
        }

        # pass PHP scripts to FastCGI server
        #
        location ^~ /pluxml_folder {
                auth_basic "admin";
                auth_basic_user_file /etc/apache2/.htpasswd;

                location ~ \.php$ {
                        include snippets/fastcgi-php.conf;
                        fastcgi_pass unix:/run/php/php7.3-fpm.sock;
                }
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        location ~ /(data/configuration|version|update|\.ht) {
                deny all;
        }
}

Then restart both services.

 

Interesting problem : Nginx sucessfully password protects PHP files, but then prompts you to download them

Answer : The problem is a fundamental misunderstanding as to how nginx processes a request. Basically, nginx chooses one location to process a request.
You want nginx to process URIs that begin with /admin in a location block that requires auth_basic. In addition, URIs that end with .php need to be sent to PHP7.
So you need two fastcgi blocks, one to process normal PHP files and one to process restricted PHP files.

 

SSL certificate with no-ip.com

SSL Certificate Now Included with No-IP Free Dynamic DNS.

Tutorial #0: How to Configure the TrustCor Standard DV SSL
Tutorial #1: NGINX Server setup for TrustCor SSL

Generate the private key and CSR from the NGINX server by running this command:

$ sudo openssl req -nodes -days 365 -newkey rsa:2048 -keyout nginx1-trustcor.key -out nginx1-trustcor.csr
Ignoring -days; not generating a certificate
Generating a RSA private key
.....................................................+++++
.................................................................+++++
writing new private key to 'nginx1-trustcor.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:FR
State or Province Name (full name) [Some-State]:France
Locality Name (eg, city) []:none
Organization Name (eg, company) [Internet Widgits Pty Ltd]:none
Organizational Unit Name (eg, section) []:none
Common Name (e.g. server FQDN or YOUR name) []:DOMAIN_NAME.ddns.net
Email Address []:none
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:none
$

 

Troubleshooting

Sometimes, I've had the situation where http requests from browsers are being served old versions of the PHP files, missing updates to the PHP scripts that have been uploaded to the server.

To solve this issue, I had to modify the php.ini file by setting :

opcache.enable=0

To find your php.ini location, run this php file on your server:

<?php
phpinfo();
?>

Then to reload php.ini file, restart php-fpm:

$ sudo systemctl stop php7.4-fpm
$ sudo systemctl start php7.4-fpm
$

Source 1, 2

Transmission exclusively using VPN

Written by pmd - - no comments

I would like to have my transmission client to exchange data only through a VPN.

Firewall

First of all, it is necessary to set some rules so the 'debian-transmission' user (running transmission) can only route through the VPN:

# 'debian-transmission' user only accepted through 'tun0'
# after these 3 first rules, transmission cannot access internet
$ sudo iptables -A OUTPUT -m owner --uid-owner 'debian-transmission' -o tun0 -j ACCEPT
$ sudo iptables -A OUTPUT -m owner --uid-owner 'debian-transmission' -o lo -j ACCEPT
$ sudo iptables -A OUTPUT -m owner --uid-owner 'debian-transmission' -j REJECT
# marking all packets used by users different than 'debian-transmission' with '42'
$ sudo iptables -t mangle -A OUTPUT -m owner ! --uid-owner 'debian-transmission' -j MARK --set-mark 42
$ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$ sudo sysctl -w 'net.ipv4.conf.eth0.rp_filter=2' # reverse path filtering

FYI: These rules will be set only until next reboot.

FYI: To get a complete presentation of all the netfilter rules, you can use these commands :

$ sudo iptables -vL -t filter
$ sudo iptables -vL -t nat
$ sudo iptables -vL -t mangle
$ sudo iptables -vL -t raw
$ sudo iptables -vL -t security

FYI: To display rules so you can verify there are good :

$ sudo iptables -S ; echo ; sudo iptables -S -t mangle ; echo ; sudo iptables -S -t nat

 

If you are sure that these iptables rules we set are OK, you can make them permanent (resistant to reboot):

$ sudo apt-get install iptables-persistent
$ sudo dpkg-reconfigure iptables-persistent # if already installed

Tap yes to both prompts. Verify it was correcly taken into account especially interfaces :

sudo nano /etc/iptables/rules.v4

Done, these rules are persistent: transmission cannot communicate without an active tun0 interface (VPN).

Routes

The strategy I used is this one:

  • all packets not marked '42' following main route table
  • all packets marked '42' following route table named '42'

Create a systemd .service to create and populate route table '42' when Pi3 starts:

$ sudo nano /etc/systemd/system/copy_route_pmd.service

We will wait to find an IP address containing "192", then we will create the new table "42" and copy all rules from main table to "42" table.

#/etc/systemd/system/copy_route_pmd.service
[Unit]
Description=Copy the route of main table to table 42 at system startup
After = network-online.target
Wants = network-online.target

[Service]
Type=oneshot
#ExecStart=/bin/bash -c "while ! [[ -n $(ifconfig | grep 192) ]]; do sleep 1; done; ip rule add fwmark 42 table 42; ip route show table main | while read LINE; do ip route add $LINE table 42; done"
ExecStart=/bin/bash -c "while ! [[ -n $(ifconfig | grep 192) ]]; do sleep 1; done; ip rule add fwmark 42 table 42; ip route show table main | grep -v tun0 | while read LINE; do ip route add $LINE table 42; done"

[Install]
WantedBy=multi-user.target

We reload systemd to update with the new service we made and make the service execute at system startup.

# Reloading
$ systemctl daemon-reload # Run if *.service file has changed
# Try the new service

$ sudo systemctl start copy_route_pmd.service
# If OK (check tables main and 42), set service to execute at startup
$ sudo systemctl enable copy_route_pmd.service

Now, debian-transmission will use main table to route packets, and users different than debian-transmission will use the table 42.

Link to article to check routes

 

openvpn

Configure openvpn

Make sure that you let openvpn push new rules in main route table in order to use the VPN.

Now that openvpn made up tun0 interface, check that the routes are respected:

$ wget -qO- ifconfig.co
$ sudo -u debian-transmission wget -qO- ifconfig.co

These two commands will return different IP address!

transmission

Install transmission

Verify that transmission-daemon is run by correct user:

$ top -u debian-transmission

Check your visible torrent IP with this: https://torguard.net/checkmytorrentipaddress.php

Access to transmission web interface will require nginx

nginx

Install nginx:

$ sudo apt-get install nginx

Edit the default configuration:

$ sudo nano /etc/nginx/sites-available/default

If the machine does not have an IPv6 address set, you need to comment this line :

listen [::]:80 default_server;

Replace section location by this :

location /transmission {
    proxy_pass http://127.0.0.1:9091;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }

Restart nginx:

$ sudo service nginx restart

Access your interface worldwide using: http://yourIPadress/transmission/web/

DNS leak

To prevent DNS leak and/or not to rely on the router you are connected to the internet, modify the file:

$ sudo nano /etc/resolv.conf.head
#OpenDns Servers
nameserver 208.67.222.222
nameserver 208.67.220.220
#Google Servers
nameserver 8.8.8.8

In fact I want to set the DNS servers directly in the Raspberry Pi, because it allows me to remove the default route to the router in table 42 and keep it as tidy as possible.

Then reboot the Pi 3:

$ sudo reboot

You can now check that these are the first DNS server is use:

pi@raspberrypi:~ $ cat /etc/resolv.conf
# Generated by resolvconf
#OpenDns Servers
nameserver 208.67.222.222
nameserver 208.67.220.220
#Google Servers
nameserver 8.8.8.8
domain home
nameserver 192.168.1.1

From your Pi3, check the DNS leakage from this website: https://dnsleaktest.com/

Rss feed of the category