Skip to content

linux

Basics web scraping using Python3 with BeautifulSoup4 and then convert to Markdown

Basics web scraping using Python3 with BeautifulSoup4 and then converting to Markdown

Basic Python BeautifulSoup4 web scraping and then Markdown

pip install requests
pip install beautifulsoup4
pip install markdownify

import markdownify 

import requests
from bs4 import BeautifulSoup

def beautifulsoup_web_scrape_url(url):
  response = requests.get(url)
  soup = BeautifulSoup(response.content, 'html.parser')
  return str(soup)

url = "https://www.antoniofeijao.com/"

data = beautifulsoup_web_scrape_url(url)

print(data)



# convert html to markdown 
h = markdownify.markdownify(data, heading_style="ATX") 
  
print(h)


f = open("result.txt", "w")
f.write("##result file done. Woops! I have deleted the content!##")
f.write(h)
f.close()

#open and read the file after the overwriting:
f = open("result.txt", "r")
print(f.read())

inspiration-from

Happy learning

by Antonio Feijao UK

lime-linux-ubuntu-step-by-step

LiME on Ubuntu Linux, live memory capture.

sources and learning material:

LiME step by step

My adaptation for manually testing LiME in a step-by-step method.

USE AT YOUR RISK

## check if LiME is installed

if [[ `lsmod|grep lime|wc -l` -gt 0 ]] ; then
	sudo rmmod lime.ko
fi

kernel_release=$(uname -r)
kernel_name=$(uname -s)

echo "
kernel_release : ${kernel_release}
kernel_name    : ${kernel_name}
"

## function - I executed one line at a time
installLimeApt() {
	sudo apt-get -y update
	sudo apt-get -y install git
	
	sudo apt-get install -y linux-headers-$1
	#sudo apt-get install -y linux-headers-${kernel_release}
	
	sudo apt-get install -y build-essential
	
	cd /tmp && sudo rm -rf LiME
	
	git clone https://github.com/504ensicsLabs/LiME
	# >> could not clone, so I copyed 1 file at a time <<
	
	cd LiME/src
	
	make
	
	lime_path=$(pwd)/lime-$1.ko
	#lime_path=$(pwd)/lime-${kernel_release}.ko
	echo "lime_path : ${lime_path}"
}


# I run the commands one by one
#installLimeApt $kernel_release

# loading the kernel module
sudo insmod $lime_path path=tcp:4444 format=lime localhostonly=1 &

# confirm the LiME kernel module is "listening" on port 4444
netstat -patnl | grep 4444

#sleep 120

if [[ `lsmod|grep lime|wc -l` -gt 0 ]] ; then
	echo "LiME has been loaded"
fi

MEMSIZE=`awk '/MemTotal/ {print $2/1024/1024}' /proc/meminfo`
echo "MEMSIZE: ${MEMSIZE}"

METADATA_FLAG="--metadata uncompressed-size=$MEMSIZE,kernel-name=$kernel_name,kernel-release=$kernel_release"
echo "METADATA_FLAG : ${METADATA_FLAG}"
# sample output >>> `METADATA_FLAG : --metadata uncompressed-size=31.0748,kernel-name=Linux,kernel-release=4.4.0-184-generic`


# copying memory dump into S3
#s3cp() {
# aws s3 cp - {{s3ArtifactLocation}}/linux_memcapture$1 $2 $3 $4
#}

# original command
# cat < /dev/tcp/127.0.0.1/4444 | tee >(gzip | s3cp \".lime.gz\" \"$EXPECTED_SIZE_FLAG\" \"$METADATA_FLAG\" \"$ACL_FLAG\") | sha256sum | s3cp \"_sha256.txt\" \"$ACL_FLAG\"",

# compressed memory
#cat < /dev/tcp/127.0.0.1/4444 | tee >(gzip > ./linux_memcapture.lime.gz)

# raw memory dump

cat < /dev/tcp/127.0.0.1/4444 > ./linux_memcapture.lime
sha256sum linux_memcapture.lime >> _sha256.txt

# remove the kernel module
# most of the time I tested, the kernel module `lime.ko` "removed" itself.

sudo rmmod lime.ko

Happy learning,

Antonio Feijao UK

Linux dnsmasq options, dns-server and more

Linux dnsmasq options, dns-server and more.

More about dnsmasq in here - https://en.wikipedia.org/wiki/Dnsmasq

Starting with an example of a DHCP pool definition for dnsmasq

(...)

 dhcp-range=192.168.100.101,192.168.100.199,255.255.255.0,8h   # dhcp range
 dhcp-option=3,192.168.100.20                                  # default-gateway
 dhcp-option=6,1.1.1.3,1.0.0.3                                 # dns
 dhcp-option=15,mylocalnetwork.local                           # local-domain
 dhcp-option=44,0.0.0.0                                        # netbios server. Disabling NetBIOS over TCP/IP can improve security by reducing the attack surface of a system. However, it may also impact the functionality of certain legacy applications and networked devices that rely on NetBIOS

(...)

and with the command dnsmasq --help dhcp you get the below output, which shows what options are available and what they are.

Known DHCP options:
  1 netmask
  2 time-offset
  3 router
  6 dns-server
  7 log-server
  9 lpr-server
 13 boot-file-size
 15 domain-name
 16 swap-server
 17 root-path
 18 extension-path
 19 ip-forward-enable
 20 non-local-source-routing
 21 policy-filter
 22 max-datagram-reassembly
 23 default-ttl
 26 mtu
 27 all-subnets-local
 31 router-discovery
 32 router-solicitation
 33 static-route
 34 trailer-encapsulation
 35 arp-timeout
 36 ethernet-encap
 37 tcp-ttl
 38 tcp-keepalive
 40 nis-domain
 41 nis-server
 42 ntp-server
 44 netbios-ns
 45 netbios-dd
 46 netbios-nodetype
 47 netbios-scope
 48 x-windows-fs
 49 x-windows-dm
 58 T1
 59 T2
 60 vendor-class
 64 nis+-domain
 65 nis+-server
 66 tftp-server
 67 bootfile-name
 68 mobile-ip-home
 69 smtp-server
 70 pop3-server
 71 nntp-server
 74 irc-server
 77 user-class
 80 rapid-commit
 93 client-arch
 94 client-interface-id
 97 client-machine-id
119 domain-search
120 sip-server
121 classless-static-route
125 vendor-id-encap
150 tftp-server-address
255 server-ip-address

Happy learning,

Antonio Feijao UK

Using a Raspberry Pi 4 as a router with iptables

Using a Raspberry Pi 4 as a router with iptables. With iptables we need to know more about what we are doing. ufw is great, it works as a leayer on top of iptables for with easy management, however you will not learn the real "thing", the network flow details, the beauty of "source" and "destination", and more...

So, I want to learn more, therefore I went on to learn the details of iptables and source destination IPs, NAT (MASQUERADE), source and destination ports, states...

the raspberry pi 4 basics

raspberry pi 4 updates

apt update && apt upgrade -y

apt autoremove -y

## useful

apt install dnsutils

raspberry pi 4 disable ipv6 at boot

vim /boot/cmdline.txt and add ipv6.disable=1 to the end of the line

example

console=tty1 root=PARTUUID=xxxxXXxx-xx rootfstype=ext4 fsck.repair=yes rootwait ipv6.disable=1

raspberry pi 4 enable IPv4 forward and disable IPv6

  • cat /etc/sysctl.d/local.conf
net.ipv4.ip_forward=1

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

raspberry pi 4 nat with iptables

  • cat reset-iptables.sh
#!/bin/bash -x
##
## source - https://www.linode.com/docs/guides/linux-router-and-ip-forwarding/
##
## 2023-06 - adapted and tweaked by AntonioFeijaoUK
##

## reset iptables
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT

## enable forwarding packets
iptables -A FORWARD -j ACCEPT

## detecting the default route interface
GATEWAY_INTERFACE=$(route -n | grep ^'0.0.0.0' | rev | cut -f 1 -d ' ' | rev | head -n1)
echo "your gateway interface is : ${GATEWAY_INTERFACE}"

## enable NAT on the outside interface for the internal subnet SOURCE_SUBNET
SOURCE_SUBNET="192.168.0.0/24"
iptables -t nat -s ${SOURCE_SUBNET} -I POSTROUTING -o ${GATEWAY_INTERFACE} -j MASQUERADE

## other good sources with details info
# - https://raspberrytips.com/raspberry-pi-firewall/
# - https://www.packetswitch.co.uk/raspberry/
## - enabled established connections - this is not needed as the FORWARD is set to default ACCEPT
#iptables -A FORWARD -i eth0 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#iptables -A FORWARD -i eth0 -d ${SOURCE_SUBNET} -m state --state RELATED,ESTABLISHED -j ACCEPT
#iptables -A FORWARD -i wlan0 -o eth0 -j ACCEPT

## save IPv4 iptables
iptables-save | sudo tee /etc/iptables/rules.v4


##################################################################
### similar but for IPv6 and to block by default
ip6tables -F
ip6tables -X
ip6tables -t nat -F
ip6tables -t nat -X
#ip6tables -t mangle -F
#ip6tables -t mangle -X
ip6tables -P INPUT DROP
ip6tables -P OUTPUT DROP
ip6tables -P FORWARD DROP

## enable forwarding packets
ip6tables -A FORWARD -j DROP

## enable NAT on the outside interface
#iptables -t nat -s 192.168.0.0/24 -I POSTROUTING -o enp0s3 -j MASQUERADE

## save IPv6 iptables
ip6tables-save | sudo tee /etc/iptables/rules.v6


## USEFUL COMMANDS

echo "

useful command to check your NAT MASQUERADE is working

	\`iptables -t nat -L -nv\`

"

raspberry pi 4 static IP, dhcp and gateway metrics

  • cat /etc/dhcpcd.conf | egrep -v '^#|^$'
hostname
clientid
persistent
option rapid_commit
option domain_name_servers, domain_name, domain_search, host_name
option classless_static_routes
option interface_mtu
require dhcp_server_identifier
slaac private
interface wlan0
metric 300
static domain_name_servers=94.140.14.15 94.140.15.16
interface eth0
domain antoniofeijaouk.local
search antoniofeijaouk.local
metric 200
static ip_address=192.168.0.4/24
static routers=192.168.0.1
static domain_name_servers=94.140.14.15 94.140.15.16

raspberry pi 4 - verify the dns upstream servers

resolvconf -l

resolvectl status

Happy learning,

Antonio Feijao UK

Raspberry Pi 4 as a router with ufw rules

Raspberry Pi 4 as a basic "router" with ufw rules.

requirements

  • update rpi
apt update && apt upgrade -y

apt autoremove -y

apt install ufw

## useful

apt install dnsutils
  • raspberry pi 4 disable ipv6 at boot

vim /boot/cmdline.txt and add ipv6.disable=1 to the end of the line

example

console=tty1 root=PARTUUID=xxxxXXxx-xx rootfstype=ext4 fsck.repair=yes rootwait ipv6.disable=1
  • vim /etc/default/ufw
DEFAULT_FORWARD_POLICY="ACCEPT"
  • cat /etc/sysctl.d/local.conf
net.ipv4.ip_forward=1

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

raspberry pi 4 nat with ufw

  • vim /etc/ufw/before.rules
(...)
#------------------------------------------------------------------------
## sources
## https://gist.github.com/kimus/9315140
## https://www.server-world.info/en/note?os=Ubuntu_22.04&p=ufw&f=2
#
# add to the end
# NAT
*nat
-F
:POSTROUTING ACCEPT [0:0]

# Forward traffic through wlan0
-A POSTROUTING -s 192.168.0.0/24 -o wlan0 -j MASQUERADE

# don't delete the 'COMMIT' line or these nat table rules won't
# be processed
COMMIT

sample of basic ufw rules

  • eth0 - "internal" network with static IP

    • metric 300, for lower priority for default gw

  • wlan0 - "external" network with DHCP

    • metric 200, for default gw priority
## in routes - FROM and TO rules

ufw allow from 192.168.0.0/24 to 192.168.0.0/24 port 22 proto tcp



## out rules

ufw allow out on wlan0 from 192.168.0.0/24 to 94.140.14.15 port 53 proto udp
ufw allow out on wlan0 from 192.168.0.0/24 to 94.140.14.15 port 53 proto udp

ufw allow out on eth0 from 192.168.0.0/24 to 94.140.14.15 port 53 proto udp
ufw allow out on eth0 from 192.168.0.0/24 to 94.140.14.15 port 53 proto udp

ufw allow out on wlan0 from 192.168.0.0/24 to any port 443 proto tcp
ufw allow out on eth0 from 192.168.0.0/24 to any port 443 proto tcp

ufw allow out from 192.168.0.0/24 to any port 443 proto tcp
ufw allow out from 192.168.0.0/24 to any port 80 proto tcp


## enable ufw logging

ufw logging on

other usefull ufw commands

# ufw reset

ufw disable

ufw enable

ufw status numbered

ufw delete RUL_NUM

raspberry pi 4 static IP, dhcp and gateway metrics

  • cat /etc/dhcpcd.conf | egrep -v '^#|^$'
hostname
clientid
persistent
option rapid_commit
option domain_name_servers, domain_name, domain_search, host_name
option classless_static_routes
option interface_mtu
require dhcp_server_identifier
slaac private
interface wlan0
metric 200
static domain_name_servers=94.140.14.15 94.140.15.16
interface eth0
domain feijaouk.local
search feijaouk.local
metric 300
static ip_address=192.168.0.4/24
static routers=192.168.0.1
static domain_name_servers=94.140.14.15 94.140.15.16

raspberry pi 4 - verify the dns upstream servers

resolvconf -l

Happy learning,

Antonio Feijao UK

2023-06-09-raspberry-pi-4-as-a-router-with-ufw-rules.md

Linux bash script, basic script that records the website availability HTTP code 200s, 300s, 400s, 500s

Linux bash script, basic script that records the website availability HTTP code 200s, 300s, 400s, 500s

"Linux bash script, basic script that records the website availability HTTP code 200s, 300s, 400s, 500s"

To do next... color coding alerts maybe?!

the-linux-bash-script

#!/bin/bash

IFS='
'

LIST_URLs="https://www.antoniofeijao.com/
https://www.antoniofeijao.pt/
https://www.antoniocloud.com/
https://antonio.cloud/
https://www.cyberantonio.com/
https://www.cloudsecurity.cc/
https://www.securitygames.net/
https://www.root.pt/
https://www.ninja.pt/
https://www.ntp.pt/"


for URL in $(echo ${LIST_URLs} | tr '\ ' '\n'); do
    while true; do
        LOGS="${URL:8:-1}-$(date +%F).txt" && \
        DATE=$(date +%F-%H%M-%Ss) && \
        RESULT=$(curl -I ${URL} --silent | head -n 1) && \
        echo -e "${DATE}; \t ${URL}; \t ${RESULT}" >> ${LOGS} && \
        sleep 15
    done &
done

Happy learning,

Antonio Feijao

cyberantonioctf

Finding duplicated files using command line CLI in Linux MacOS Ubuntu

Easy way to find duplicated files in a folder or in all disk

find . -type f -name "*" -print0 | xargs -0 -I {} shasum -a 256 {}

Or course, output the above command to a file, cut -f 1, then sort and pipe it into uniq -c to count duplicates

find . -type f -name "*.JPG" -print0 | xargs -0 -I {} shasum -a 256 {} > finding-duplicates.txt

cat finding-duplicates.txt | cut -f 1 -d ' ' | sort | uniq -c | sort -rn | head -n10

grep 30848de6dba6f90bef4027fbf97a66dcc6f1f2eb3e8a6e47f0e9ce3fc411ce79 finding-duplicates.txt

of course, we now can automate this to keep the first file but move the duplicated into a backup folder before deleting them.

Example of my output on a folder with old photos that got duplicated over type... some photo is now 6x duplicated... time to automate tidying up!

Happy learning,

Antonio Feijao

Raspberry Pi 4 testing USB Wireless

Raspberry Pi 4 USB Wireless dongles that work out-of-the-box

Tested on Raspberry Pi 4 with Linux version 5.10.59-v7l+

dmesg | grep "Machine model"
[    0.000000] OF: fdt: Machine model: Raspberry Pi 4 Model B Rev 1.1


cat /proc/version
Linux version 5.10.59-v7l+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1447 SMP Thu Aug 19 12:25:41 BST 2021

Hercules USB dongle - DID worked out-of-the-box

usb 1-1.3: new high-speed USB device number 3 using xhci_hcd
usb 1-1.3: New USB device found, idVendor=06f8, idProduct=e033, bcdDevice= 2.00
usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-1.3: Product: 802.11n WLAN Adapter
usb 1-1.3: Manufacturer: Realtek
usb 1-1.3: SerialNumber: 00e04c000001
rtl8192cu: Chip version 0x10
rtl8192cu: Board Type 0
rtl_usb: rx_max_size 15360, rx_urb_num 8, in_ep 1
rtl8192cu: Loading firmware rtlwifi/rtl8192cufw_TMSC.bin
ieee80211 phy1: Selected rate control algorithm 'rtl_rc'
usbcore: registered new interface driver rtl8192cu
rtl8192cu: MAC auto ON okay!
rtl8192cu: Tx queue select: 0x05



lsusb  | grep Realtek
Bus 001 Device 003: ID 06f8:e033 Guillemot Corp. Hercules HWNUp-150 802.11n Wireless N Pico [Realtek RTL8188CUS]


lsusb -t

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 3: Dev 3, If 0, Class=Vendor Specific Class, Driver=rtl8192cu, 480M


iwconfig wlan1
wlan1     IEEE 802.11  ESSID:off/any
          Mode:Managed  Access Point: Not-Associated   Tx-Power=20 dBm
          Retry short limit:7   RTS thr=2347 B   Fragment thr:off
          Encryption key:off
          Power Management:off
          

usb 1-1.2: new high-speed USB device number 4 using xhci_hcd
usb 1-1.2: New USB device found, idVendor=2357, idProduct=012d, bcdDevice= 2.10
usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-1.2: Product: 802.11ac NIC
usb 1-1.2: Manufacturer: Realtek
usb 1-1.2: SerialNumber: 123456


lsusb | grep TP
Bus 001 Device 004: ID 2357:012d TP-Link

lsusb -t

 Port 2: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 480M


iwconfig
# show no new interface as it does not have drivers for it...

Wifi AC - did NOT work out-of-the-box

usb 1-1.1: USB disconnect, device number 5
usb 1-1.3: new high-speed USB device number 6 using xhci_hcd
usb 1-1.3: New USB device found, idVendor=0bda, idProduct=c811, bcdDevice= 2.00
usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-1.3: Product: 802.11ac NIC
usb 1-1.3: Manufacturer: Realtek
usb 1-1.3: SerialNumber: 123456


lsusb
Bus 001 Device 006: ID 0bda:c811 Realtek Semiconductor Corp.

lsusb -t
 Port 3: Dev 6, If 0, Class=Vendor Specific Class, Driver=, 480M
 
 iwconfig
# show no new interface as it does not have drivers for it...

testing drivers - https://github.com/cilynx/rtl88x2bu

or follow these instructions https://thepihut.com/blogs/raspberry-pi-tutorials/how-to-setup-a-rtl881cu-usb-wifi-adapter-with-the-raspberry-pi-4

this didn't work for me...

sudo apt install git bc dkms

mkdir usb-wifi-drivers

cd usb-wifi-drivers

git clone https://github.com/whitebatman2/rtl8821CU

cd rtl8821CU


## edit Makefil and update/change for your Raspberry Pi

vim Makefile

look for section 

###################### Platform Related #######################

## update for your "Platform", example below for Raspberry Pi 4

CONFIG_PLATFORM_I386_PC = n
CONFIG_PLATFORM_ARM_RPI = y
CONFIG_PLATFORM_ARM_RPI3 = n

Happy learning

Antonio Feijao UK

Micro-SD cards write speed test

While copy the Raspberry Pi image into a couple of micro-sd cards using the method "setting up a headless raspberry pi" (link below),

source https://www.raspberrypi.org/documentation/computers/configuration.html#setting-up-a-headless-raspberry-pi

I noticed the different speeds writes betweek micro-sd cards, therefore, create this post to list their writing speed.

diskutil list

diskutil unmountDisk /dev/disk4

sudo dd bs=1m if=2021-05-07-raspios-buster-armhf-lite.img of=/dev/rdisk4 ; sync

touch /Volumes/boot/ssh

touch /Volumes/boot/wpa_supplicant.conf

#vim /Volumes/boot/wpa_supplicant.conf

cat <<EOF > /Volumes/boot/wpa_supplicant.conf
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
        scan_ssid=1
        ssid="MY_WIFI_NAME"
        psk="MY_WIFI_PASSWORD"
        proto=RSN
        key_mgmt=WPA-PSK
        pairwise=CCMP
        auth_alg=OPEN
}

EOF

cat /Volumes/boot/wpa_supplicant.conf

diskutil list

sudo diskutil eject /dev/rdisk4

micro-sd card write speed results

sd-card - Sandisk Extreme 32 GB, V30, [3] A1
1874853888 bytes transferred in 26.577356 secs (70543281 bytes/sec) - 70.54 MB/sec

sd-card - Lexar 1000x, 32 GB [3]
1874853888 bytes transferred in 40.834400 secs (45913590 bytes/sec) - 45.91 MB/sec

sd-card - Lexar 633x, 32 GB [1]
1874853888 bytes transferred in 105.653969 secs (17745229 bytes/sec) - 17.74 MB/sec

sd-card - ScanDisk Ultra, 16 GB, A1 (10)
1874853888 bytes transferred in 138.986259 secs (13489491 bytes/sec) - 13.48 MB/sec

sd-card - Sandisk Ultra, 64 GB, (10)
1874853888 bytes transferred in 258.922797 secs (7240976 bytes/sec) - 7.24 MB/sec

Happy learning

Antonio Feijao UK