RPI3 virtual AP + STA - Is this the state of the art?

[epoch1970]

Trying to make use of the virtual AP on a Pi3. This is with the latest Raspbian Lite, dhcpcd disabled to avoid interferences.
The following sequence seems to produce a working AP and STA out of the builtin wifi adapter in Pi 3. This works around a few issues that I suspect related to the brcmfmac module.
This is a long post, so I'll go ahead and ask my question in advance:is this really the best way to make AP+STA work?

The virtual AP interface must be created before first use of wlan0. Removing/reloading the brcmfmac module could work to reset the system state, but I've seen crashes. A udev rule looks like the best solution to have the AP ready to serve when needed.

Code: Select all

rasp# cat /etc/udev/rules.d/virt-ap.rules
ACTION=="add", SUBSYSTEM=="ieee80211", ATTR{macaddress}=="b8:27:eb:??:??:??", KERNEL=="phy[0-9]", RUN+="/sbin/iw phy %k interface add ap%n type __ap"

Also note the interface must absolutely not have its MAC address changed, otherwise AP clients get an IP but can't communicate properly.

Starting the AP breaks down the STA's association with its AP, and it never recovers. The only workaround I have found is to kill/restart the wpa_supplicant process altogether. No measure of reauth/reset/reload with wpa_cli would do.
It is possible to start wpa_supplicant standalone, and remove/add wlan0 via wpa_cli. It works and is easier than chasing for a PID in some circumstances.
Here is a sequence I've been running after boot, ap0 created by udev, eth0 networking up:
(Redirections to /dev/null to conceal my MACs)

Code: Select all

# execute as root
export WLAN_GW='172.17.0.1'
for IF in wlan0 ap0; do for OPT in c f m p; do iwgetid -${OPT} $IF; done; done # Live interfaces configs
grep channel /etc/hostapd.conf # AP channel in config file

wpa_supplicant -B -g/var/run/wpa_supplicant_master -Groot # Start WPA Supplicant master process
wpa_cli -g/var/run/wpa_supplicant_master interface_add wlan0 /etc/wpa_supplicant.wlan0.conf nl80211,wext >/dev/null 2>&1 # Start wlan0
sleep 1
udhcpc -i wlan0 -q >/dev/null 2>&1 # Get an IP addr
ping -c 3 ${WLAN_GW} # Check networking
sleep 5

ifconfig ap0 192.168.1.1 netmask 255.255.255.0 up # Config AP IP addr
dnsmasq -C /etc/dnsmasq.conf                      # For clients
lighttpd -f /etc/lighttpd/lighttpd.conf           # To test clients access

wpa_cli -g/var/run/wpa_supplicant_master interface_remove wlan0 >/dev/null 2>&1 # Delete wlan0
hostapd -B /etc/hostapd.conf >/dev/null 2>&1 # Start the AP
wpa_cli -g/var/run/wpa_supplicant_master interface_add wlan0 /etc/wpa_supplicant.wlan0.conf nl80211,wext >/dev/null 2>&1 # Start wlan0 again
sleep 5
ping -c 3 ${WLAN_GW} # Check networking

# Interfaces configs
for IF in wlan0 ap0; do for OPT in c f m p; do iwgetid -${OPT} $IF; done; done # Live interfaces configs
iw ap0 info | grep -o 'channel.*' # AP channel in actual operation

# Platform info
cat /proc/cpuinfo | grep -P '(Hardware|Revision)'
uname -a
lsb_release -a

Here is a run obtained by copy-pasting the previous block:

Code: Select all

rasp# # execute as root
rasp# export WLAN_GW='172.17.0.1'
rasp# for IF in wlan0 ap0; do for OPT in c f m p; do iwgetid -${OPT} $IF; done; done # Live interfaces configs
wlan0     Mode:Managed
wlan0     Protocol Name:"IEEE 802.11"
ap0       Mode:Master
ap0       Protocol Name:"IEEE 802.11"
rasp# grep channel /etc/hostapd.conf # AP channel in config file
channel=11
rasp# 
rasp# wpa_supplicant -B -g/var/run/wpa_supplicant_master -Groot # Start WPA Supplicant master process
Successfully initialized wpa_supplicant
rasp# wpa_cli -g/var/run/wpa_supplicant_master interface_add wlan0 /etc/wpa_supplicant.wlan0.conf nl80211,wext >/dev/null 2>&1 # Start wlan0
rasp# sleep 1
rasp# udhcpc -i wlan0 -q >/dev/null 2>&1 # Get an IP addr
rasp# ping -c 3 ${WLAN_GW} # Check networking
PING 172.17.0.1 (172.17.0.1) 56(84) bytes of data.
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=1102 ms
64 bytes from 172.17.0.1: icmp_seq=2 ttl=64 time=39.0 ms
64 bytes from 172.17.0.1: icmp_seq=3 ttl=64 time=24.0 ms

--- 172.17.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2064ms
rtt min/avg/max/mdev = 24.054/388.698/1102.949/505.089 ms, pipe 2
rasp# sleep 5
rasp# 
rasp# ifconfig ap0 192.168.1.1 netmask 255.255.255.0 up # Config AP IP addr
rasp# dnsmasq -C /etc/dnsmasq.conf                      # For clients
rasp# lighttpd -f /etc/lighttpd/lighttpd.conf           # To test clients access
rasp# 
rasp# wpa_cli -g/var/run/wpa_supplicant_master interface_remove wlan0 >/dev/null 2>&1 # Delete wlan0
rasp# hostapd -B /etc/hostapd.conf >/dev/null 2>&1 # Start the AP
rasp# wpa_cli -g/var/run/wpa_supplicant_master interface_add wlan0 /etc/wpa_supplicant.wlan0.conf nl80211,wext >/dev/null 2>&1 # Start wlan0 again
rasp# sleep 5
rasp# ping -c 3 ${WLAN_GW} # Check networking
PING 172.17.0.1 (172.17.0.1) 56(84) bytes of data.
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=1088 ms
64 bytes from 172.17.0.1: icmp_seq=2 ttl=64 time=39.6 ms
64 bytes from 172.17.0.1: icmp_seq=3 ttl=64 time=8.24 ms

--- 172.17.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2052ms
rtt min/avg/max/mdev = 8.245/378.815/1088.566/502.033 ms, pipe 2
rasp# # Interfaces configs
rasp# for IF in wlan0 ap0; do for OPT in c f m p; do iwgetid -${OPT} $IF; done; done # Live interfaces configs
wlan0     Channel:1
wlan0     Frequency:2.412 GHz
wlan0     Mode:Managed
wlan0     Protocol Name:"IEEE 802.11"
ap0       Mode:Master
ap0       Protocol Name:"IEEE 802.11"
rasp# iw ap0 info | grep -o 'channel.*' # AP channel in actual operation
channel 1 (2412 MHz), width: 20 MHz, center1: 2412 MHz
rasp# 
rasp# # Platform info
rasp# cat /proc/cpuinfo | grep -P '(Hardware|Revision)'
Hardware	: BCM2835
Revision	: a02082
rasp# uname -a
Linux rasp 4.14.34-v7+ #1110 SMP Mon Apr 16 15:18:51 BST 2018 armv7l GNU/Linux
rasp# lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 9.4 (stretch)
Release:	9.4
Codename:	stretch
rasp# 

Thanks for reading
So my question again: is there anything that can be done to improve on this?

[epoch1970]

That wasn't hacky enough. Here is a new version that allows to bridge ap0 and use an open AP while the STA uses WPA-PSK.
Don't ask my why this works. But it doesn't seem to crash.

Udev rule with power saving disabled (not sure this makes any difference.)

Code: Select all

pi@rasp:~ $ cat /etc/udev/rules.d/virt-ap.rules 
ACTION=="add", SUBSYSTEM=="ieee80211", ATTR{macaddress}=="b8:27:eb:??:??:??", KERNEL=="phy[0-9]", \
RUN+="/sbin/iw wlan%n set power_save off", \
RUN+="/sbin/iw phy %k interface add ap%n type __ap", \
RUN+="/sbin/iw ap%n set power_save off"

STA config (WPA 2):

Code: Select all

pi@rasp:~ $ cat /etc/wpa_supplicant.wlan0.conf 
#ctrl_interface=udp
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=root
update_config=1
country=FR

network={
ssid="whatever"
psk="password"
}

AP config. Fairly complete, I removed stuff from a working WPA2 hostapd config until I got to an AP that clients see as "open", yet the STA would stay happy with its encrypted setup. wpa=0 is nonsense, but hey, it works:

Code: Select all

pi@rasp:~ $ cat /etc/hostapd.conf 
ctrl_interface=/var/run/hostapd
ctrl_interface_group=0
country_code=FR
bridge=local0
interface=ap0
driver=nl80211
ssid=test
hw_mode=g
channel=11
wmm_enabled=1
macaddr_acl=0
max_num_sta=5
wpa=0
#auth_algs=1
#wpa=2
#wpa_passphrase=thisisatest
#wpa_key_mgmt=WPA-PSK
#wpa_pairwise=TKIP CCMP
#rsn_pairwise=CCMP
logger_syslog=-1
logger_stdout=0

Dnsmasq config. Nothing to see, really.

Code: Select all

pi@rasp:~ $ cat /etc/dnsmasq.conf 
interface=local0
bind-interfaces
# Just in case
no-dhcp-interface=eth0
no-dhcp-interface=wlan0

domain=test,192.168.22.1,192.168.22.255
expand-hosts
strict-order
cache-size=255
no-negcache
domain-needed
local-ttl=60
server=172.17.0.2

# Local-only domains answered from static hosts file or DHCP leases.
local=/test/
dhcp-authoritative
dhcp-range=set:test,192.168.22.100,192.168.22.200,15m
dhcp-lease-max=25

This mess added to /etc/rc.local. I can't explain why apparently the bridge local0 can adopt any MAC for itself and ap0 will still communicate via wlan0. And conversely that explicitly making ap0 promiscuous -once wlan0 is restarted- is mandatory to get wifi clients to communicate, although ap0 is a bridge member.

Code: Select all

echo 1 > /proc/sys/net/ipv4/ip_forward
udhcpc -i eth0 -q
sleep 3
ip route delete default dev eth0
#ip route add default via 172.19.0.1 metric 100 dev eth0
ip link add local0 type bridge # Add a bridge
ip link set local0 up
ip link set eth1 down
ip link set eth1 address fe:00:00:11:22:33 # Have the bridge adopt ap0's MAC. Not necessary?
ip link set eth1 up
ip link set ap0 master local0
ip link set eth1 master local0

ip addr add 192.168.22.1/24 broadcast 192.168.22.255 dev local0 # Config AP IP addr
wpa_supplicant -B -g/var/run/wpa_supplicant_master -Groot # Start WPA Supplicant master process
sleep 5

wpa_cli -g/var/run/wpa_supplicant_master interface_add wlan0 /etc/wpa_supplicant.wlan0.conf nl80211,wext  # Start wlan0
sleep 1
udhcpc -i wlan0 -q  # Get an IP addr
sleep 5

dnsmasq -C /etc/dnsmasq.conf                      # For clients
lighttpd -f /etc/lighttpd/lighttpd.conf           # To test clients access

wpa_cli -g/var/run/wpa_supplicant_master interface_remove wlan0  # Delete wlan0
sleep 3
hostapd -B /etc/hostapd.conf  # Start the AP
sleep 3
wpa_cli -g/var/run/wpa_supplicant_master interface_add wlan0 /etc/wpa_supplicant.wlan0.conf nl80211,wext  # Start wlan0 again
sleep 5
udhcpc -i wlan0 -q  # Get an IP addr
ip link set ap0 promisc on # Is necessary when bridged?

iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

I have to say I am a bit surprised, but all this seems to lead to a working AP+STA setup, the AP being password-free. Throughput has its ups and downs it seems, but local connectivity is fine between wireless clients and wired clients connected to the bridge.

I am posting this while connected to the open AP. If you read this post, it a good sign... and if the config crashes in the next few days I'll post an update.
There should be an update

[epoch1970]

Ok final update I think. This version doesn't use a bridge I don't really need it, so one less concern.
On my test setup network starts in rc.local, I have run the machine in 2 configs:
1) wifi (+ether) as uplink, 2) ethernet as uplink. To switch from one to another I've put a switch in rc.local. This is the non-standard part of rc.local in my test rig:

Code: Select all

echo "Stuff added to rc.local"
echo 1 > /proc/sys/net/ipv4/ip_forward
[ ! -f /var/run/wpa_supplicant_master ] && \
  /sbin/wpa_supplicant -B -g/var/run/wpa_supplicant_master -Groot && \
  sleep 5

ACTIVATE="ETH WIFI"
for IF in ${ACTIVATE}; do
  case ${IF} in
	ETH)
	# eth0 as uplink
	/sbin/udhcpc -p /var/run/udhcp.eth0.pid -i eth0 -b
	/sbin/iptables -t nat -A POSTROUTING -s 192.168.22.0/24 -o eth0 -j MASQUERADE
	;;
	WIFI)
	# wlan as uplink - metric?
	/sbin/wpa_cli -g/var/run/wpa_supplicant_master interface_add wlan0 /etc/wpa_supplicant.wlan0.conf nl80211,wext
	sleep 3
	/sbin/udhcpc -p /var/run/udhcp.wlan0.pid -i wlan0 -b
	/sbin/iptables -t nat -A POSTROUTING -s 192.168.22.0/24 -o wlan0 -j MASQUERADE
	;;
  esac
done

Still using the same udev rule as in the previous post, I think I should remove the power_save option thing.

Currently I have an /etc/modprobe.d/cfg80211.conf file, I suspect it is useless as well:

Code: Select all

options cfg80211 ieee80211_regdom=FR

I wrote a script to harness the startup/shutdown process of the AP in the 2 network configs. I am afraid this is the minimal recipe. I had to resort to unloading/reloading brcmfmac, this seems unavoidable sometimes. Rather than catching arcane errors I unload/reload the module every time, the OS seems to hold on. Here is /home/pi/virt-ap.sh:

Code: Select all

#!/bin/sh
action=$1

RUNS='/var/run/virt-ap'
IF_STA='wlan0'
IF_AP='ap0'
NET_IP_AP='192.168.22.1/24'
NET_BC_AP='192.168.22.255'

CONF_STA="/etc/wpa_supplicant.${IF_STA}.conf"
CONF_AP='/etc/hostapd.conf'
CONF_NS='/etc/dnsmasq.conf'
PF_AP='/var/run/hostapd.pid'
PF_NS='/var/run/dnsmasq.pid'
PF_DHCP_STA="/var/run/udhcp.${IF_STA}.pid"

WS_SOCK='/var/run/wpa_supplicant_master'
WS_STA=0

is_STA_up(){
	# If wlan0 is active we need to know
	WS_IFS=$(/sbin/wpa_cli -g${WS_SOCK} interface 2>/dev/null)
	while read -r line; do
 	  ( echo "$line" | grep -q "^${IF_STA}" ) && WS_STA=1
	done <<EOF
echo "${WS_IFS}"
EOF
}

do_start(){
	echo -n "Starting virtual AP: "
	is_STA_up
	if [ $WS_STA -eq 1 ]; then
	  # Remove interface to cover the ordinary failure, module for that
	  # little extra push sometimes required...
	  echo -n "reset $IF_STA... "
	  /sbin/wpa_cli -g${WS_SOCK} interface_remove ${IF_STA} >/dev/null 2>&1
	  sleep 1
	  /sbin/modprobe -r brcmfmac
	  sleep 1
	  /sbin/modprobe brcmfmac
	  sleep 5 # Seems necessary
	fi
	echo -n "start $IF_AP... "
	/usr/sbin/hostapd -B -P ${PF_AP} ${CONF_AP} >/dev/null 2>&1
	if [ $WS_STA -eq 1 ]; then
	  echo -n "reconfigure $IF_STA... "
	  sleep 5 # Seems necessary
	  /sbin/wpa_cli -g${WS_SOCK} interface_add ${IF_STA} ${CONF_STA} >/dev/null 2>&1
	fi
	echo -n "AP IP config... "
	/sbin/ip addr add ${NET_IP_AP} broadcast ${NET_BC_AP} dev ${IF_AP}
	/usr/sbin/dnsmasq -x ${PF_NS} -C ${CONF_NS} >/dev/null 2>&1
	touch ${RUNS}
	echo "done."
}

do_stop(){
	echo -n "Stopping virtual AP... "
	kill $(cat ${PF_NS}) $(cat ${PF_AP}) >/dev/null 2>&1
	/sbin/ip addr delete ${NET_IP_AP} dev ${IF_AP}
	/sbin/ip link set ${IF_AP} down
	rm -f ${RUNS}
	echo "done."
}

case $action in
 "start")
	[ ! -f ${RUNS} ] && do_start || echo "AP already active"
	;;
 "stop")
	[ -f ${RUNS} ] && do_stop || echo "AP is already inactive"
	;;
 *)
	echo "Usage: $0 start|stop"
	exit 0
esac

exit

Hostapd config /etc/hostapd.conf, open AP:

Code: Select all

ctrl_interface=/var/run/hostapd
ctrl_interface_group=0

interface=ap0
driver=nl80211
max_num_sta=15

ssid=test
hw_mode=g
channel=11
wmm_enabled=1
ieee80211n=1

ieee80211d=1
country_code=FR

wpa=0

logger_syslog=-1
logger_stdout=0

(BTW, I also played with an AP setup in wpa_supplicant, via option "mode=2". I made it sort of work, it proved no less troublesome, and the AP didn't seem to switch automatically to the channel used by the STA, as hostapd does now.)

Standard WPA2 config for the STA, nothing to see in /etc/wpa_supplicant.wlan0.conf. Same with dnsmasq.conf.

HTH

[IngoH]

I'm also working on this issue but used another way. I use systemd to solve this because you can very easy define dependencies between the needed services, for example AP + STA with routing or a little bit more complicated AP + STA + ETH with bridging.

[epoch1970]

Hmm. Your bridged setup does 2 interesting things: giving a MAC to uap0 and bridging it. As I recall, in my tests changing the MAC was harmful and bridging didn’t work properly (setting uap0 promisc was only a point fix)

I understand you’ve been using another version of the OS and a 3b+. Could these make a difference?

[IngoH]

@epoch1970

I understand you’ve been using another version of the OS and a 3b+. Could these make a difference?

Im also using "the latest Raspbian Lite". Where is the difference?
Afaik it makes no difference using a RPi 3B or 3B+.

I'm just trying to use only wpa_supplicant instead of hostapd but had problems with the wifi driver brcmfmac. It complains errors when using wpa_supplicant on two interfaces (ap0 and wlan0). Now just for some days there is coming an update. I will try again.