A RISC'y cluster - Part III

This may lead to accusations of over-engineering, but I solved a problem and had fun while doing so.

The four node cluster runs two different types of 5V fan, two fans are fairly quiet but clearly noticeable and two are somewhat loud. All in all the cluster is too loud, and noise aside, it’s silly to be running fans at full tilt when there is no need for cooling.

The plan

The VisionFive2 board has a 5V fan header, but no electronics behind it to allow regulating (or even switching on/off) the fan. However, it does have a good set of GPIO (general purpose I/O) pins, including a couple of PWM (pulse width modulation) pins. Unfortunately, the driver for the PWM output is not included in the mainline kernel and as such does not work out of the box.

The fans I use are simple 5V brushless fans; it is possible to regulate their speed (by regulating the 5V input voltage) and they will operate down to around 2V. They may need a bit more initially for starting up (to overcome static friction in the bearings), but details aside, they can certainly be regulated down to a speed where they are virtually impossible to hear.

In order to take the 3.3V PWM signal and “convert” it into 0-5V 150mA DC output for the fan, we are going to need a bit of electronics. This should not need to be complicated, and it should be small enough to fit inside the existing system (maybe stick it to the rear wall of the carrier with a piece of double sided tape).

Thus, the plan is:

1. Design PWM driver hardware
2. Make PWM output work on the board
3. Build regulation software to control the fan

The regulation software can be a PID (proportional, integral and differential) controller; the focus being on low resource consumption and “good enough” fan regulation.

Power management precursor

Now before over-thinking how we cool our processors, let’s at least be sure they are not burning more power than they need to be. The JH7110 supports clock scaling but this is not enabled by default in a Trixie install.

To enable this, there are two simple steps. First, install the linux-cpupower package (technically you can do without it, but it makes the setup marginally nicer):

$ sudo apt install linux-cpupower

Now, this tool will tell you that your CPU is running at top speed but not actually solve anything for you right away. I looked for an easy way to configure this, but I could not find any good and simple way to ask for CPU speed regulation in a Trixie install (at least when you’re not on an amd64 architecture). The solution, which is simple enough when you know it, is to create a systemd service file set up CPU frequency scaling. Simply create the file /etc/systemd/system/cpufreq-setup.service with the following contents:

[Unit]
Description=CPU scaling governor setup

[Service]
Type=oneshot
ExecStart=/usr/bin/cpupower frequency-set -g conservative
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

Go through the usual routine to enable the service on boot and start it up:

# systemctl daemon-reload
# systemctl enable cpufreq-setup.service
# systemctl start cpufreq-setup.service

Now you should be able to see that it works by calling:

# cpupower frequency-info
analyzing CPU 3:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency: 1000 us
  hardware limits: 375 MHz - 1.50 GHz
  available frequency steps:  375 MHz, 500 MHz, 750 MHz, 1.50 GHz
  available cpufreq governors: conservative performance schedutil
  current policy: frequency should be within 375 MHz and 1.50 GHz.
                  The governor "conservative" may decide which speed to use
                  within this range.
  current CPU frequency: 375 MHz (asserted by call to hardware)

So far so good. The CPU is now at 375MHz because the system is idle - it will scale up to 1.5GHz when needed. This does not remove the need for active cooling, it just limits pointlessly wasting energy.

Now let’s move on to the actual project…

PWM Driver hardware

A PWM output can be converted to a regulated DC output with simply an RC filter (a series resistor and a capacitor in parallel). However, there are a couple of reasons why we need something more involved:

1. PWM input is 3.3V, fan needs up to 5V
2. PWM input is very low current, fan needs hundreds of mA

A circuit that does low-side switching with an N-channel MOSFET before an LC filter could solve both of these. It would look something like:

This would work. However, since I’m going ahead and building a number of these anyway, I figured, this might be a little board that can be useful in other applications another day. For that reason, I don’t like driving the low side. I want the system ground to be the ground that goes to the fan - which means I need high side switching, not low side. Also, and I’m not a EMI (electro-magnetic interference) expert, but I like being able to route the system ground to power consumers. Ground is good.

With high side switching there are basically two ways to go

• High side N-channel MOSFET - which will require a driver with a boost capacitor (since gate needs to go a few volts above source and source is already at +5V in the on state)
• High side P-channel MOSFET - which can be driven much easier, but P-channel MOSFETs tend to have higher on resistances, higher cost and are just all around less nice.

Driver ICs for driving a high side NMOS are readily available, but, the boost capacitor tends to require that they are never driven at 100% duty cycle. The boost capacitor will need a least a tiny charge every now and again, and this can only happen when the high side NMOS is not on. To me this is a very inconvenient limitation in this application, since that will require not just my controller software to guarantee we never reach 100% duty cycle, but it will make it impossible to jury rig the controller with a simple GPIO pin, pull it to +3.3V and have the fan spin at full throttle (since DC is basically a 100% duty cycle PWM signal - or vice versa). So for reasons of versatility and simplicity of use (or “tolerance to improvisation”), I disqualify the high side NMOS.

A high side PMOS can be de-activated with a simple pull-up resistor and it can be activated with a pull-down NMOS which gets driven by the 3.3V PWM input. This would look like the following:

Now, the problem with this circuit is that the pull-up resistor R5 has to either be very small and thereby burn a lot of power when the low-side NMOS Q5 is on, or, if it is larger (even just 1k) the gate charge on Q6 will be too large for the pull-up resistor to be able to pull the Q6 gate up and de-activate the transistor in time.

On the topic of time… I plan to run the PWM output at 200kHz since this is supported by the JH7110 CPU, it is a high enough frequency that the LC filter can be relatively small, and it is a low enough frequency that I won’t have to worry about transmitting a radio signal through my control wire. But with a 200kHz PWM frequency, the period is only 5µs - meaning, if I don’t want to spend more than a few percent of the cycle switching the transistors on or off, the gate needs to move in maybe 1% of 5µs - which is 50ns. That is not a whole lot of time. Now if we end up spending 150ns every cycle that will be fine as well, but it is useful to know what sort of ball park timing we’re looking at.

A simulation (using the built-in Spice simulator in KiCad which I use for these schematics) will show this effect in great detail:

The blue line is the 3.3V PWM input, the brown line is the voltage on the gate of the high side PMOS switching transistor. It is clearly seen that when the PWM input goes high, the low-side NMOS activates and the PMOS gate gets pulled down hard. This is almost instant. However, when the PWM input goes low, the pull-up resistor takes its sweet time to pull the PMOS gate high. With a 200kHz PWM frequency, a 1k pull-up resistor and a PMOS I chose for the application, it takes more than half the off-period before the PMOS is actually fully off. Clearly this will not work - if only we could amplify the pull-up current without having it amplified when the low side NMOS is on… That would look like the actual final circuit:

The bipolar transistor Q2 will amplify the pull-up current (the pull-up resistor drives the base, Q2 emitter does the pulling-up of the Q3 gate) - but because of the trick with the diode D1 the low-side NMOS can actually pull down both the Q2 base (effectively stopping the pull-up amplification) and the Q3 gate (through the diode).

With a hFE of about 100, 5mA on the Q2 base becomes 500mA pulled up through the Q2 emitter/collector. The same simulation as before, looking at the Q3 gate voltage now looks like this:

As can be seen, both the rising and falling edges are sharp now. This will make for efficient switching at 200kHz, easy.

The two input resistors R3 and R2 could be left out. R2 pulls the input to ground if it would otherwise be floating - I think this is a nice addition. The R3 series resistor is a gate resistor to mitigate ringing that could otherwise occur when a gate is driven directly from a PWM output. Given that there is a piece of wire in between as well, we would probably be OK without it too. They are both there for good measure (and combined they add a whopping 0.01€ to the total cost of the board).

PCB design and assembly

The circuit was laid out on a double sided PCB; 13mm by 42.6mm, small enough to fit on the back of the node carrier (see the chassis post).

You can download the Gerber files below, which you can submit directly to JLCPCB or some other PCB manufacturer.

• gerbers.zip

As for the BOM, I used the following components:

You can get them all at DigiKey or wherever you buy parts.

Early assembly of the boards. You can see three transistors on the table in the top of the image. Resistors are rather large 805 (2012 metric - so 2mm long, 1.2mm wide) size, simply because I picked that size to standardise on when I started doing surface mount electronics. They are kind of large for what they are. Most capacitors I use are 603 (1608 metric, so 1.6mm long, 0.8mm wide) except for high capacitance ones that need more volume. The transistors in SOT-23 (Q3 and Q2) are easy to solder while the Q4 input NMOS is a SOT-416 case which is on the smaller side of what is comfortable to hand solder.

The SOT-416 easily gets big blobs of solder on it. This will work of course, but brushing it over with flux and simply re-heating with a clean tip will remove the excess solder and leave it more clean looking:

Anyway, throwing components on the boards is pretty straight forward.

Finished boards with the JST-PH connectors on the ends for fan, power and PWM connections:

The boards were sprayed with conformal coating and inserted in a piece of heat shrink just to protect the boards and the computer in case the board would come lose. We don’t want any electrical drama in the cluster.

The full board, mounted with double sided tape to the rear of the node carrier, wired in with a “franken-lead” getting power from the fan header and PWM signal from GPIO pin 32 (which is GPIO46 which is also PWM0):

Now with the hardware side done, let’s get the PWM signal going…

DKMS for out-of-tree driver

Fortunately, StarFive, the makers of the VisionFive2 board, have developed and submitted for inclusion in the Linux kernel, a driver for the JH7110 PWM functionality. The processor actually can drive 8 PWM channels, but it seems in the board configuration two are available (not sure about the details - not important for me either as I need just a single PWM channel).

Unfortunately, while I could find 17(!) versions of the patch to get the driver included, it has not yet been accepted in the mainline kernel. This means the stock Trixie install cannot do PWM. The most recent patch submission I found was this one.

Fortunately, this is not the first time someone wants a module that isn’t in mainline. The DKMS system is made for specifically this purpose - to allow a user to add a module source, and to automatically have that re-compiled against a new kernel and installed, whenever a kernel upgrade is applied. This allows us to include an out-of-tree driver but not have to deal manually with recompiling etc. whenever we apply updates to the system. Sweet!

I created a Debian DKMS package to install simply using the patch submitted by StarFive (all credit goes to them for the work). You can get the three files below to build the package yourself from sources:

• pwm-ocores-module-dkms_17.orig.tar.xz
• pwm-ocores-module-dkms_17-1.debian.tar.xz
• pwm-ocores-module-dkms_17-1.dsc

Download the three of them, put them in a folder and do as follows:

joe@chip:~/build$ ls -l
total 7
-rw-rw-r-- 1 joe joe 2696 Feb  4 08:16 pwm-ocores-module-dkms_17-1.debian.tar.xz
-rw-rw-r-- 1 joe joe  949 Feb  4 08:16 pwm-ocores-module-dkms_17-1.dsc
-rw-rw-r-- 1 joe joe 2548 Feb  4 08:16 pwm-ocores-module-dkms_17.orig.tar.xz
joe@chip:~/build$ dpkg-source -x pwm-ocores-module-dkms_17-1.dsc 
dpkg-source: warning: extracting unsigned source package (pwm-ocores-module-dkms_17-1.dsc)
dpkg-source: info: extracting pwm-ocores-module-dkms in pwm-ocores-module-dkms-17
dpkg-source: info: unpacking pwm-ocores-module-dkms_17.orig.tar.xz
dpkg-source: info: unpacking pwm-ocores-module-dkms_17-1.debian.tar.xz
joe@chip:~/build$ ls -l
total 7
drwxrwxr-x 5 joe joe   49 Feb  4 08:21 pwm-ocores-module-dkms-17
-rw-rw-r-- 1 joe joe 2696 Feb  4 08:16 pwm-ocores-module-dkms_17-1.debian.tar.xz
-rw-rw-r-- 1 joe joe  949 Feb  4 08:16 pwm-ocores-module-dkms_17-1.dsc
-rw-rw-r-- 1 joe joe 2548 Feb  4 08:16 pwm-ocores-module-dkms_17.orig.tar.xz
joe@chip:~/build$

Now you can build the binary package:

joe@chip:~/build$ cd pwm-ocores-module-dkms-17/
joe@chip:~/build/pwm-ocores-module-dkms-17$ dpkg-buildpackage -uc -us -b
dpkg-buildpackage: info: source package pwm-ocores-module-dkms
dpkg-buildpackage: info: source version 17-1
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by Jakob Oestergaard <joe@unthought.net>
dpkg-buildpackage: info: host architecture riscv64
 dpkg-source --before-build .
...
dpkg-genbuildinfo --build=binary -O../pwm-ocores-module-dkms_17-1_riscv64.buildinfo
 dpkg-genchanges --build=binary -O../pwm-ocores-module-dkms_17-1_riscv64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
joe@chip:~/build/pwm-ocores-module-dkms-17$ cd ..
joe@chip:~/build$ ls -l
total 19
drwxrwxr-x 5 joe joe   49 Feb  4 08:21 pwm-ocores-module-dkms-17
-rw-rw-r-- 1 joe joe 2696 Feb  4 08:16 pwm-ocores-module-dkms_17-1.debian.tar.xz
-rw-rw-r-- 1 joe joe  949 Feb  4 08:16 pwm-ocores-module-dkms_17-1.dsc
-rw-rw-r-- 1 joe joe 4643 Feb  4 08:22 pwm-ocores-module-dkms_17-1_riscv64.buildinfo
-rw-rw-r-- 1 joe joe 1103 Feb  4 08:22 pwm-ocores-module-dkms_17-1_riscv64.changes
-rw-r--r-- 1 joe joe 4808 Feb  4 08:22 pwm-ocores-module-dkms_17-1_riscv64.deb
-rw-rw-r-- 1 joe joe 2548 Feb  4 08:16 pwm-ocores-module-dkms_17.orig.tar.xz
joe@chip:~/build$

Now you have your “binary” Debian package (which in reality just contains the driver sources still - this is the whole point of a DKMS package, to enable recompilation whenever a kernel upgrade is installed). Installing the binary package will install the sources (for now and future use), compile the PWM module and install it:

joe@chip:~/build$ sudo dpkg -i pwm-ocores-module-dkms_17-1_riscv64.deb
(Reading database ... 76966 files and directories currently installed.)
Preparing to unpack pwm-ocores-module-dkms_17-1_riscv64.deb ...
Module pwm-ocores/17 for kernel 6.12.63+deb13-riscv64 (riscv64):
Before uninstall, this module version was ACTIVE on this kernel.
Deleting /lib/modules/6.12.63+deb13-riscv64/updates/dkms/pwm-ocores.ko.xz
Running depmod.............. done.

Deleting module pwm-ocores/17 completely from the DKMS tree.
Unpacking pwm-ocores-module-dkms (17-1) over (17-1) ...
Setting up pwm-ocores-module-dkms (17-1) ...
Loading new pwm-ocores/17 DKMS files...
Building for 6.12.63+deb13-riscv64

Building initial module pwm-ocores/17 for 6.12.63+deb13-riscv64
Sign command: /lib/modules/6.12.63+deb13-riscv64/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub

Building module(s)...... done.
Signing module /var/lib/dkms/pwm-ocores/17/build/pwm-ocores.ko
Installing /lib/modules/6.12.63+deb13-riscv64/updates/dkms/pwm-ocores.ko.xz
Running depmod............. done.

The output above is slightly contaminated by the fact that I already had the module installed when I re-installed it for this example, but your output should look similar, except for the the uninstall part.

Don’t forget to load the module:

joe@chip:~/build$ sudo modprobe pwm-ocores
joe@chip:~/build$

Now you should be able to verify that your PWM functionality is available:

joe@chip:~/build$ ls /sys/class/pwm/
pwmchip0
joe@chip:~/build$

Now, let’s put this functionality to good use…

Fan manager regulation software

The goal of the fan speed regulation software would be something like the following:

• Keep temperature at acceptable level
• Run fan as slow as possible to achieve the above
• No “hunting” - regulation must converge quickly

This probably lends itself well to a PID (proportional, integral, differential) controller. So let’s build one!

I won’t bore you with the details of the software, but put very simply the following were the goals or guiding ideas:

• Keep it Simple Stupid - this is not rocket surgery, a relatively simple solution to a relatively simple problem is possible
• That also means; not a forest of dependencies
• Low resource consumption - it’s just the fan
• Neatly integrated into systemd

I picked C++ as the implementation language because it is low-level enough to allow for CPU-efficient implementation and high-level enough to allow for human-efficient implementation (in other words; saving me from having to type too much). I know I know, I should have used whatever language came into fashion five minutes ago - but frankly I can’t be bothered to re-write this in two years when the fads have changed - a C++ implementation will work in fifty years too and it is a very good general purpose language.

The main challenges during implementation were:

Sample rate

To be CPU efficient we should sample as rarely as possible. But to have proper regulation and be able to respond to sudden load changes, we should sample fairly frequently.

The JH7110 processor has a fifth core, a tiny low-powered 32-bit core intended for monitoring / management applications such as the fan control. This core is not yet supported by mainline Linux so running the fan manager on that is not (yet) an option. We need to be mindful of not creating unnecessary load on the system - having the fan spin up to run the fan manager would be a little silly.

From experiments I ended up settling on 10 samples per second. It may sound like a lot but with the code doing minimal work and then sleeping for 100ms (which in computer terms is “forever” almost), it isn’t very expensive.

The final implementation used something like 2 minutes on the CPU after the first 22 hours of operation:

root@pluto:~# systemctl status fanman
● fanman.service - Fan speed management service
     Loaded: loaded (/usr/lib/systemd/system/fanman.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-02-03 18:43:50 CET; 22h ago
 Invocation: a454cd38640a4758bc3a95fb0d29b97d
       Docs: man:fanman
             man:fanman_config
   Main PID: 633 (fanman)
      Tasks: 1 (limit: 9453)
     Memory: 1.9M (peak: 2.1M)
        CPU: 2min 5.470s
     CGroup: /system.slice/fanman.service
             └─633 /usr/sbin/fanman /etc/fanman.conf

This amounts to 0.15% utilisation of a single core and 0.025% consumption of the 8G system memory. I think we can live with that.

Noisy temperature input

The program reads CPU temperature data from /sys/class/thermal/thermal_zone0/temp which presents the temperature in m°C (so 48000 is 40°C). This is great - an integer is efficient to read and the resolution is good enough for meaningful differentials.

The actual accuracy is not a single mK (or m°C if you will) though, and therefore the temperature reading will make abrupt jumps. This is ok for the Proportional and fine for the Integral part of the PID controller, but it really messes up the Differential part. Differentiating a discontinuous function is not a recipe for success.

For the proportional reading, I do not use the raw temperature data, instead I calculate a median of the past half second of readings (so five data points). This is a basic noise filtering mechanism.

For the differential reading I tried a few different models - I ended up settling on using a Savitzky-Golay filter which very efficiently does a least-squares fit of a polynomial to the data points and gets me the differential from there. This is CPU efficient (much of the arithmetic is integer and can pipeline well) and works at least as good as other models I tried involving sliding window medians etc.

Tuning the PID

A PID controller takes an “error input”; the error being the difference between the actual process variable (the measured CPU temperature in our case) and the desired set-point (the ideal CPU temperature in our case). Using three weights on each of the Proportional, Integral and Differential components, it sums them up and this becomes the PID control output - the “correction” to the process if you will (or the fan speed in our case).

The values of these three weights will decide how well the PID controller regulates. They determine over-shoot, oscillation, reaction-time and all of that.

I can highly recommend reading the Wikipedia article on the topic of PID controllers as it also covers tuning. I ended up writing a tuning utility which first uses the Åström–Hägglund method to determine a time period for system oscillation and amplitude. From there on, I use the Ziegler–Nichols method for calculating the three weights. This gives a starting point for the PID regulation parameters - from there on, actual live experiments need to be made.

In the configuration file for the fan manager, there is a gnuplot_out parameter which will make the software dump out key parameters every samle (so 10 lines per second). This allows for experimentation with hand-tuning the parameters - one can easily see impulse response to system influences like sudden load spikes etc.

The above plot is generated using the gnuplot output option - the system was at steady-state, then I ran a make -j4 compilation job, and then the system settled back into steady-state when the job had completed.

There is visible over-shoot when system load drops - we drop about 0.8°C under the ideal temperature (48°C). There is also a bit of over-shoot when settling into equilibrium again - the system temperature goes briefly about half a degree above the set-point.

Because of the limited history size, the integral component only looks back a finite period of time - this causes the regulated temperature to hover about 0.2°C above the actual set-point. This is well within what I can accept as a compromise.

One really important aspect, of course, is how quickly the system responds to load changes. You can see in the graph that the response is almost instantaneous - the moment the temperature rises sharply, the fan duty-cycle goes to 100% and stays there until the temperature is clearly on its way to dropping below the set-point.

Fan manager software download and setup

As before, I provide a source package that you can download:

• fanman_1.orig.tar.xz
• fanman_1-1.debian.tar.xz
• fanman_1-1.dsc

As before, download the files, unpack the source package, and build a binary package:

joe@pluto:~/build$ dpkg-source -x fanman_1-1.dsc 
dpkg-source: warning: extracting unsigned source package (fanman_1-1.dsc)
dpkg-source: info: extracting fanman in fanman-1
dpkg-source: info: unpacking fanman_1.orig.tar.xz
dpkg-source: info: unpacking fanman_1-1.debian.tar.xz
joe@pluto:~/build$ cd fanman-1/
joe@pluto:~/build/fanman-1$ dpkg-buildpackage -uc -us -b
dpkg-buildpackage: info: source package fanman
dpkg-buildpackage: info: source version 1-1
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by Jakob Oestergaard <joe@unthought.net>
dpkg-buildpackage: info: host architecture riscv64
...
dpkg-deb: building package 'fanman' in '../fanman_1-1_riscv64.deb'.
dpkg-deb: building package 'fanman-dbgsym' in '../fanman-dbgsym_1-1_riscv64.deb'.
 dpkg-genbuildinfo --build=binary -O../fanman_1-1_riscv64.buildinfo
 dpkg-genchanges --build=binary -O../fanman_1-1_riscv64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
joe@pluto:~/build/fanman-1$ ls -l ../fanman*deb
-rw-r--r-- 1 joe joe 23568 Feb  4 17:16 ../fanman_1-1_riscv64.deb
-rw-r--r-- 1 joe joe  5392 Feb  4 17:16 ../fanman-dbgsym_1-1_riscv64.deb
joe@pluto:~/build/fanman-1$ 

Now, as for setup, install the package and start it up. The default configuration will work on a VisionFive2. The configuration is in /etc/fanman.conf in a familiar format (like sysctl). You can use systemctl reload fanman to adjust parameters or enable/disable logging to the trace file. However, you cannot change PWM chip or channel, or change temperature sensor without restarting the service.

Oh and… I had every intention of writing man pages. I will probably do that some day. Or you can do it, and send them to me - thank you very much!

Conclusion

The cluster is very quiet now - except for when it’s busy.

With regulation, the noisy fans are now completely acceptable - almost impossible to hear (they run around 40-45% duty-cycle on the PWM) most of the time. The quiet fans, however, move much less air and even at full tilt the CPU temperature exceeds 53°C. Probably the longer term solution is to standardise on the noisy fans that can move more air - and maybe re-purpose the quiet fans for something where madly over-engineering a regulation solution isn’t an option.

Cheers!