Saturday, September 26, 2026

Why just switcharoo'ing RAM on a RPI isn't guaranteed to work, or "wait, you have to do WHAT to get DDR to work?"

There's been some fun stuff lately in the maker community around Raspberry Pi and RAM upgrades. Specifically, a recentish (~ 2025) boot firmware upgrade locks the RAM size to what it shipped with. There's been some write-ups about it - https://www.jeffgeerling.com/blog/2026/raspberry-pi-ram-lockdown/ and https://hackaday.com/2026/09/22/raspberry-pi-ram-restrictions-no-big-deal-frankly/ come to mind - but they don't REALLY go into all the gory details behind the scenes.

So let's do a quick dive into RAM and how it's changed from the past until now.

First off - a lot of tinkering in retro computing exposes people to static and dynamic RAM chips, first dating back to the late 70s. After a few attempts at stuff, manufacturers soon ended up kind of iterating towards a few standard pinouts and specification timings - likely so new upstart RAM makers could sell their chips as "drop in replacements" for existing RAM.

This led manufacturers towards defining an actual standard for this stuff - JEDEC (https://en.wikipedia.org/wiki/JEDEC_memory_standards) - which defines what and how various types of RAM need to behave. This is needed so you didn't end up with RAM chips needing to be paired with very specific controllers / CPUs. Some vendors kept the market on their toes with very not compatible stuff (think the RDRAM from Rambus - https://en.wikipedia.org/wiki/RDRAM) but in general one of the big reasons we had affordable RAM until recently (and will again, I'm sure) is because there's basically no vendor lock-in and a lot of interoperability.

However, that interoperability comes at a cost. Silicon isn't just some magical thing you can stamp out and get a million chips that all behave exactly the same. There's always going to be subtle behavioural differences across them. For CPUs this can actually show up as different maximum clock frequency versus heat dissipation - some CPUs will run at the maximum clock frequency at a lower temperature versus others - and yes when you graph them you'll likely get a normal or bi-modal distribution. For desktop CPUs with active cooling this may result in the same CPU being sold at a higher price with a higher maximum clock frequency, and then a "slightly cheaper" CPU being sold with a lower maximum clock frequency. If it's going into something like a cellphone or a piece of wearable technology where there's no active cooling, you either have to choose whether to lower EVERYONES maximum device performance, allow SOME devices to operate at a maximum higher performance, or you reject a percentage of the CPUs from the product and use them elsewhere.

(Yes, this is what they call "binning".)

For RAM, stable frequency, timing and bus characteristics are critical. Remember these parts aren't operating at 1 or 10MHz anymore - these chips operate their busses on both sides of the clock cycle at over 1GHz. At that point you're well into the "this behaves like RF" domain. As part of this, the PCB layout design, the connector design, the individual track lengths (think phase delays) + lengths inside the memory controller all have to line up exactly correctly - and then you still have weird RF style behaviour going on. Remember how I said every chip behaves differently? They're going to have slightly different timing and slightly different RF characteristics. So you can't just whack a single set of parameters down and hope they work - you need to go through calibration, validation and bulk testing before you know you've got something that works.

First up - the DDR4 power-up sequence is not just "apply power, add RAS/CAS timings and go." There's a whole calibration sequence that has to happen to calibrate the memory controller PHY (physical interface). For an example of this, take a look at the AMD Soft core (FPGA) DDR4 SDRAM controller calibration / setup documentation at https://docs.amd.com/r/en-US/pg353-versal-acap-soft-ddr4-mem-ip/Memory-Initialization-and-Calibration-Sequence . It has to be able to run this sequence to completion, every time, or you won't have reliably working RAM. 

What parameters are required to make this work?

Well, we get cheap, mostly interoperable DIMMs because the JEDEC standard defines what parameters to put into the DIMM EEPROM module that is read by the memory controller during power on. Yes, it's also documented online - https://en.wikipedia.org/wiki/Serial_presence_detect has all the fun details. Note that there's many, many more parameters here than just RAS/CAS timings! There's all kinds of timings to do with voltage calibration management, nanosecond accurate wait and hold times for various parts of the RAS/CAS signaling, data line skew, and other fun behavioural things.

Now here comes the fun part.

A DIMM manufacturer is going to make some test DIMMs and test them on a whole variety of motherboards to make sure that the parameters they program in reliably result in the above PHY calibration/setup succeeding 100% of the time on all the hardware they care about. They're then going to choose (hopefully conservatively) suitable values to write into the SPD to allow this to, yes, calibrate/setup 100% of the time. They may even slow down the RAM to meet timing if needed. But typically they're going to hope that the manufacturer binning of the SDRAM parts into specific part numbers and performance/behavioural bands will result in being able to make a DIMM that consistently operates at a specific performance tier without them having to further bin things when they're on the DIMM. Noone wants to throw away manufactured things, so the manufacturer has the option to just write slower parameters into SPD and sell the DIMM as a slower DIMM.

But an embedded board developer doesn't have some EEPROM attached to each group of RAM modules they've put on the board. They will have likely characterised one, two or three RAM module suppliers (either for different sizes, or the same size but multiple manufacturers so they can play off pricing against each other when making different batches of boards) and put those parameters into a fixed lookup table on said embedded board as a starting point. If you're fancy (and I don't know if RPI is being fancy or not) you'll actually run the calibration on your manufacturing line and write the SDRAM parameters for your chosen RAM into the EEPROM as the "active" configuration.

And here you see the dilemma.

See, the configuration parameters are either conservative ones for a handful of tested RAM chips from specific manufacturers, or they're very specifically for the RAM you've had installed on your embedded board. They've done a whole lot of characterization of the RAM across first one, then three, then a dozen or two, then hundreds, and finally the whole run as they're making them, to minimise the chances that during actual bulk manufacturing that a board fails RAM calibration/setup.

So when you just switch the SDRAM chip over to some other manufacturer part? You're basically hoping that given the initial RAM configuration parameters stored in EEPROM/flash somewhere on your board, the RAM calibration/setup will always succeed and then normal operation is stable. And you may get lucky! You may not! But you absolutely aren't guaranteed anything at this point.

And this is why the Raspberry Pi Foundation is doing the right thing here. Sorry y'all, but at this point you're actually likely not qualified to do all of the above work.

... and why not? Why can't you calibrate/test the RAM and write the parameters back?

A great question! The short answer is this.

It's typically vendor provided tooling that is not open source and is very proprietary, and they don't want you to know how to do it or what their memory controller is doing.

And THAT is a discussion for another time.

So now you know. It's not as simple as "RPI foundation is good" or "RPI foundation is bad". It's an intersection of how modern hardware works, how much effort goes into making sure it's bullet proof (or as bullet proof as they can get it) and what vendors / manufacturing lines are willing to release versus what they're requiring to keep as closed / NDAed stuff.

 

(Don't hate the player, hate the game, etc) 

Sunday, June 7, 2026

The great 2024/2025/2026 tech firings, and what happens to all that clue

So yeah, I was caught up in the Meta/Facebook layoffs in March 2026.

There are a lot of parallels to the 2001 dot com bust, and I'd like to just briefly touch on one of them now.

There's been a lot of layoffs over the last year or three in tech. I think the tech market has seen what, more than 100,000 in the last year alone? And most of those aren't due to performance. There are a lot of very smart people who've spent quite a few years learning how to build, debug and maintain all kinds of interesting stuff.

And they're now in the marketplace.

When this happened in 2000/2001, you had a whole lot of people who learned how to build internet tech and infrastructure at companies that were pushing the boundaries with things. And they were let go, due to downturns, change of focus, all kinds of reasons.

A lot of those people went off to eventually build the new stuff that likely overtook a lot of those internet and telco companies in the 90s.

I think this is going to happen with the layoffs from Meta/Facebook and other technology companies. Meta ran (runs?) a huge research arm in Reality Labs pushing the boundaries in AR, XR, wearable/portable technology. There have been plenty of public technology demonstrations showing all the stuff going on before the current AI trend.

Just over in Reality Labs - people learned how to build stuff from ASIC design up through optics, display, camera technology, highly miniaturized electronic design and power, all the fun stuff around 2D/3D audio and video stuff on wearable glasses (how do you provide head and world locked surfaces to your applications and not have it be so laggy that it gives people headaches?), making it all work over wifi, and then .. well imagine the lessons learned in what can and can't work in manufacturing the devices and where the pain points are in current technology. (And yes, a lot more I can't talk about, for hopefully obvious reasons.)

That knowledge is now embedded in a few hundred people, soon to be a few thousand people, who are being laid off. And that's just reality labs - the other people spread throughout the company and other technology companies have gathered a lot of experience about how to make and grow technology "stuff", what works and what doesn't.

At some point some of those people are going to make startups that do really amazing things. The technology underpinning a lot of what people have been trying to make now will get better (and I will argue that right now it is good enough - if you're willing to shift your focuses a little) and things will appear that will knock the socks off the current offerings. They're not what you would view as "founders" in the bay area tech scene. They're the people who know how to make things work, not necessarily the people who got rich early on in the scene.

Just like what happened after the dot com bust of 2001.

I've heard from a few people now that those they're hiring from Meta and other large technology companies are top notch and know what they're doing. You can't spend 10 years at a company that heavily invested in cutting edge R&D and not learn a thing or two.

I think its their loss, and .. eventually, everyone elses gain.

 

Sunday, February 15, 2026

Being accused of taking down a Data Centre, or "what happens when adrian is the only one who isn't oversubscribed on power?"

This happened circa what, 2008? 2007? I forget now. Anyway, it was a while ago now - and it wasn't my fault - but at the time it was pretty "wtf oh shit I'm in so much trouble" scary levels of scary.

So here we go.

Around that time I was running a little consulting / hosting company. I went into the data centre which hosted my equipment to install a second hand Dell I had acquired and tested. I plugged it into the rack, then plugged in the power, then pushed the on button.

Then bang. Then everything went dark. Then bang again.

Then I get a phone call on the VOIP phone in the data centre. It was their owners, asking me what the fuck I had done.

Anyway, it turned out that everything was dead. It wasn't just that the circuit breakers had tripped. A large part of their power equipment in the DC had also gone.

So yes, I was blamed for taking out a whole data centre. By plugging in one Dell server.

But why was I not in court over it? Why am I not still paying back the damages? Well, it turns out there's way, way more to the story.

First up they added a new rule - "You need to test equipment at this outlet/breaker before you install it." Cool, my server definitely passed that test. It worked just fine.

But then it turns out that although my rack was perfectly under the rated current limits, the other racks were not. Like, in any meaningful way. There were some other customers way, way over their allotted power. So when I plugged in my server - again, I'm way way under my own rack power allotment - I tripped some breaker on the distribution board.

Tripping that breaker meant that the other phase now took the brunt of the load. Again, my rack is fine, but everyone elses was apparently not, so it .. pulled very hard on that rail. And it immediately tripped the second phase breaker.

But that wasn't it.

Then, the second click was when they tried remote flipping the breakers back on.  The massive draw of power on phases when the computers in the data centre were powered back up caused some part of their power distribution setup to just plain fail. I forget the exact details here; I think the UPS was pulled on pretty hard too in that instant and I vaguely recall it also got cooked.

I had like, four? servers, a router and a switch. I was definitely not going to cause inrush problems. But the big hosting customers? Apparently they .. had more. Much, much more. Now this isn't my first rodeo when it comes to power sequencing of servers in a data centre - I had done this for like, a LOT of Sun T1s in circa 2000, staging how to turn them on a rack at a time upon a full power-off / power-on cycle event. But apparently this wasn't done by either the data centre or the hosting customers. All power, all on, all at once.

Now, I'm a small fry customer with one rack still paying the early adopter pricing. The companies in question had a lot more racks and were paying a lot more. So, this was all mostly swept under the rug, I stopped being blamed, and over the next few months we all got emails from the data centre telling us about the "new, very enforced power limits per rack, and we're going to keep an eye on it."

Anyway, fun times from ye olde past when I was doing dumb stuff but people were making much more money doing much dumber stuff at times. 

Monday, December 15, 2025

Installing a 32k DS1386 into an SGI Indy that expects an 8k DS1386

 The TL;DR is - you can't just plug a 32k DS1386 into an SGI Indy and have it work, as a bunch of stuff is wrong.

 The longer version!

Here's a picture from the datasheet:

Now, at the outset it looks like A13 and A14 on the 32k module need to at least be grounded - if they're floated or high then the RAM will be selected when you don't want it to be, and you'll just fail to see the RTC.

 However, if you just do that your Indy won't boot because it turns out that pin 3 on the 8k module is an undocumented (in this datasheet) auxiliary Vcc input - the 5v auxiliary / always-on rail is actually there. Shorting it to ground will just make the Indy super sad. Don't do that.

So, in lieu of making up a PCB (which I think I'm going to do anyway just to be "clean" - I used some pin headers to raise the DS1386 above the Indy PCB.


 Note that pin 3 on and pin 28 don't have pins (and I'm going to put some tape over the pins just to be super clear nothing shorts out).

Then, I did a quick bodge wire job on the underside of the DS1386-32K module:


 Where pin 3 and pin 28 are tied to pin 16.

 Then, well, plug it in, align it right, and it should just work! It did for me!

Thursday, December 4, 2025

Blinking the SGI Indy Power Light

I figured out what broke in netbsd to make the R5000SC I have here poweroff during startup.

In any case, the hack for blinking the front panel LED, inspired from their BLINKY driver, shouldn't be lost when i clean up my tree.

 

So, I give you all:

+void
+blink_led(void)
+{
+       uint32_t reg;
+       uint32_t i;
+
+       reg = *(volatile uint32_t *)MIPS_PHYS_TO_KSEG1(0x1fbd9870) & 0xff;
+       reg = reg ^ 0x10; /* toggle? */
+       *(volatile uint32_t *)MIPS_PHYS_TO_KSEG1(0x1fbd9870) = reg;
+       asm(".set mips3; sync");
+
+       for (i = 0; i < 20 * 1000 * 1000; i++)
+               asm(".set mips3; nop ; nop");
+}

It's terrible but it works!

Also, amusingly, the ARC BIOS console IO was available at this point but nothing was being printed early enough for it to be seen! Debugging was quite a bit faster once I realised I was getting far enough for the ARC BIOS entrypoint to be initialised so I could printf().

Tuesday, December 2, 2025

Reverse Engineering the SGI Indy Monitor Detection, or "thank god someone added SGI indy / indigo 2 support to MAME"

 I have a bit of a soft spot in my heart for the SGI Indy and (purple, not teal, heh) Indigo 2.

So imagine my surprise when NetBSD "almost" booted just fine on the Indy I have acquired. R4600PC-100, XL8 graphics .. and wonky console colours in netbsd / wonky xorg.

The first deep dive is "why are the monitor colours so unpredictable?" and that got me into a fun deep dive into how the SGI Indy Newport graphics works, the whole SGI Indy Linux project circa 2000, hardware shortcuts and software shortcuts.

Anyway.

The TL;DR is here - https://erikarn.github.io/sgi/indy/monitor_detection  . I've listed the monitor resolution/refresh rates the internet and my reverse engineering based on what MAME was programming.

So the long version.

First up - I've put all the hardware documentation I've found so far at https://erikarn.github.io/sgi/indy/notes . 

The Indy was booting NetBSD in either correct colours - green kernel text, white userland console text - or incorrect colours - green kernel text, but blue console text. It was random, and it was per boot. X11 was no better - sometimes it had the correct colours, sometimes everything was wonky.

The NetBSD console code tries to setup the following things for 8 bit graphics mode (which is used for console, even for 24 bit cards):

  • Program in an 256 entry colourmap table, matching what the NetBSD RGB 332 colour scheme is;
  • Add in a 1:1 RGB ramp in another colour table (RGB2);
  • A bunch of "XMAP9 mode" lines mapping 32 entries of "something" to RGB8 pixel format, RGB2 colourmap.

I was very confused as to what was and what should be going on, and I don't want to dig into the journey I took to get here. But the TL;DR here is that everything in the NetBSD console setup path is wrong and when it "worked", it ended up with the wrong colours. And when it "didn't work", it sometimes ended up with the wrong colours.

I'll write a separate post later about how the whole newport graphics system holds together, but fixing this requires a whole lot of driver changes to correctly program the hardware, and then some funky monitor timing specific programming.

The 13W3 port on the Newport graphics boards have a 4 bit monitor ID which compatible monitors will output. There's more details available at https://old.pinouts.ru/Video/sgivideo_pinout.shtml . 


The "universal 13W3 interface input cable" that I bought has a bunch of DIP switches controlling this.


 

If you have the four monitor ID bits set on or off, then you still get 1024x768 @ 60Hz.

The "fun" part of this story is if I were using 1280x1024 straight off the bat then I'd likely not have seen these problems happen so often.

Anyway.

Depending upon the settings, the Indy will boot with a bunch of different possible monitor setups:

  • 1024x768, 60Hz
  • 1024x768, 70Hz
  • 1280x1024, 60Hz
  • 1280x1024, 72Hz
  • 1280x1024, 76Hz

I enumerated this list and threw them up on the monitor detection link at the beginning of the article.

So, the firmware reads these four bits at boot (via 4 IO bits on one of the CMAP chips - again, see the links at the top of the post) sets up the monitor timing and then displays stuff. But when NetBSD's console programming is getting the colours wrong when I'm using 1024x768 60Hz.

It turns out that the XMAP chips - which handle the final mapping of incoming framebuffer pixel data to what 24 bit RGB is sent to the CMAP chip and then the RAMDAC -  were being programmed inconsistently. (again, they were being programmed incorrectly too in NetBSD, but I've got a big diff to fix that. With that, they're programmed correctly inconsistently.)

There's a "display control bus" that the Newport raster chip (REX3) has to peripheral chips. The peripheral chips - the XMAPs, the VC for timing, the RAMDAC, the CMAP for 8/24 bit colour table mapping - they're all DCB peripherals. The DCB has some address lines, 8 data bits, programmable chip select line, chip select setup, hold and release timing, optional request/ACK signaling, and register auto-increment functionality.

However!

  • The REX3 chip runs at 33MHz;
  • The XMAP chips run at 1/2 the pixel clock (they're interleaved);
  • The DCB has support for explicit ACK signaling from the peripheral, but as long as the peripheral uses it;
  • The XMAP does not have an ACK line, just an incoming chip select line, and
  • When writing the XMAP mode table lines - which map the display information to pixel format / colour table selection - it's done as back to back bursts to the same register, not an auto-increment and NOT using an ACK line.

This means that if the XMAP chip is running at a speed that doesn't entirely line up with the programmed chipselect timing, the mode writes will be junk. The normal 8 bit read/writes are "mostly fine" as they just show up as multiple 8 bit read/writes to the same register and for all the OTHER registers that is just fine. But for the mode register - where the DCB needs to write 4 bytes to the same individual address - it's absolutely not fine.

And that's the kicker.

After spending some quality time with the MAME emulator and some local hacks to enable the newport peripheral IO logging and setting the monitor ID, I found out that the timing used for the XMAP chips is different for 1024x768 60Hz versus 1280x1024 76Hz.

Everything worked just fine when I adjusted it.

So ok, I went back to the Linux and X11 drivers to see what's going on there, as I know the C code wasn't doing this. And I found this gem in the Linux newport.h header file:

 static __inline__ void
xmap9SetModeReg (struct newport_regs *rex, unsigned int modereg, unsigned int data24, int cfreq)
{
        if (cfreq > 119)
            rex->set.dcbmode = DCB_XMAP_ALL | XM9_CRS_MODE_REG_DATA |
                        DCB_DATAWIDTH_4 | W_DCB_XMAP9_PROTOCOL;
        else if (cfreq > 59)
            rex->set.dcbmode = DCB_XMAP_ALL | XM9_CRS_MODE_REG_DATA |
                    DCB_DATAWIDTH_4 | WSLOW_DCB_XMAP9_PROTOCOL;
        else
            rex->set.dcbmode = DCB_XMAP_ALL | XM9_CRS_MODE_REG_DATA |
                        DCB_DATAWIDTH_4 | WAYSLOW_DCB_XMAP9_PROTOCOL;
        rex->set.dcbdata0.byword = ((modereg) << 24) | (data24 & 0xffffff);
}

It's choosing different DCB timing based on the pixel clock. It lines up with what I've been seeing from MAME and it adds a third one - WAYSLOW - which I bet I'm only going to see on the PAL/NTSC timings or if something really wants to do something like 1024x768 50Hz.

The timings are in the header file, but .. nothing is using xmap9setModeReg(). It was likely copied from some internal SGI code (the PROM? X server? Who knows!) as part of the code bring-up but it was never used.

Anyway! With this in the NetBSD console code the console finally works reliably in all the modes I've tested. I'm going to try and get my big diff stack landed in NetBSD and then I'll work on the X11 newport code so it too supports 8 and 24 bit graphics at 1024x768 reliably.

  • Read the CMAP1 register (and PROM on SGI Indy) to determine the monitor type
  • The default monitor on SGI Indy is 1024x768 60Hz, and for Indigo2 its 1280x1024 60Hz
  • Select an XMAP9 mode DCB timing set based on the pixel clock
  • 8 bit mode for console and X11 needs the colour index table programmed into the CMAP at CI offset 0, and appropriate XMAP config for the display mode table to use 8 bit pixels, PIXMODE_CI, offset 0, NOT 8 bit RGB
  • 24 bit mode for X11 needs the 24 bit RGB ramp programmed into the CMAP RGB2 table (which is not a colour index table!), and no CMAP
  • Importantly, the X11 server uses truecolour for 24 bit mode, and pseudocolour / colourmaps for 8 bit mode, so all of this need repeating in the X11 server code! 

Here's how the console looks, complete with the correct XMAP9 mode table:

And here's how x11 looks:


 

 

(And the X11 support is even more fun, because I had to fix some stuff to make acceleration in the driver work, and that's going to be a whole fun DIFFERENT post.)

Addendum:

Oh, and the sync on green? It's generated by the RAMDAC. Once this all has landed in NetBSD I'm tempted to try to add a sysctl / boot parameter to disable the sync on green bit so normal monitors work on the SGI Indy. Let me know if you'd like that! 

Tuesday, November 11, 2025

A tale of an SGI Indy, a Sony power supply, and how to keep the fan spinning

Let's not dwell on why I bought an SGI Indy. Anyway.

One of the common things that I've seen is failure due to power supplies or heat death. The irony is:

  • The Nidec power supply has dirty power, fails hilariously, but the fan is at least always spinning, and
  • The Sony power supply has clean power, less hilarious failures, but the fan only comes on when the unit is hot. Sometimes.

 The fan in the Sony PSU is a 12V fan, and it turns on based on a thermal control line from the Indy. There's been plenty of research into the behaviour of that signal, and I'm not going to go into it here. What I instead want to talk about is a quick way to actually just get the fan constantly spinning, without having the open up and modify the power supply itself.

The TL;DR is this:

  • Make a small voltage gate using two diodes - one from the 3.3v power supply rail (via a resistor, I used 100 ohms; (1Kohm was too high) but I may try something smaller like 56 ohms to make sure enough current is flowing) and the other from the Indy board;
  • That way the Sony PSU is always fed at least 3.3v into that thermal sensor input.

 


So!

  • There will always be a minimum 3.3v voltage into the Sony PSU fan control, which is enough to turn it on; 
  • The fan will always spin at a minimum voltage;
  • if it DOES get warm enough for the Indy thermal sensor circuitry to feed above 3.3v (+ diode drop) into the thermal control line, it will also increase the fan speed.

It's not too hard - tie two diodes together at the cathode side, cut the brown control wire, feed it into the power supply, and then tie the two anodes in as above.


 

(The grey wire in the image is going from one diode anode to the 3.3v (white) wire in the Indy main PSU connector; there's 100 ohm resistor at the end of said grey wire that's temporarily jammed in for testing.)

With this the Sony PSU fan is always spinning, and your SGI Indy should die less of a heat death.