Another consequence of supply shortage: mass production mishaps

Chips may suddenly disappear from the supply chain or get really expensive due to the recurring supply shortage, companies are now designing their PCBs to support multiple chips either selecting drop-in replacements or creating multiple footprints to cater to at least one alternative part. That means one PCB and multiple bill-of-materials are needed to be more resilient to any supply disruption.

Radxa did that on the ROCK 3A board for the USB PD circuitry with the ability to use either Injoinic IP2315 or WCH CH224D IC. Those are not pin-to-pin compatible chips, and two circuitry were made for the board requiring two BoMs as it’s not just possible to only replace IP2315 by CH224D.

Radxa 3A schematics IP2315 CH224DBut this information did not get to the inventory manager, and once IP2315 was out of stock, he/she believed it would be possible to simply replace IP2315 with CH224D. The unlucky part is that the factory did not notice/report the mismatch, and the board still passed QC during mass production using a 5V power adapter, although no voltage higher than 5V could be negotiated. This resulted in the company shipping Rock 3A boards with the wrong circuitry between April 22, 2022 and June 15, 2022, or nearly 2 months.

Since the board works fine under light loads or with few peripherals connected, it’s quite possible many users did not notice. But the board is more prone to random reboots under heavy loads or when multiple peripherals are connected to the board which increases the power consumption. But wait… How do I know all that? That’s because Radxa did the right thing and publicly announced the issue on its forum, apologized to its users, and offered resolutions we’ll discuss below.

ROCK 3A Production Batch

You can check the manufacturing date of your board on the box as shown above.

Radxa also provides three potential resolutions to the issue:

  1. Replace or refund the affected board through the distributor where the board(s) was/were purchased
  2. Provide a free IP2315 IC for DIY users to self-repair (Advanced users only)
  3. Provide a free 12V/3A USB power adapter

We’ve recently been complaining about the (cursed) 12V-only USB power adapters provided with some mini PCs and a Raspberry Pi 4 laptop, but it turns out they may have their use…

mass production mishap IP2315 CH224D
CH224D can be removed, and replaced by IP2315

Radxa also took measures to prevent such incidents to occur in the future:

  1. Addition of an ADC voltage test case that requires a voltage greater than 9V to be detected for it to pass QC.
  2. Production must follow the BOM strictly, and for any change of components, the ERP administrator must be notified and provide a new and correct BOM.
  3. The factory has been informed that If there is a mismatch between the BOM and the parts, they must check with our engineers…

While there have been some mistakes along the way and lessons learned, this particular mass production mishap would not have happened without the current supply shortage that increased the complexity of the design, sourcing, and manufacturing.

Share this:
FacebookTwitterHacker NewsSlashdotRedditLinkedInPinterestFlipboardMeWeLineEmailShare

Support CNX Software! Donate via cryptocurrencies or become a Patron on Patreon

ROCK Pi 4C Plus

11 Replies to “Another consequence of supply shortage: mass production mishaps”

  1. I am not and have never been a custom of Radxa, so I don’t have a dog in this fight. This is exactly the kind of good design I would want to see from a board supplier. It’s unfortunate that they didn’t have a test in place for this substitution and I hope they use this as a learning experience so they don’t run into this again in the future. That said, they did the right thing by admiting it and offering several ways to make good thie mistake.

    If I can offer any advice as an engineer who’s had their designs mass manufactured, it’s “expect the factory to do something horrible and be prepared”. Board testing isn’t just there to see if the parts work, it’s to make sure that it was *made* right as well. Oh, and visit the factory. It’s good not only because you can see how they do things (which should help you design more defensively), but you get to meet the people and establish a relationship with them–which can go a long was towards preventing problems like this. If the factory people know how to reach you and feel they *can* talk to you, you’re way more likely to get a call asking about things like this before they just go and do it. If you’re really smart, involve someone from the factory in the design process so they feel involved–it can give them a sense or belonging and ownership of the design. They’ll be much less likely to cut corners in ways that can cause problems.

    Factories are made of people. The sooner you realize that and accept it, the better things will be for everyone.

    1. I totally agree with you, for having done this in a different context (studies vs production). Keeping a good relationship with those doing the dirty work is essential to help them help you when you make a mistake, and to let them know that the earlier their own mistake is reported, the better and that they won’t be blamed for this and that what matters is that a solution is found. This relation helps a lot to report stuff that looks strange such as “are they really hoping to make that IC work with no resistors around?” instead of thinking “pfff yet another improvised electronics engineer who will have a funny surprise”.

      And yes, I was surprised that Radxa had to design for multiple chips, that must not be fun at all and that implies different BOMs. So it *is* expected that trouble may happen anywhere down that chain, starting from the usual “let’s ignore this failure, it’s probably caused by the chip replacement”.

  2. I think such warning text should be good to see?

    Don't use this power supply for powering any other device than ROCK 3A!

      1. True. Then let me rephrase the text then (based on radxa wiki).

        Warning: 12V USB C dummy power adapter is dangerous to other 5V only devices, don’t mix them.

          1. I am actually using a laptop power adapter higher voltage than 12V for few of boards I mentioned.
            To avoid problems I put white tag on all cables with info “19V”.

          2. Living dangerously, huh? I understand, I have 5.5×2.5mm plugs for my Odroid boards which are in both 5V and 12V, so I understand that practice. They are well labeled and I am very careful plugging them in.

            At least they’ve moved to a much better power front end that can accept >12V. I sure wish they would go as high as 20V so that cheap 19(.5)V laptop supplies could be used.

          3. I had a cheap 12V power adapter as well but when I did check on power requirements of radxa boards I was like with higher voltage current goes down even more than on 12V. So decided to swap these and that first one I started using on something else.

          4. Yeah, they use a switching regulator, so the Wattage has to remain the same (Volts times Amperes), so the current goes down as the Voltage goes up. Helpful hint: Current is normally where switching power supplies lose efficiency, so upping the input voltage of a buck regulator often increases its efficiency. Also, used 19-20V laptop power bricks are effectively free given how many go to scrap every year. Many of them are well built and efficient.

          5. Actually, the opposite is true. See e.g. this figure for the TI LM62460:



            Efficiency at 4A Output current (5V) goes down from 94% to 91% when going from 13.5V to 24V Input voltage, i.e. losses go up by 50% (6% vs 9%, 1.2W vs 1.8W).

            This is typical for all switching converters.

Leave a Reply

Your email address will not be published. Required fields are marked *

Weller PCB manufacturer
Weller PCB manufacturer